A method performed by one or more computers, the method comprising: training a first machine learning model, wherein the first machine learning model is configured to receive an input and generate a respective score for each of a plurality of classes representing a probability that the class is a classification of the input; and training a second machine learning model on a plurality of training inputs, wherein the second machine learning model is also configured to receive inputs and generate scores for the plurality of classes representing a probability that the class is a classification of the input, wherein the second machine learning model has a different architecture to the first machine learning model, comprising: processing each training input using the first machine learning model to generate a first target soft output for the training input; and training the second machine learning model to, for each of the training inputs, generate a soft output that matches the first target soft output for the training input, wherein the first target soft output and the soft output that matches the first target soft output comprises a respective soft score for each of the plurality of classes generated by a last layer of the respective machine learning model, and wherein each soft score satisfies: q i = exp z i / T ∑ j exp z j / T, wherein q i is the score for a class i, z i is a weighted combination of outputs of a previous layer of the respective machine learning model received by the last layer, j ranges from 1 to a total number of classes in the plurality of classes, T is a temperature constant, and wherein to generate the soft scores, T is set to a higher value than used to generate scores for the classes after the respective machine learning model has been trained; and wherein the input is an image and wherein the scores for the plurality of classes represent a probability that the input contains an image of an object that belongs to the class. training a first machine learning model, wherein the first machine learning model is configured to receive an input and generate a respective score for each of a plurality of classes representing a probability that the class is a classification of the input; and training a second machine learning model on a plurality of training inputs, wherein the second machine learning model is also configured to receive inputs and generate scores for the plurality of classes representing a probability that the class is a classification of the input, wherein the second machine learning model has a different architecture to the first machine learning model, comprising: processing each training input using the first machine learning model to generate a first target soft output for the training input; and training the second machine learning model to, for each of the training inputs, generate a soft output that matches the first target soft output for the training input, wherein the first target soft output and the soft output that matches the first target soft output comprises a respective soft score for each of the plurality of classes generated by a last layer of the respective machine learning model, and wherein each soft score satisfies: q i = exp z i / T ∑ j exp z j / T, wherein q i is the score for a class i, z i is a weighted combination of outputs of a previous layer of the respective machine learning model received by the last layer, j ranges from 1 to a total number of classes in the plurality of classes, T is a temperature constant, and wherein to generate the soft scores, T is set to a higher value than used to generate scores for the classes after the respective machine learning model has been trained; and wherein the input is an image and wherein the scores for the plurality of classes represent a probability that the input contains an image of an object that belongs to the class. processing each training input using the first machine learning model to generate a first target soft output for the training input; and training the second machine learning model to, for each of the training inputs, generate a soft output that matches the first target soft output for the training input, wherein the first target soft output and the soft output that matches the first target soft output comprises a respective soft score for each of the plurality of classes generated by a last layer of the respective machine learning model, and wherein each soft score satisfies: q i = exp z i / T ∑ j exp z j / T, wherein q i is the score for a class i, z i is a weighted combination of outputs of a previous layer of the respective machine learning model received by the last layer, j ranges from 1 to a total number of classes in the plurality of classes, T is a temperature constant, and wherein to generate the soft scores, T is set to a higher value than used to generate scores for the classes after the respective machine learning model has been trained; and wherein the input is an image and wherein the scores for the plurality of classes represent a probability that the input contains an image of an object that belongs to the class.