Patent ID: 11900232
Assignee: GOOGLE LLC
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 0:
1. A method performed by one or more data processing apparatus, the method comprising:
training a student machine learning model having a plurality of student model parameters on a set of multiple training inputs, wherein:
during the training, the student machine learning model is configured to process an input to generate an output score distribution over a plurality of classes by performing operations comprising:
processing the input to generate an intermediate score distribution over the plurality of classes; and
processing the intermediate score distribution using an output layer of the student machine learning model to generate the output score distribution,
wherein the output layer is parameterized by a temperature parameter having a training value, wherein the temperature parameter controls a softness of the output score distribution; and

the training comprises:
processing each training input of the set of multiple training inputs using a teacher machine learning model to generate a target score distribution over the plurality of classes; and
training the student machine learning model to, for each training input of the set of multiple training inputs, process the training input to generate an output score distribution that matches the target score distribution generated by the teacher machine learning model for the training input;

wherein after the training, the temperature parameter of the output layer of the student machine learning model is configured to have an inference value which is different than the training value of the temperature parameter used during training.