Patent ID: 11922281
Assignee: GOOGLE LLC
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 11:
12. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one more computers to perform operations for training a student machine learning model having a plurality of student parameters to perform a plurality of machine learning tasks, wherein the student machine learning model is configured to receive a model input and to process the model input in accordance with the student parameters to generate an output comprising a respective student output for each of the plurality of machine learning tasks, the operations comprising:
training the student machine learning model on training data comprising a plurality of first training inputs and, for each first training input, a respective ground truth output for the first training input for each of the plurality of machine learning tasks, the training comprising, for each first training input:
for each of the plurality of machine learning tasks, processing the training input using a trained teacher machine learning model for the machine learning task to generate a teacher output for the machine learning task, wherein the trained teacher machine learning model for the machine learning task is a single-task model that has already been trained to perform the machine learning task;
for each of the plurality of machine learning tasks, determining a weighted combination of the teacher output for the machine learning task and the ground truth output for the first training input in accordance with a weight value that defines a weighting between teacher outputs and ground truth outputs;

processing the training input using the student machine learning model and in accordance with the student parameters to generate a respective student output for each of the plurality of machine learning tasks;
determining a gradient with respect to the student parameters of an objective function that measures, for each of the plurality of machine learning tasks, an error between the weighted combination for the machine learning task and the student output for the machine learning task; and
determining an update to the student parameters from the gradient.