Patent ID: 11914969
Assignee: GOOGLE LLC
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 14:
15. A computing system for fine-tuning a machine-learned language encoder model, the computing system comprising:
one or more processors; and
one or more non-transitory computer-readable media that store:
a machine-learned language encoder model, the machine-learned language encoder model having been pre-trained by performance of pre-training operations, the pre-training operations comprising:
obtaining an original language input that comprises a plurality of original input tokens;
selecting one or more of the plurality of original input tokens to serve as one or more masked tokens;
generating one or more replacement tokens, wherein the one or more replacement tokens comprise alternative natural language tokens;
respectively replacing the one or more masked tokens in the original language input with the one or more replacement tokens to form a noised language input that comprises a plurality of updated input tokens, the plurality of updated input tokens comprising a mixture of the one or more replacement tokens and the original input tokens that were not selected to serve as masked tokens;
processing the noised language input with the machine-learned language encoder model to produce a respective prediction for each updated input token included in the plurality of updated input tokens, wherein the prediction produced by the machine-learned language encoder model for each updated input token predicts whether such updated input token is one of the original input tokens or one of the replacement input tokens; and
training the machine-learned language encoder model based at least in part on a loss function that evaluates the plurality of predictions produced by the machine-learned language encoder model; and

instructions that, when executed by the one or more processors, cause the computing system to perform one or more fine-tuning training iterations in which the machine-learned language encoder model is trained to perform a language task.