Patent ID: 11914969
Assignee: GOOGLE LLC
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 0:
1. A computer-implemented method to perform inference with a machine-learned language encoder model, the method comprising:
processing an input with the machine-learned language encoder model to generate an output;
wherein the machine-learned language encoder model has been previously trained for one or more training iterations, each of the one or more training iterations comprising:
obtaining an original language input that comprises a plurality of original input tokens;
selecting one or more of the plurality of original input tokens to serve as one or more masked tokens;
generating one or more replacement tokens, wherein the one or more replacement tokens comprise alternative natural language tokens;
respectively replacing the one or more masked tokens in the original language input with the one or more replacement tokens to form a noised language input that comprises a plurality of updated input tokens, the plurality of updated input tokens comprising a mixture of the one or more replacement tokens and the original input tokens that were not selected to serve as masked tokens;
processing the noised language input with the machine-learned language encoder model to produce a respective prediction for each updated input token included in the plurality of updated input tokens, wherein the prediction produced by the machine-learned language encoder model for each updated input token predicts whether such updated input token is one of the original input tokens or one of the replacement input tokens; and
training the machine-learned language encoder model based at least in part on a loss function that evaluates the plurality of predictions produced by the machine-learned language encoder model.