Patent ID: 11914969
Assignee: GOOGLE LLC
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 17:
18. One or more non-transitory computer-readable media that store a machine-learned language encoder model, the machine-learned language encoder model having been trained by performance of operations, the operations comprising:
obtaining an original language input that comprises a plurality of original input tokens;
selecting one or more of the plurality of original input tokens to serve as one or more masked tokens;
generating one or more replacement tokens, wherein the one or more replacement tokens comprise alternative natural language tokens;
respectively replacing the one or more masked tokens in the original language input with the one or more replacement tokens to form a noised language input that comprises a plurality of updated input tokens, wherein the plurality of updated input tokens comprises the one or more replacement tokens and the plurality of original input tokens that were not selected to serve as masked tokens;
processing the noised language input with the machine-learned language encoder model to produce a respective prediction for each updated input token included in the plurality of updated input tokens, wherein the prediction produced by the machine-learned language encoder model for each updated input token predicts whether such updated input token is one of the original input tokens or one of the replacement input tokens; and
training the machine-learned language encoder model based at least in part on a loss function that evaluates the plurality of predictions produced by the machine-learned language encoder model.