Patent ID: 11868888
Assignee: GOOGLE LLC
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 10:
11. The system of claim 10, wherein training the language model neural network comprises, for a particular input document:
obtaining a sequence of input text inputs that comprises text inputs from the particular input document arranged in the order in which the text inputs appear in the particular input document;
obtaining a held-out text input that appears in a corresponding position in the particular input document relative to the text inputs in the sequence of input text inputs;
generating an initial output by processing each input text input in the sequence of input text inputs using the language model neural network in accordance with current values of the parameters of the initial neural network layers; and
processing the initial output using the language model output layer to generate a set of text input scores, wherein the set of text input scores comprises a respective score for each of a plurality of vocabulary text inputs that represents a likelihood that the vocabulary text input is the text input that appears in the corresponding position in the particular input document relative to the text inputs in the sequence of input text inputs.