Patent ID: 11868888
Assignee: GOOGLE LLC
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 7:
8. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations for training a first neural network to perform a document processing task, wherein the first neural network comprises one or more initial neural network layers and a first output layer, wherein the one or more initial neural network layers have parameters, and wherein the operations comprise:
training a language model neural network to predict missing text inputs in text sequences that each include a respective plurality of text inputs, wherein the language model neural network comprises the one or more initial neural network layers and a language model output layer, and wherein training the language model neural network comprises determining pre-trained values of the parameters of the one or more initial neural network layers from initial values of the parameters of the one or more initial neural network layers; and
training the first neural network on a plurality of training documents to determine trained values of the parameters of the one or more initial neural network layers from the pre-trained values of the parameters of the one or more initial neural network layers.