Patent ID: 11886998
Assignee: GOOGLE LLC
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 16:
17. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training a self-attention decoder neural network for processing an input sequence comprising a plurality of input tokens to generate an output sequence comprising a plurality of output tokens, wherein the self-attention decoder neural network comprises a plurality of neural network layers that include a plurality of masked self-attention neural network layers, the operations comprising, for each of a plurality of iterations:
obtaining one or more training examples, wherein each training example comprises a respective training input sequence and a respective training output sequence;
for each respective training example, processing using the self-attention decoder neural network according to current values of parameters of the self-attention decoder neural network, a combined sequence comprising (i) tokens in the respective training input sequence and (ii) tokens in the respective training output sequence to generate a respective prediction output that comprises a time step output for each of the tokens in the respective training output sequence; and
updating the current values of the parameters of the self-attention decoder neural network based on (i) the respective training output sequences and (ii) the respective prediction outputs.