Patent ID: 11893483
Assignee: GOOGLE LLC
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 7:
8. The system of claim 7, wherein each self-attention sub-layer comprises a plurality of self-attention layers, and wherein each self-attention layer is configured to:
apply a learned query linear transformation to the input at each particular input position of the plurality of preceding input positions in the ordering of the input positions to generate a respective query for each particular input position,
apply a learned key linear transformation to each input at each particular input position of the plurality of preceding input positions in the ordering of the input positions to generate a respective key for each particular input position,
apply a learned value linear transformation to each input at each particular input position of the plurality of preceding input positions in the ordering of the input positions to generate a respective key for each particular input position, and
for each of the particular input positions of the plurality of preceding input positions in the ordering of the input positions,
determine a respective input-position specific weight for the particular input position by applying a comparison function between the query for the particular input position and the keys, and
determine an initial attention output for the particular input position by determining a weighted sum of the values weighted by the corresponding input-position specific weights for the particular input position.