Patent ID: 11954448
Assignee: MICROSOFT TECHNOLOGY LICENSING, LLC
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 15:
16. A non-transitory machine-readable medium storing a program executable by at least one processing unit of a computer system, the program comprising sets of instructions for:
receiving a set of input data for training a transformer model, the set of input data comprising a plurality of tokens;
selecting an offset value based on a maximum sequence length that the transformer model is configured to process and a token length of the plurality of tokens;
based on the offset value, determining a plurality of successive position values for the plurality of tokens, wherein each position value in the plurality of successive position values represents a position of a token in the plurality of tokens relative to other tokens in the plurality of tokens;
generating a set of training data to comprise the plurality of tokens and the plurality of successive position values; and
training the transformer model using the set of training data.