Patent ID: 11893469
Assignee: MICROSOFT TECHNOLOGY LICENSING, LLC
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 19:
20. A non-transitory machine-readable medium storing a program executable by at least one processing unit of a computer system, the program comprising sets of instructions for:
receive a set of data for training a transformer model, the set of data comprising a sequence of tokens and a set of position values, wherein each position value in the set of position values represents a position of a token in the sequence of tokens relative to other tokens in the sequence of tokens;
selecting a subset of the sequence of tokens in the set of data;
replacing each token in the subset of the sequence of tokens with a same first defined value to form a modified version of the sequence of tokens comprising a set of same first defined values;
selecting a subset of the set of position values in the set of data;
replacing each position value in the subset of the set of position values with a same second defined value to form a modified version of the set of position values comprising a set of same second defined values, wherein the same second defined value represents a masked position value that is used when training the transformer model; and
training the transformer model using the set of data that comprises the modified version of the sequence of tokens and the modified version of the set of position values.