Patent ID: 11941356
Assignee: SALESFORCE, INC.
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 9:
10. A method for multi-scale discriminative pre-training with densely connected transformer network, the method comprising:
receiving, at a first transformer network, a masked input sequence of tokens;
generating, by the first transformer network, alternative tokens that replace the masked-out tokens;
sending, from the first transformer network to a second transformer network, a reconstructed sequence of tokens with masked-out tokens replaced by the generated alternative tokens;
generating a first probability predicting whether a subset of tokens from the reconstructed sequence contains a replaced token;
computing a discriminator loss based at least in part on the generated probability;
updating the second transformer network based at least on the computed discriminator loss; and
during a training stage, updating the second transformer network by a weighted sum of a first loss computed based on first output probabilities of the first transformer network, and a second loss computed based on second output probabilities of the second transformer network.