Patent ID: 11941356
Assignee: SALESFORCE, INC.
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 17:
18. A non-transitory processor-readable medium storing processor-executable instructions for multi-scale discriminative pre-training with densely connected transformer network, the instructions being executed by one or more processors to perform operations comprising:
receiving, at a first transformer network, a masked input sequence of tokens;
generating, by the first transformer network, alternative tokens that replace the masked-out tokens;
sending, from the first transformer network to a second transformer network, a reconstructed sequence of tokens with masked-out tokens replaced by the generated alternative tokens;
predicting, by the second transformer network, whether a subset of tokens from the reconstructed sequence contains a replaced token;
generating, by the second transformer network, a first probability predicting whether the subset of tokens having a pre-defined length from the reconstructed sequence contains a replaced token;
computing a discriminator loss based at least in part on the generated probability; and
updating the second transformer network based at least on the computed discriminator loss.