Patent ID: 11928600
Assignee: SALESFORCE, INC.
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 0:
1. A method for sequence-to-sequence prediction using a neural network model comprising an encoder and a decoder, wherein the encoder comprises a set of branched attention encoder layers arranged sequentially and the decoder comprises a set of branched attention decoder layers, the method comprising:
for each branched attention encoder layer of the set of branched attention encoder layers:
generating, by the respective branched attention encoder layer, a respective layer encoded representation for a corresponding branched attention decoder layer of the decoder based on an input of an input sequence, wherein the respective branched attention encoder layer includes a plurality of branches arranged in parallel, and for each branch of the plurality of branches, the branch includes a respective attention sublayer and a respective scaling sublayer, wherein the generating includes:
determining, at the respective attention sublayer of a respective branch, a respective learned scaling parameter depending on one or more other learned scaling parameters from one or more other branches; and
applying, at the respective scaling sublayer of the respective branch, a respective attention to the input of the input sequence in parallel to other branches, based on scaling a respective intermediate representation of the respective branch by the respective learned scaling parameter; and

predicting, by the decoder, an output sequence based on a set of respective layer encoded representations sequentially received from the set of branched attention encoder layers, wherein the neural network model includes a plurality of model parameters learned according to a machine learning process.