Patent ID: 11908457
Assignee: QUALCOMM INCORPORATED
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 24:
25. A non-transitory computer readable medium having encoded thereon program code for operating a neural network, the program code being executed by a processor and comprising:
program code to receive an input sequence at an encoder;
program code to separate the input sequence into sequence parts;
program code to encode the sequence parts to produce a set of hidden representations of time frame t;
program code to provide the set of hidden representations to each of a plurality of attention heads of the neural network, each of the plurality of attention heads configured to focus on a different subset of the set of hidden representations;
program code to calculate corresponding attention weights for each of the different subsets of the set of hidden representations of time frame tin each attention head of the plurality of attention-heads based on a corresponding score, each score being computed based on a nonlinear scoring function of the set of hidden representations of time frame t and learnable parameters;
program code to calculate a context vector for each attention-head based on the attention weights of time frame t and the set of hidden representations of time frame t, each context vector corresponding to a portion of the input sequence; and
program code to output an inference based on the context vectors.