Patent ID: 11908457
Assignee: QUALCOMM INCORPORATED
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 0:
1. A processor-implemented method for operating a neural network, the method comprising:
receiving an input sequence at an encoder;
encoding the input sequence to produce a set of hidden representations of time frame t;
providing the set of hidden representations to each of a plurality of attention heads of the neural network, each of the plurality of attention heads configured to focus on a different subset of the set of hidden representations;
calculating corresponding attention weights for each of the different subsets of the set of hidden representations of time frame tin each attention head of the plurality of attention-heads based on a corresponding score, each score being computed based on a nonlinear scoring function of the set of hidden representations of time frame t and learnable parameters;
calculating a context vector for each attention-head based on the attention weights of time frame t and the set of hidden representations of time frame t, each context vector corresponding to a portion of the input sequence; and
outputting an inference based on the context vectors.