Patent ID: 11908457
Assignee: QUALCOMM INCORPORATED
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 16:
17. An apparatus for operating a neural network, the apparatus comprising:
means for receiving an input sequence at an encoder;
means for encoding the input sequence to produce a set of hidden representations of time frame t;
means for providing the set of hidden representations to each of a plurality of attention heads of the neural network, each of the plurality of attention heads configured to focus on a different subset of the set of hidden representations;
means for calculating corresponding attention weights for each of the different subsets of the set of hidden representations of time frame tin each attention head of the plurality of attention-heads based on a corresponding score, each score being computed based on a nonlinear scoring function of the set of hidden representations of time frame t and learnable parameters;
means for calculating a context vector for each attention-head based on the attention weights of time frame t and the set of hidden representations of time frame t, each context vector corresponding to a portion of the input sequence; and
means for outputting an inference based on the context vectors.