Patent ID: 11948066
Assignee: DEEPMIND TECHNOLOGIES LIMITED
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 8:
9. A method of performing a sequence processing task by generating a neural network output from an input sequence of data elements comprising a plurality of data elements, wherein the neural network output which characterizes the input sequence, the method comprising:
for each of the plurality of data elements, providing a current input sequence that comprises the data element and the data elements preceding the data element in the input sequence to a convolutional subnetwork comprising a plurality of convolutional neural network layers, wherein the convolutional subnetwork is configured to, for each of the plurality of data elements:
receive the current input sequence for the data element, and
process the current input sequence, in accordance with trained values of a set of neural network parameters of the convolutional subnetwork, to generate an alternative representation for the data element, wherein the trained values of the set of neural network parameters of the convolutional subnetwork have been determined by a machine learning training technique, wherein the convolutional neural network layers are causal convolutional neural network layers and the alternative representation for the data element does not depend on any data elements that follow the data element in the input sequence; and

providing the alternative representations to an output subnetwork, wherein the output subnetwork is configured to receive the alternative representations and to process the alternative representations to generate the neural network output characterizing the input sequence, wherein the neural network output comprises a plurality of sets of scores, wherein each set of scores includes a respective score for each of a plurality of possible outputs; and
selecting outputs in accordance with the sets of scores to generate an output sequence comprising a plurality of outputs;
wherein:
the sequence processing task is a speech recognition task, the input sequence is an audio data input sequence of audio data elements, and the output sequence is a sequence of graphemes that is a grapheme representation of words spoken in the audio data input sequence; or
the sequence processing task is a speech recognition task, the input sequence is an audio data input sequence of audio data elements, and the output sequence is a sequence of phonemes that is a phoneme representation of words spoken in the audio data input sequence; or
the sequence processing task is a speech recognition task, the input sequence is an audio data input sequence of audio data elements, and the output sequence is a sequence of words in a first natural language that represents worked spoken in the audio data input sequence; or
the sequence processing task is a language modeling task, the input sequence is a sequence of phonemes, and the output sequence is a sequence of words in a natural language that represents the input sequence; or
the sequence processing task is a language modeling task, the input sequence is a sequence of graphemes, and the output sequence is a sequence of words in a natural language that represents the input sequence; or
the sequence processing task is a machine translation task, the input sequence is a sequence of words in a first natural language, and the output sequence is a sequence of words in a second natural language that is a translation of the input sequence into the second natural language; or
the sequence processing task is a natural language processing task, the input sequence is a sequence of words in a natural language, and the output sequence is a sequence that classifies the words in in the input sequence according to a kind of output required by the natural language processing task; or
the sequence processing task is a compression task, the input sequence is a sequence of data elements drawn from an input sample, and the output sequence is a compressed or encoded sample; or
the sequence processing task is a medical analysis task, the input sequence is a sequence of health data elements for a particular patient, and the output sequence includes data that characterizes health of the patient or predicts future health of the patient.