Patent Description:
Many data processing tasks involve converting an ordered sequence of inputs into an ordered sequence of outputs. For example, machine translation systems translate an input sequence of words in one language into a sequence of words in another language. As another example, pronunciation systems convert an input sequence of graphemes into a target sequence of phonemes.

Some systems use auto-regressive sequence models based on deep neural networks to perform a sequence processing task.

Deep neural networks include multiple hidden layers in addition to an input layer and an output layer. The output of each hidden layer is generally used as input to a next layer in the network, i.e., the next hidden layer or the output layer.

Some deep neural networks are recurrent neural networks. A recurrent neural network is a neural network that receives an input sequence and generates an output sequence from the input sequence. In particular, a recurrent neural network can use some or all of the internal state of the network from a previous time step in computing an output at a current time step.

Another type of deep neural network architecture is deep convolutional neural network, of which Wavenet is an example. In particular, convolutional neural networks having a Wavenet architecture auto-regressively generate outputs by repeatedly adding a new output to the output sequence by processing the already-generated output sequence through multiple blocks of masked convolutional layers.

A different type of deep neural network architecture is the Transformer architecture. The Transformer architecture includes an encoder neural network that repeatedly applies self-attention over the input sequence to generate encoded representations of the inputs in the input sequence and a decoder neural network that autoregressively generates the output sequence by applying attention over the encoded representations of the input sequence and masked self-attention over the already-generated output sequence.

<NPL> relates to model abstractive text summarization using Attentional Encoder-Decoder Recurrent Neural Networks.

This specification describes a system implemented as computer programs on one or more computers in one or more locations that generates an output sequence that includes a respective output at each of multiple positions in an output order from an input sequence that includes a respective input at each of multiple positions in an input order, i.e., transduces the input sequence into the output sequence.

Auto-regressive models, e.g., Wavenet models or Transformers models, have been shown to achieve high-quality performance on a variety of output generation tasks, e.g., speech recognition, machine translation, image generation, and so on. However, auto-regressive models require a new output to be added to the end of the current output sequence at each of multiple time steps, with the new output being conditioned on the current output sequence. Generating outputs sequentially in this manner results in long inference times and significant computational resource consumption, particularly when output sequences being generated are long. The described techniques, on the other hand, only generate a shorter (for example, four or eight times shorter) sequence of latent variables auto-regressively and then generate the output sequence in parallel from the sequence of latent variables. Therefore, decoding time (i.e., the time required to generate an output sequence) and resource consumption are drastically reduced relative to conventional auto-regressive models. Moreover, the quality of the generated output sequence remains high. Thus, the described techniques allow for generation of a high quality output sequence in less time and while consuming fewer computational resources than conventional approaches.

This specification describes a system implemented as computer programs on one or more computers in one or more locations that generates a predicted target sequence that includes a respective output at each of multiple positions in an output order from an input sequence that includes a respective input at each of multiple positions in an input order, i.e., transduces the input sequence into the predicted target sequence.

For example, the system may be a neural machine translation system. That is, if the input sequence is a sequence of words in an original language, e.g., a sentence or phrase, the target sequence may be a translation of the input sequence into a target language, i.e., a sequence of words in the target language that represents the sequence of words in the original language.

As another example, the system may be a speech recognition system. That is, if the input sequence is a sequence of audio data representing a spoken utterance, the target sequence may be a sequence of graphemes, characters, or words that represents the utterance, i.e., is a transcription of the input sequence.

As another example, the system may be a natural language processing system. For example, if the input sequence is a sequence of words in an original language, e.g., a sentence or phrase, the target sequence may be a summary of the input sequence in the original language, i.e., a sequence that has fewer words than the input sequence but that retains the essential meaning of the input sequence. As another example, if the input sequence is a sequence of words that form a question, the target sequence can be a sequence of words that form an answer to the question.

As another example, the system may be part of a computer-assisted medical diagnosis system. For example, the input sequence can be a sequence of data from an electronic medical record and the target sequence can be a sequence of predicted treatments.

As another example, the system may be part of an image processing system. For example, the input sequence can be an image, i.e., a sequence of color values from the image, and the output can be a sequence of text that describes the image. As another example, the input sequence can be a sequence of text or a different context and the output sequence can be an image that describes the context.

As another example, the system may be an image generation system that generates images conditioned on a particular type of input, e.g., a smaller image, an object category, or a natural language text sequence. In these examples, the system may receive a representation of the image as a sequence and then generate the output image as a sequence of color values, i.e., of color channel values for the pixels of the output image, or as a two-dimensional structure of color values.

As another example, the system may be part of an extractive summarization system. In particular, the input sequence can be text from multiple input documents and, optionally, a topic of the documents, and the output sequence can be a text summary of the input documents.

<FIG> shows an example neural network system <NUM>. The neural network system <NUM> is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented.

The neural network system <NUM> receives an input sequence <NUM> and processes the input sequence <NUM> to transduce the input sequence <NUM> into an output sequence <NUM>, i.e., a predicted target sequence for the input sequence <NUM>.

The input sequence <NUM> has a respective input token at each of multiple input positions in an input order and the output sequence <NUM> has a respective output token at each of multiple output positions in an output order. That is, the input sequence <NUM> has multiple inputs arranged according to an input order and the output sequence <NUM> has multiple outputs arranged according to an output order.

As described above, the neural network system <NUM> can perform any of a variety of tasks that require processing sequential inputs to generate sequential outputs.

The neural network system <NUM> includes a latent prediction model <NUM> and a parallel decoder model <NUM>.

The latent prediction model <NUM> is configured to receive the input sequence and to process the input sequence to predict a sequence of discrete latent variables that encodes a target sequence for the input sequence, i.e., to predict a sequence of discrete latent variables that encodes the output sequence that should be generated by the system <NUM> for the input sequence. In other words, the latent prediction model <NUM> processes the input sequence x to generate a predicted sequence lp(x) that is a prediction of a sequence of discrete latent variables that would encode the target sequence.

The sequence of latent variables is referred to as a sequence of discrete latent variables because each latent variable in the sequence is discrete, i.e., selected from a discrete set of possible latent variables that has a fixed number of latent variables, rather than continuous.

As will be described in more detail below, the latent prediction model <NUM> has been trained to generate predicted sequences that match sequences that would be generated by an autoencoder function <NUM> that has access to the known target output sequence for the input sequence. As part of generating the sequence, the autoencoder function <NUM> selects each latent variable from a discrete set of latent variables.

The latent prediction model <NUM> is configured to autoregressively predict the latent variables in the predicted sequence. That is, the latent prediction model <NUM> generates each latent variable in the sequence one after the other, i.e., with each latent variable being conditioned on the latent variables in the sequence that have already been generated. In particular, the latent prediction model <NUM> can be a model that is based on multiple self-attention layers, e.g., a model that has the Transformer architecture.

Although the latent prediction model <NUM> autoregressively generates the sequence, the sequence of latent variables is shorter than the output sequence that will be generated. For example, the sequence of latent variables can include m latent variables while the output sequence can include n outputs, with m being less than n. For example, m can be <NUM>/<NUM>, <NUM>/<NUM>, or <NUM>/<NUM> of n. Because the sequence of latent variables is shorter, the latent prediction model <NUM> can generate the sequence of latent variables significantly faster than a conventional autoregressive model could generate the output sequence.

The parallel decoder model <NUM> generates the output sequence <NUM> from the predicted sequence of latent variables and the input sequence <NUM>. In particular, the parallel decoder model is a parallel model that generates the predicted target sequence ad(l,x) from the predicted sequence of latent variables lp(x) and the input sequence x. Because the parallel decoder model <NUM> is a parallel model, the parallel decoder model <NUM> generates all of the outputs in the predicted target sequence in one pass and independently of one another. Thus, the parallel decoder model <NUM> can extend the predicted latent sequence to a sequence that includes n outputs in minimal time and the generation of the predicted latent sequence does not significantly impact the latency of the generation of the predicted target sequence.

The parallel decoder model <NUM> can be implemented as a deep neural network that includes multiple steps of layers that each double the length of the input sequence to the step. Once the steps of layers have generated a sequence that is the same length as the predicted target sequence, the same length sequence can be processed by a self-attention decoder that has a Transformer architecture to generate the output sequence <NUM>.

Each of the multiple steps of layers can include a residual block of convolutional layers that receives an input sequence for the step and generates an alternative input sequence that has the same length as the input sequence for the step. Each step can also include an encoder-decoder attention layer with dot product attention that attends over the input sequence x at each position in the alternative input sequence. Optionally, this encoder-decoder attention layer can be followed by a residual connection. The output of the encoder-decoder attention layer (or the output of the residual connection) can be processed by an up-convolution that doubles the internal dimension of the sequence and then a reshape operation that reshapes the sequence to have twice the length of the input to the step of layers. The parallel decoder model <NUM> repeatedly doubles the length of the predicted latent variable sequence until the model has generated a sequence that is the same length as the output sequence using the multiple steps of layers.

Thus, the parallel decoder model <NUM> generates the output sequence in parallel conditioned on the predicted latent variable sequence and the input sequence.

As compared to conventional auto-regressive models, instead of directly auto-regressively generating the entire output sequence, the described system only auto-regressively generates a shorter sequence of discrete variables and then, in parallel, generates the output sequence from the discrete latent variable sequence.

In order to configure the latent prediction model <NUM> and the parallel decoder model <NUM> to generate accurate output sequences, a training engine <NUM> trains the models <NUM> and <NUM> to determine trained values of the parameters of the models <NUM> and <NUM>. In particular, the training engine <NUM> trains the latent prediction model <NUM> and the parallel decoder model <NUM> jointly with an autoencoder function <NUM> on training data that includes multiple input-output pairs. Each pair includes an input sequence and a known target sequence that should be generated for the input sequence. Thus, at training time, the system <NUM> has access to a known target sequence <NUM> for the input sequence <NUM>.

The autoencoder function <NUM> is configured to process the known target sequence <NUM> and the input sequence <NUM> to generate a sequence of discrete latent variables that encodes the the target sequence <NUM>, i.e., a sequence of discrete latent variables from which the target sequence <NUM> can be accurately reconstructed.

In particular, the autoencoder function <NUM> can be implemented as a deep neural network that includes multiple convolutional layers and one or more attention layers that attend to the input sequence x as described above. For example, the autoencoder function <NUM> can be implemented as a stack of residual convolutions, i.e., as a residual block of convolutional layers, followed by an attention layer attending to x and a stack of strided convolutions. Thus, these layers generate an initial encoder output that includes m initial encoder outputs.

The autoencoder function <NUM> also includes a discretization bottleneck that maps each of the initial encoder outputs to a discrete latent variable from a set of discrete latent variables to generate the sequence of discrete latent variables. The latent variables in the set of discrete latent variables also adjusted by the training engine <NUM> during the training.

The autoencoder function <NUM> can use any of a variety of discretization bottlenecks to perform the mapping.

One example of a discretization bottleneck is the Gumbel-Softmax discretization function. Such a bottleneck and ways to adjust the latent variables during training when this bottleneck is employed are described in <NPL>.

Another example of a discretization bottleneck is the Improved Semantic Hashing bottleneck. Such a bottleneck and ways to adjust the latent variables during training when this bottleneck is employed are described in <NPL>.

Another example of a discretization bottleneck is the Vector Quantized - Variational Autoencoder (VQ-VAE) bottleneck. A VQ-VAE bottleneck maintains a set of latent variables, and for each initial encoder output, selects the closest latent variable in the set as the discrete latent variable corresponding to the initial encoder output. Such a bottleneck and ways to adjust the latent variables during training when this bottleneck is employed are described in van den <NPL>.

While any of these bottlenecks may be employed, some of these approaches may not perform as well when the set of discrete latent variables K is large.

To account for this, the autoencoder may employ either Projected Vector Quantization (DVQ) bottleneck or a Sliced Vector Quantization (SVQ) bottleneck.

In SVQ, the discretization bottleneck divides each initial encoder output into nd smaller slices. The discretization bottleneck maintains a separate subset of discrete latent variables slices for each of the slices and selects, for each slice, the discrete latent variable slice from the corresponding subset that is closest to the slice. The system then generates the discrete latent variable corresponding to the encoder output as a concatenation of the selected latent variable slices. The latent variable slices are updated during training in the same manner as the latent variables in the VQ-VAE bottleneck.

In PVQ, the discretization bottleneck maintains a fixed set of nd randomly initialized projection functions. Each projection function projects an encoder output of length D into a slice of dimension D/nd. For each initial encoder output, the discretization bottleneck applies each of the projections to project the initial encoder output into nd slices. The discretization bottleneck maintains a separate subset of discrete latent variables slices for each of the slices and selects, for each slice, the discrete latent variable slice from the corresponding subset that is closest to the slice. The system then generates the discrete latent variable corresponding to the encoder output as a concatenation of the selected latent variable slices. The latent variable slices are updated during training in the same manner as the latent variables in the VQ-VAE bottleneck.

Thus, all of the discretization bottlenecks map each initial encoder output in the initial encoder sequence to a discrete latent variable from a discrete set of latent variables.

While training the models on a given input-output pair, the system <NUM> also processes the latent variable sequence generated by the autoencoder function <NUM> using the parallel decoder model <NUM> to generate a reconstructed target sequence ad(ae(y,x),x). The training engine <NUM> then trains the models <NUM>, <NUM>, and <NUM> using ad(ae(y,x),x), lp(x), ae(y,x) and the known target sequence y.

Thus, during training, the functions ae(y, x) and ad(l, x) together form an autoencoder of the targets y that has additional access to the input sequence x.

Training the models <NUM>, <NUM>, and <NUM> jointly will be discussed in more detail below with reference to <FIG>.

<FIG> is a diagram <NUM> showing the dependence structure of the discrete latent variables and the predicted outputs. In particular, in the graph representation of the diagram <NUM>, the input sequence x is represented as a single node <NUM>, the predicted outputs y<NUM> through yn generated by the parallel decoder model are each represented as respective nodes <NUM>, and the discrete latent variables l<NUM> through lm generated by the latent prediction model are represented as respective nodes <NUM>. In the diagram <NUM>, an arrow is drawn from node a to another node b if the probability of the node a depends on the node b.

As can be seen in the diagram <NUM>, each of the latent variables l<NUM> through lm depend on the input sequence x. Additionally, each latent variable li depends on the generated latent variables with index less than i in the sequence, i.e., the latent variables that precede the latent variable in the sequence. Thus, this represents that the latent prediction model generates the sequence of latent variables auto-regressively conditioned on the input sequence.

Additionally, each of the predicted outputs y<NUM> through yn depends on all of the latent variables l<NUM> through lm and on the input sequence x. However, none of the predicted outputs depends on any other predicted output. Thus, this represents that the parallel decoder model generates all of the predicted outputs in parallel and independently of one another while conditioned on the latent variables.

Thus, the diagram <NUM> illustrates the increased parallelism and faster decoding achieved by the described systems relative to conventional auto-regressive models. In particular, conventional auto-regressive models generate the entire output sequence auto-regressively, i.e., with each output in the output sequence being dependent on all of the outputs preceding the output in the output sequence. The described system, on the other hand, only auto-regressively generates a much shorter latent sequence and then generate the entire output sequence in parallel conditioned on the shorter latent sequence. This allows for an order of magnitude decrease in decoding time relative to comparably or even worse performing auto-regressive models.

<FIG> is a flow diagram of an example process for generating an output sequence from an input sequence. For convenience, the process <NUM> will be described as being performed by a system of one or more computers located in one or more locations. For example, a neural network system, e.g., the neural network system <NUM> of <FIG>, appropriately programmed in accordance with this specification, can perform the process <NUM>.

The system receives an input sequence x that includes inputs x<NUM> through xk (step <NUM>).

The system processes the input sequence using a latent prediction model to predict a sequence of discrete latent variables that encodes a target sequence for the input sequence but is shorter than the target sequence (step <NUM>). In other words, the predicted sequence of discrete latent variables has m latent variables, while the target sequence has n outputs, with n being greater than m. In particular, the latent prediction model is configured to autoregressively predict the sequence of discrete latent variable conditioned on the input sequence.

The system processes the predicted sequence of discrete latent variables and the input sequence using a parallel decoding model to predict the target sequence for the input sequence (step <NUM>). The parallel decoding model is configured to generate all of the outputs in the target sequence in parallel and independently of one another conditioned on the sequence of discrete latent variables. That is, the parallel decoding model extends the sequence of m latent variables to a sequence that includes n outputs in a single pass through the parallel decoding model.

In some implementations, the system generates multiple candidate predicted target sequences using the parallel decoding model and then re-scores the candidate predicted target sequences using a higher-powered auto-regressive decoding model. The system can then select the highest scoring candidate predicted translation as the final predicted target sequence. Such a technique is described in more detail in <NPL>.

<FIG> is a flow diagram of an example process for training the latent prediction model and the parallel decoder model. For convenience, the process <NUM> will be described as being performed by a system of one or more computers located in one or more locations. For example, a neural network system, e.g., the neural network system <NUM> of <FIG>, appropriately programmed in accordance with this specification, can perform the process <NUM>.

The system receives an input-output pair that includes an input sequence x that includes inputs x<NUM> through xk and a target sequence y that includes outputs y<NUM> through yn (step <NUM>).

The system processes the target sequence and the input sequence using the autoencoder function to generate a sequence of discrete latent variables ae(y,x) (step <NUM>).

The system processes the input sequence using a latent prediction model to generate a predicted sequence of discrete latent variables lp(x) that represents a target sequence for the input sequence but is shorter than the target sequence (step <NUM>). In particular, the latent prediction model is configured to autoregressively predict the sequence of discrete latent variable conditioned on the input sequence.

The system processes the sequence of discrete latent variables ae(y,x) and the input sequence x using a parallel decoding model to generate a predicted target sequence ad(ae(y,x)) for the input sequence (step <NUM>). The parallel decoding model is configured to generate all of the outputs in the target sequence in parallel and independently of one another conditioned on the sequence of discrete latent variables.

As described above, in some implementations the parallel decoding model includes a first subnetwork that decompresses the latent variables to generate a sequence that is the same length as the target sequence and a self-attention decoder that generates the predicted target sequence from the same length sequence. In some of these implementations, the system provides the true target sequence y as input to the self-attention decoder at the initial stages of training, i.e., for a threshold number of initial input-output pairs processed during the training. This ensures that, after this "pre-training" is completed, the self-attention decoder has reasonable gradients that can be backpropagated through the remainder of the parallel decoding model and into the auto encoder function and the discretization bottleneck.

The system determines a gradient with respect to the parameters of the three models of a loss function that depends on (i) an autoencoder reconstruction loss coming from comparing the predicted target sequence ad(ae(y,x)) to the target sequence y and (ii) a latent prediction loss coming from comparing the sequence of discrete latent variables ae(y,x) to the predicted sequence of latent variables lp(x) (step <NUM>). For example, the loss function can be a sum or weighted sum of the autoencoder reconstruction loss and the latent prediction loss. The system can compute the gradient with respect to each of the parameters through backpropagation, i.e., by backpropagating the gradients through each of the three models. The function used to compare the sequences of latent variables can be, e.g., a distance measure while the function used to compare the output sequence can depend on the types of outputs that the system is configured to generate. For example, when the outputs are text sequences, the comparison function can be a perplexity function. The system also computes an update to the latent variables in the set of discrete latent variables using a technique that is appropriate for the discretization bottleneck that is employed.

The system can perform the process <NUM> for each of multiple input-output pairs in a mini batch of pairs to determine a respective gradient for each of the pairs. The system can then combine, e.g., average or sum, the gradients, determine an update to the current values of the parameters of the three models from the combined gradients in accordance with the update rule of the gradient-based technique that is being used to train the technique, e.g., an rmsProp update rule, an SGD update rule, or an Adam update rule, and add the update to the current values of the parameters to determine updated values of the parameters.

The system can repeatedly update the parameter values and the latent variables as described above to train the models to effectively generate predicted target outputs for received inputs.

However, embodiments in which the data processing apparatus does not comprise multiple processors do not fall within the scope of the claims.

Claim 1:
A method performed by an apparatus, the apparatus comprising multiple processors, of generating an output sequence (<NUM>;<NUM>) comprising a plurality of outputs from an input sequence (<NUM>;<NUM>) comprising a plurality of inputs, the method comprising:
receiving the input sequence;
processing the input sequence using a latent prediction model (<NUM>) configured to autoregressively predict a sequence of discrete latent variables (<NUM>) that is shorter than the output sequence and that encodes the output sequence, wherein each discrete latent variable in the sequence is selected from a discrete set of latent variables; and
processing, the input sequence and the predicted sequence of discrete latent variables using a parallel decoder model (<NUM>) configured to generate the outputs in the output sequence in parallel from the input sequence and the predicted sequence of discrete latent variables, wherein each one of the outputs of the output sequence is generated independently from the other outputs of the output sequence.