Patent Description:
Some neural networks are recurrent neural networks. A recurrent neural network is a neural network that receives an input sequence and generates an output sequence from the input sequence. In particular, a recurrent neural network can use some or all of the internal state of the network from a previous time step in computing an output at a current time step. An example of a recurrent neural network is a Long Short-Term Memory (LSTM) neural network that includes one or more LSTM memory blocks. Each LSTM memory block can include one or more cells that each include an input gate, a forget gate, and an output gate that allow the cell to store previous states for the cell, e.g., for use in generating a current activation or to be provided to other components of the LSTM neural network. <NPL>, describes a computationally efficient approach to unconditional generation of audio with recurrent neural networks involving modules operating at different clock rates, the lowest module processing individual samples.

This specification describes a system implemented as computer programs on one or more computers in one or more locations that generates output examples using a recurrent neural network by generating a respective output value at each of multiple generation time steps. Each output value in the output example is an N-bit value. At each generation time step, the system generates the values of the first half of the N bits and then generates the values of the second half of the N bits conditioned on the values of the first half of the N bits. For example, the system can generate the values of the N/<NUM> most significant bits of an output value and then generate the values of the N/<NUM> least significant bits of the output value conditioned on the already-generated values of the most significant bits.

Thus there is described a method of generating an output example comprising a respective N-bit output value at each generation time step of a sequence of generation time steps. The method comprises, for each generation time step, processing a first recurrent input comprising the N-bit output value at the preceding generation time step in the sequence using a recurrent neural network and in accordance with a hidden state of the recurrent neural network to generate a first score distribution over possible values for a first half of the N bits in the output value at the generation time step, and selecting, using the first score distribution, values for the first half of the N bits of the output value. The method further comprises, for each generation time step, processing a second recurrent input comprising (i) the N-bit output value at the preceding generation time step in the sequence and (ii) the values for the first half of the N bits using the recurrent neural network and in accordance with the same hidden state to generate a second score distribution over possible values for a second half of the N bits in the output value at the generation time step, and selecting, using the second score distribution, values for the second half of the N bits of the output value.

As described further later, implementations of this method are adapted for implementation on a processing unit with limited computational ability and memory bandwidth, such as found in a mobile device. For example, implementations of the method use a computational architecture in which a processing unit is configured to split generating the N-bit output value into two halves, generating a first half of the N bits and then a second half of the N bits. By dividing the output space in this way, into two smaller output spaces rather than one large output space, the number of sequential matrix-vector product computations may be reduced by an order of magnitude, facilitating real-time implementation of the method.

The output value is conditioned on a respective conditioning input at each of the generation time steps. The first recurrent input and the second recurrent input may each also comprise the conditioning input for the generation time step. Generating the conditioning input at the generation time step comprises processing conditioning features using a conditioning neural network.

In implementations processing the first recurrent input may comprise processing the first recurrent input to generate a first half of an updated hidden state, and then processing the first half of the updated hidden state using one or more first output layers to generate the first score distribution. Processing the second recurrent input may comprise processing the second recurrent input to generate a second half of an updated hidden state, and processing the second half of the updated hidden state using the one or more second output layers to generate the second score distribution.

In particular implementations the one or more first output layers may be configured to apply a first weight matrix to the first half of the updated hidden state to generate a first projected updated hidden state. The method may then apply an element-wise non-linear activation function to the first projected updated hidden state to generate a first activation vector, apply a second weight matrix to the first activation vector to generate first logits, and then apply a softmax function to the first logits to generate the first score distribution. Similarly the one or more second output layers may be configured to apply a third weight matrix to the second half of the updated hidden state to generate a second projected updated hidden state. The method may then apply the element-wise non-linear activation function to the second projected updated hidden state to generate a second activation vector, apply a fourth weight matrix to the second activation vector to generate second logits, and apply the softmax function to the second logits to generate the second score distribution.

The recurrent neural network may include one or more gates. Processing the first recurrent input may then comprise determining a respective recurrent contribution for each gate by applying a recurrent weight matrix to the hidden state. The method may then further comprise, for each of the one or more gates, determining a first input contribution for the gate from the first recurrent input, determining a first gate vector for the gate from at least the recurrent contribution for the gate and the first input contribution for the gate, and generating the first half of the updated hidden state from the first gate vectors and the hidden state. Similarly processing the second recurrent input may comprise, for each of the one or more gates, determining a second input contribution for the gate from the second recurrent input, determining a second gate vector for the gate from at least the recurrent contribution for the gate and the second input contribution for the gate, and generating the second half of the updated hidden state from the second gate vectors and the hidden state.

In implementations processing the second recurrent input may comprise generating the second half of the updated hidden state without re-computing recurrent contributions for the gates from processing of the first recurrent input.

In implementations the recurrent neural network includes just a single recurrent layer. A single recurrent layer applied to a previous state of the recurrent neural network can provide a highly non-linear transformation of the context, which is represented in an already compressed form by the hidden state of the recurrent neural network. Thus this further facilitates deployment of the method on a processing unit with limited computational power.

In some implementations of the method the first half of the N bits are the most significant bits and the second half of the N bits are the least significant bits of the output example (data item). In some implementations values for the first and second halves of the N bits may be selected by sampling from, respectively, the first and second score distributions.

Implementations of the method allow loading parameters of the recurrent neural network into registers of the processing unit only once at the start of generating the output example. The parameters of the recurrent neural network may then persist in the registers throughout the generation of the output example. This can help to reduce the memory bandwidth used when implementing the method.

In some implementations the method operates on the processing unit of a mobile device such as a mobile phone.

There is also described a processing unit configured for generating an output example comprising a respective N-bit output value at each generation time step of a sequence of generation time steps. The processing unit is configured to split the generating of the N-bit output value into two halves, to generate a first half of the N bits and then a second half of the N bits. The processing unit may be configured to, for each generation time step, process a first recurrent input comprising the N-bit output value at the preceding generation time step in the sequence using a recurrent neural network and in accordance with a hidden state of the recurrent neural network to generate a first score distribution over possible values for a first half of the N bits in the output value at the generation time step, and then select, using the first score distribution, values for the first half of the N bits of the output value. The processing unit is further configured to, for each generation time step, process a second recurrent input comprising (i) the N-bit output value at the preceding generation time step in the sequence and (ii) the values for the first half of the N bits using the recurrent neural network and in accordance with the same hidden state to generate a second score distribution over possible values for a second half of the N bits in the output value at the generation time step, and then select, using the second score distribution, values for the second half of the N bits of the output value.

By first generating the values for the first half of the N bits, e.g., the most significant bits, and then generating the values for the second half of the N bits, e.g., the least significant bits, conditioned on the values of the first half, the system improves the accuracy of the outputs generated. Moreover, the network can achieve this improved accuracy while being computationally compact. In particular, because the neural network includes only a single recurrent layer (and output layers that have relatively low computational cost) to operate on a conditioning input, the network can generate high-quality outputs quicker than conventional approaches. This may allow the network to be used in environments where outputs need to be generated in real-time, e.g., when deployed on a mobile device or another user device.

Additionally, the system can generate two outputs per time step without needing to re-compute computationally expensive matrix multiplies, i.e., by only computing the recurrent contribution to the updated hidden state once, minimizing the additional computation and processing power required to generate the second output.

Moreover, by generating separate score distributions over the first half of the bits and second half of the bits, the system reduces the output space, allowing for efficient prediction of multi-bit values. As a particular example, when N is <NUM>, i.e., all of the values are <NUM>-bit values, the neural network only requires two relatively small output spaces (two score distributions of <NUM>^<NUM> scores each) instead of one large output space (with each time step needing to have a score distribution that has <NUM>^<NUM> scores), reducing the amount of computation required to effectively predict <NUM>-bit values.

The architecture of the network lends itself well to being optimized on a GPU or other special-purpose hardware, resulting in additional computational and time savings. For example, in implementations the network has only a single recurrent layer, and the size of the hidden state of the recurrent layer, i.e., the number of units in the hidden state, can be readily configured to optimize for the memory available on the special-purpose hardware.

This specification describes a system implemented as computer programs on one or more computers in one or more locations that generates output examples using a recurrent neural network. Each output example includes a respective output value at each of multiple generation time steps. The system generates the output example by conditioning the recurrent neural network on a conditioning input.

The system generates audio data and at each generation time step the output value may be e.g. a time-domain or frequency audio data value for defining an audio waveform. In particular, the system is part of a system that converts text to speech. Thus, the output example is a sequence of audio data, e.g., a waveform, that represents an utterance of a piece of text and the recurrent neural network is conditioned on a sequence of linguistic features of the piece of text. That is, at each generation time step, the conditioning input is a set of linguistic features and the output value is a value that defines the amplitude of the waveform. In some implementations, the audio value at each generation time step in the sequence is the amplitude of the audio waveform at the corresponding time, i.e., the output example is a raw audio waveform. In other implementations, the audio value at each generation time step in the sequence is a compressed or companded representation of the waveform at the corresponding time. For example, the audio value can be a µ-law transformed representation of the waveform. In other implementations the audio value at a time step may be for a STFT (short-time Fourier transform) representation of the audio waveform.

In particular, each output value in the output example is an N-bit value, e.g., a <NUM>-bit value, <NUM>-bit value, <NUM>-bit value, or <NUM>-bit value. That is, each output value in the output example <NUM> is represented as a sequence of N bits. For example, the output example can be an ordered, i.e., by time, collection of N-bit amplitude values or N-bit compressed or companded amplitude values.

<FIG> shows an example neural network system <NUM>. The neural network system <NUM> is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented.

The neural network system <NUM> generates an output example that includes multiple output values <NUM> by generating at least one output value <NUM> at each of multiple generation time steps. As described above, each output value is an N-bit value and is represented as a sequence of N bits. In the example of <FIG>, the output value <NUM> is an <NUM>-bit value, i.e., represented by the sequence of bits [<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>].

At a given generation time step, the system <NUM> processes a first recurrent input that includes the previous output value <NUM>, i.e., the N-bit output value at the preceding generation time step in the sequence, using a recurrent neural network <NUM> and in accordance with a hidden state <NUM> of the recurrent neural network <NUM> to generate a first score distribution over possible values for a first half of the N bits in the output value at the generation time step. If the output is conditioned on some external conditioning input, the first recurrent input also includes a conditioning input <NUM> for the generation time step.

The system <NUM> then selects, using the first score distribution, values <NUM> for the first half of the N bits of the output value, e.g., by sampling a sequence of N bits from the possible sequences of N bits in accordance with the score distribution. In the example of <FIG>, the system <NUM> has selected the values [<NUM>, <NUM>, <NUM>, <NUM>] as the values <NUM> of the first half of the N bits.

The system <NUM> then processes a second recurrent input that includes (i) the previous output value <NUM> and (ii) the values <NUM> for the first half of the N bits (and, if the output is conditioned on some input, the conditioning input <NUM> for the generation time step) using the recurrent neural network <NUM> and in accordance with the same hidden state <NUM> to generate a second score distribution over possible values for the second half of the N bits in the output value at the generation time step.

The system then selects, using the second score distribution, values <NUM> for the second half of the N bits of the output value, e.g., by sampling from the second score distribution. For example, the system <NUM> has selected the values [<NUM>, <NUM>, <NUM>, <NUM>] as the values <NUM> for the second half of the bits, resulting in the output value <NUM> being represented as [<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>].

Thus, at a given generation time step, the system first generates the values <NUM> for the first half of the N bits, e.g., the most significant bits, and then generates the values <NUM> for the second half of the N bits, e.g., the least significant bits, conditioned on the values of the first half of the N bits.

The architecture of the recurrent neural network <NUM> and the operations performed by the recurrent neural network to generate an output value are described below with reference to <FIG>.

By generating output examples in this manner and using a recurrent neural network as described in this specification, the system <NUM> generates accurate outputs with minimal latency and while consuming fewer computational resources. Thus, the system <NUM> may be implemented, for example, on a mobile device, on a smart speaker, or on another resource-constrained computer that nonetheless requires high-quality output examples to be generated with low latency.

<FIG> is a diagram showing the processing performed by the recurrent neural network <NUM> to generate an output value <NUM> at a generation time step t.

The processing shown in the diagram can be repeated for multiple generation time steps to generate each output value in an output example.

As shown in <FIG>, the recurrent neural network <NUM> includes only a single recurrent neural network layer <NUM>, i.e., as opposed to multiple recurrent neural network layers stacked one after the other. The recurrent neural network <NUM> also includes a set of one or more coarse output layers <NUM> and a set of one or more fine output layers <NUM>.

Generally, the single recurrent neural network layer <NUM> maintains a hidden state h that is updated at each generation time step. The hidden state is a vector of numeric values.

At generation time step t, the recurrent neural network <NUM> receives a first input <NUM> that includes the output value generated at the preceding generation time step, i.e., a combination of the values for the first half of bits ct-<NUM> from the preceding output value and the values of the second half of bits ft-<NUM> from the preceding output value. For example, the system can encode ct-<NUM> and ft-<NUM> as scalars in the range of [<NUM>,<NUM>^(N/<NUM>) - <NUM>] and then scale the scalars to the appropriate interval for processing by the recurrent neural network, e.g., [-<NUM>,<NUM>].

The recurrent neural network <NUM> processes the first input <NUM> in accordance with the hidden state from the previous generation time step ht-<NUM> to generate a first score distribution over possible values for the first half of the bits. In other words, the score distribution includes a respective score for each possible combination of values for the first half of the N bits. For example, when N is <NUM>, the score distribution will include a respective score for each of the <NUM>^<NUM> possible combinations of values for the first <NUM> bits of the output value.

The system then selects the values of the first half of the bits (the "coarse bits") ct using the score distribution, e.g., by sampling from the score distribution or selecting the combination of values with the highest score.

The recurrent neural network <NUM> then processes a second input <NUM> that includes the output value generated at the preceding generation time step (ct-<NUM>, ft-<NUM>) and the values ct (in accordance with ht-<NUM>) to generate a second score distribution over possible values for the second half of the bits (the "fine bits"). For example, the system can encode ct in the same manner as ct-<NUM> and ft-<NUM> to generate the second input <NUM>. Like the first score distribution, the second score distribution includes a respective score for each possible combination of values for the second half of the N bits. The system then selects the second half of the bits ft using the second score distribution.

In particular, the recurrent layer <NUM> processes the first input <NUM> in accordance with ht-<NUM> to generate the first half yc of the updated hidden state ht for the generation time step t.

To generate yc, the recurrent layer <NUM> determines a respective recurrent contribution for each gate in a set of one or more gates of the recurrent layer <NUM> by applying a recurrent weight matrix R to the hidden state ht-<NUM>. In other words, the recurrent layer <NUM> computes a matrix-vector product between R and ht-<NUM> to generate a vector that includes the recurrent contributions for each of the gates, i.e., with different portions of the vector corresponding to different gates.

The recurrent layer <NUM> then determines, for each gate, a respective input contribution for the gate from the first input <NUM> and determines the gate vector for the gate from at least the recurrent contribution for the gate and the input contribution for the gate. In the example of <FIG>, the recurrent layer <NUM> has three gates u, r, and e and therefore computes three gate vectors ut, rt, and et. In particular, in the example of <FIG>, the gate vectors ut, rt, and et satisfy: <MAT> <MAT> and <MAT> where σ is e.g. the sigmoid non-linear function, rcu is the recurrent contribution for gate u, <MAT> is an input weight matrix for the gate u that is masked such that ct does not affect the first half of the state yc, xt is an input to the recurrent layer, i.e., either the first input <NUM> or the second input <NUM>, rcr is the recurrent contribution for gate r, <MAT> is an input weight matrix for the gate r that is masked such that ct does not affect the first half of the state yc, τ is e.g. the tanh non-linear function, ∘ denotes element-wise multiplication, rce is the recurrent contribution for gate e, and <MAT> is an input weight matrix for the gate e that is masked such that ct does not affect the first half of the state yc. In other words, the matrices I are masked such that, when xt is the second input <NUM>, the values of ct being missing does not affect the first half of the resulting hidden state ht as shown below. While not shown in the above equations, some or all of the gates can include an addition of one or more bias vectors for the gate as part of the calculation of the gate vector for the gate, i.e., before the application of the non-linearity for the gate.

The recurrent layer <NUM> then computes yc from ht-<NUM> and the gate vectors. In the example of <FIG>, yc is the first half of the temporary hidden state ht that is computed as: <MAT> In other words, the recurrent layer <NUM> splits ht computed as above into two vectors and uses the first vector as yc.

The set of coarse output layers <NUM> then processes yc to generate the score distribution over values for the first half of bits.

In particular, in the example of <FIG>, the coarse output layers <NUM> are configured to apply a first weight matrix O<NUM> to yc to generate a first projected updated hidden state, apply an element-wise non-linear activation function (e.g., the rectified liner unit ("relu") function) to the first projected updated hidden state to generate a first activation vector, apply a second weight matrix O<NUM> to the first activation vector to generate first logits, and apply a softmax function ("softmax") to the first logits to generate the first score distribution.

Once the system has selected the first half of bits, the recurrent layer <NUM> processes the second input <NUM> in accordance with ht-<NUM> to generate the second half yf of the updated hidden state ht. In particular, the final updated hidden state ht is a concatenation ("concat") of yc and yf. In other words, the final updated hidden state ht can be split into yc and yf.

To generate yf, the recurrent layer <NUM> re-computes the gate vectors for the set of gates by using the second input <NUM> instead of the first input <NUM> and then computes yf from ht-<NUM> and the re-computed gate vectors. Advantageously, when re-computing the gate vectors for the set of gates, the recurrent layer <NUM> re-uses and does not re-compute the recurrent contributions to the gates from the processing of the first input <NUM>.

In other words, when determining the gate vectors for the second input <NUM>, the recurrent layer <NUM> does not re-compute the matrix-vector product between R and ht-<NUM> and only re-computes the input contributions for the respective gates by using the second input <NUM> as xt in the equations above.

The recurrent layer <NUM> then computes yf from ht-<NUM> and the re-computed gate vectors as described above, but with yf being the second half of the updated hidden state ht (and not the first half as described above for computing yc).

The set of fine output layers <NUM> then processes yf to generate the score distribution over values for the second half of bits.

In particular, in the example of <FIG>, the fine output layers <NUM> are configured to apply a weight matrix O<NUM> to yf to generate a second projected updated hidden state, apply an element-wise non-linear activation function (e.g., the rectified liner unit ("relu") function) to the second projected updated hidden state to generate a second activation vector, apply a weight matrix O<NUM> to the second activation vector to generate second logits, and apply a softmax function ("softmax") to the second logits to generate the second score distribution.

Thus, as can be seen from the description of <FIG>, the recurrent neural network <NUM> is able to generate the output value at the generation time step by generating two score distributions over small output spaces instead of one score distribution over a large output space. This allows the output value to be generated in a more computationally efficient manner.

As a particular example, when N is <NUM>, i.e., all of the values are <NUM>-bit values, the neural network only requires two relatively small output spaces (two score distributions of <NUM>^<NUM> scores each) instead of one large output space (with each time step needing to have <NUM>^<NUM> scores,), reducing the amount of computation required to effectively predict <NUM>-bit values. In particular, the matrices O can be of a much smaller size than the matrices that would be required to project the hidden state ht into a vector having <NUM>^<NUM> values.

Moreover, the recurrent layer <NUM> computes yf without needing to re-compute the matrix-vector product involving R, which is the most computationally-intensive operation performed by the recurrent layer <NUM>, i.e., because the operation involves multiplying a large matrix by a large vector. Thus, only a limited amount of relatively less computationally-intensive operations need to be added in order to allow for generating the output value in two passes at a given time step.

Further, the recurrent neural network <NUM> includes only a single recurrent layer. Because the recurrent layer has a hidden state that maintains an already compressed representation of the context for a given output value, an RNN is able to combine the context with the input within a single transformation. Thus, the recurrent neural network <NUM> is able to avoid using a deep and narrow architecture requiring a long chain of layers to be executed for each value and drastically reduces the number of operations that need to be performed at each time step.

A conditioning input is used to further condition the generation of the output value, and both the first input <NUM> and the second input <NUM> also include the conditioning input for the generation time step. In particular, the recurrent neural network can process received conditioning features, i.e., linguistic features, through a conditioning neural network that includes one or more neural network layers, e.g., convolutional layers, to generate a conditioning input <NUM> that is a vector that has the same dimensionality as the hidden state. Each of the gate vectors can then be based on the conditioning input <NUM>. The conditioning input <NUM> can be applied at any appropriate point before the gate vectors are calculated. In the example shown in <FIG>, the conditioning input <NUM> is added to each of the recurrent contributions, i.e. after the matrix-vector product between between R and ht-<NUM> has been computed. This can then be re-used instead of needing to be re-computed when the gate vectors are re-computed.

<FIG> is a flow diagram of an example process <NUM> for generating an output value. For convenience, the process <NUM> will be described as being performed by a system of one or more computers located in one or more locations. For example, a neural network system, e.g., the neural network system <NUM> of <FIG>, appropriately programmed, can perform the process <NUM>.

The system can repeat the process <NUM> at multiple generation time steps to generate an output example that includes multiple N-bit output values.

The system processes a first input to generate a first score distribution (step <NUM>). In particular, the first input includes the preceding output value generated at the preceding time step and, when the output example is conditioned on a conditioning input, a conditioning input for the generation time step.

The system selects values for the first half of the N bits from the first score distribution (step <NUM>), e.g., by sampling from the first score distribution.

The system processes a second input to generate a second score distribution (step <NUM>). The second input includes the preceding output value and the values of the first half of bits from the current output value.

The system selects values for the second half of the N bits from the second score distribution (step <NUM>), e.g., by sampling from the second score distribution.

Claim 1:
A computer-implemented method of generating an output example comprising a respective N-bit output value (<NUM>) at each generation time step of a sequence of generation time steps, wherein the output value comprises an audio data value for defining an audio waveform that represents an utterance of a piece of text for converting text to speech, the method comprising, for each generation time step:
processing a first recurrent input comprising the N-bit output value (<NUM>; ct-<NUM>, ft-<NUM>) at the preceding generation time step in the sequence using a recurrent neural network (<NUM>) and in accordance with a hidden state of the recurrent neural network (<NUM>; ht-<NUM>) to generate a first score distribution over possible values for a first half of the N bits in the output value at the generation time step;
selecting, using the first score distribution, values for the first half (<NUM>; ct) of the N bits of the output value;
processing a second recurrent input comprising (i) the N-bit output value (<NUM>; ct-<NUM>, ft-<NUM>) at the preceding generation time step in the sequence and (ii) the values for the first half of the N bits (<NUM>; ct) using the recurrent neural network (<NUM>) and in accordance with the same hidden state (<NUM>; ht-<NUM>) to generate a second score distribution over possible values for a second half of the N bits in the output value at the generation time step; and
selecting, using the second score distribution, values for the second half (<NUM>; ft) of the N bits of the output value;
wherein the recurrent neural network is conditioned on a sequence of linguistic features of the piece of text;
wherein the output value is conditioned on a respective conditioning input (<NUM>) at each of the generation time steps, and wherein the first recurrent input and the second recurrent input each also comprise the conditioning input for the generation time step; the method further comprising:
generating the conditioning input at the generation time step by processing the linguistic features using a conditioning neural network.