Grid long short-term memory neural networks

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for implementing grid Long Short-Term Memory (LSTM) neural networks that includes a plurality of N-LSTM blocks arranged in an N-dimensional grid. Each N-LSTM block is configured to: receive N input hidden vectors, the N input hidden vectors each corresponding to a respective one of the N dimensions; receive N input memory vectors, the N input memory vectors each corresponding to a respective one of the N dimensions; and, for each of the dimensions, apply a respective transform for the dimension to the memory hidden vector corresponding to the dimension and the input hidden vector corresponding to the dimension to generate a new hidden vector corresponding to the dimension and a new memory vector corresponding to the dimension.

BACKGROUND

This specification relates to neural network architectures.

Some neural networks are recurrent neural networks. A recurrent neural network is a neural network that receives an input sequence and generates an output sequence from the input sequence. In particular, a recurrent neural network can use some or all of the internal state of the network from a previous time step in computing an output at a current time step. An example of a recurrent neural network is a Long Short-Term Memory (LSTM) neural network that includes one or more LSTM cells that each include an input gate, a forget gate, and an output gate that allow the cell to store previous states for the cell, e.g., for use in generating a current activation or to be provided to other components of the LSTM neural network.

SUMMARY

This specification describes technologies that relate to neural network architectures. In general, a grid Long Short-Term Memory (LSTM) neural network includes multiple N-LSTM blocks arranged in an N-dimensional grid. Each N-LSTM block is configured to, for each of the N dimensions, receive (i) an input hidden vector for the dimension and (ii) an input memory vector for the dimension and, for each of the N dimensions, apply a transform to the input hidden vector for the dimension and the input memory vector for the dimension to determine a new hidden vector and a new memory vector for the dimension.

The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages. A neural network system implementing a grid LSTM neural network as described in this specification effectively modulates N-way communication between the N-LSTM blocks in the grid LSTM neural network, allowing the grid LSTM neural network to be effectively used for both deep and sequential computation. In particular, as the number of paths in the grid grows combinatorically with the size of each dimension and the total number of dimensions N, the values in memory vectors passed between N-LSTM blocks are prevented from growing at the same rate. Depending on the value of N and because of the effective modulation of the communication, the grid LSTM neural network can be effectively applied to input vectors, input sequences, or higher dimensional data such as images. Within a given N-LSTM block, the architecture of the block ensures that the hidden and memory vectors from the different dimensions will interact closely without being conflated, resulting in improved performance on machine learning tasks.

DETAILED DESCRIPTION

FIG. 1shows an example neural network system100. The neural network system100is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below are implemented.

The neural network system100is a machine learning system that receives a neural network input102and generates a neural network output132from the neural network input102.

The neural network system100can store the generated neural network output132in an output data repository or provide the neural network outputs for use for some other immediate purpose, e.g., for presentation to a user in a user interface of a user device. The user device may be located remotely from the neural network system100and the system100may transmit the neural network output132over a data communication network for presentation on the user device or the user device may be one of the one or more computers on which the neural network system100is implemented.

The neural network system100can be configured to receive any kind of digital data input and to generate any kind of score or classification output based on the input.

For example, if the inputs to the neural network system100are images or features that have been extracted from images, the output generated by the neural network system100for a given image may be scores for each of a set of object categories, with each score representing an estimated likelihood that the image contains an image of an object belonging to the category.

As another example, if the inputs to the neural network system100are Internet resources (e.g., web pages), documents, or portions of documents or features extracted from Internet resources, documents, or portions of documents, the output generated by the neural network system100for a given Internet resource, document, or portion of a document may be a score for each of a set of topics, with each score representing an estimated likelihood that the Internet resource, document, or document portion is about the topic.

As another example, if the inputs to the neural network system100are features of a personalized recommendation for a user, e.g., features characterizing the context for the recommendation, e.g., features characterizing previous actions taken by the user, the output generated by the neural network system100may be a score for each of a set of content items, with each score representing an estimated likelihood that the user will respond favorably to being recommended the content item. In some of these examples, the neural network system100is part of a reinforcement learning system that provides content recommendations to users.

As another example, if the input to the neural network system100is a sequence of text in one language, the output generated by the neural network system100may be a score for each of a set of pieces of text in another language, with each score representing an estimated likelihood that the piece of text in the other language is a proper translation of the input text into the other language.

As another example, if the input to the neural network system100is a sequence of features of a spoken utterance, the output generated by the neural network system100may be a score for each of a set of pieces of text, each score representing an estimated likelihood that the piece of text is the correct transcription for the utterance.

As another example if the inputs to the neural network system100are images, the output generated by the neural network system100may be a score for each of a set of pieces of text, each score representing an estimated likelihood that the piece of text is text that is present in the input image.

In particular, the neural network system100includes a grid Long Short-Term Memory (LSTM) neural network110. The grid LSTM neural network110includes multiple N-LSTM blocks arranged in an N-dimensional grid, where N is an integer greater than or equal to 1.

Each N-LSTM block in the grid is configured to, for each of the N dimensions, receive (i) an input hidden vector for the dimension and (ii) an input memory vector for the dimension and, for each of the N dimensions, apply a transform to the input hidden vector for the dimension and the input memory vector for the dimension to determine a new hidden vector and a new memory vector for the dimension.

For example, an N-LSTM block120in the grid LSTM neural network110is configured to receive N input hidden vectors122and N input memory vectors124. The N-LSTM block120is then configured to, for each of the N dimensions, apply a transform to the input hidden vector for the dimension and the input memory vector for the dimension to determine N new hidden vectors126and N new memory vectors128. Generating new hidden vectors and new memory vectors is described in more detail below with reference toFIGS. 3-5.

As will be described in more detail below, in order to cause the grid LSTM neural network110to generate a neural network output for the neural network input102, the neural network system100provides all of or a portion of the neural network input102as the input hidden vector, the input memory vector, or both for one or more predetermined dimensions to a predetermined subset of the N-LSTM blocks in the grid. In particular, the neural network system100can provide all of or a portion of the neural network input102as input to the N-LSTM blocks that are first in the grid along the one or more predetermined dimensions.

Once generated, each N-LSTM block is configured to provide the new hidden vector and the new memory vector generated by the N-LSTM block for a given dimension as input to the next N-LSTM block along that dimension in the grid unless the N-LSTM block is the last N-LSTM block in the grid along that dimension.

In some implementations, the neural network output132is made up of the new memory vectors, the new hidden vectors, or both for a particular dimension generated by one or more predetermined N-LSTM blocks in the grid, e.g., the last N-LSTM in the grid along the particular dimension.

In some other implementations, the grid LSTM neural network layer110also includes an output layer, e.g., a softmax layer or a logistic regression classifier layer, that is configured to receive the new memory vectors, the new hidden vectors, or both for the particular dimension generated by the one or more predetermined N-LSTM blocks and to process the received input to generate the neural network output132.

FIG. 2Ashows an example 1D Grid LSTM neural network200that includes two 1-LSTM blocks210and220arranged in a one-dimensional grid, i.e., stacked one on top of the other. While the example ofFIG. 2Ashows two 1-LSTM blocks, the one-dimensional grid may include any number of 1-LSTM blocks.

In the example ofFIG. 2A, each of the 1-LSTM blocks210and220receives a single input memory vector and a single input hidden vector and applies a transform to the input memory vector and the input hidden vector to generate a single new input memory vector and a single new input hidden vector. In particular, the input hidden vector and the input memory vector to the 1-LSTM block210may be the neural network input102, and the input hidden vector and the input memory vector to the 1-LSTM block220may be the new input memory vector and the new input hidden vector generated by the 1-LSTM block210. The new input memory vector, the new input hidden vector, or both generated by the 1-LSTM block220may be used as the neural network output132or may be provided to an output layer for use in generating the neural network output132.

In particular, the 2-dimensional grid is a 2×2 grid, having two 2-LSTM blocks along a dimension t, which may be conceptualized as a time dimension for processing neural network inputs that are sequences, and two 2-LSTM blocks along a dimension d, which may be conceptualized as a depth dimension representing the depth of computation for each input in the sequence.

In the example ofFIG. 2B, the neural network input102is a sequence made up of an input A252followed by an input B254. Accordingly, the input A252is provided as the input hidden vector, the input memory vector, or both to the 2-LSTM block256along the d dimension, while the input B254is provided as the input hidden vector, the input memory vector, or both to the 2-LSTM block260along the d dimension.

The new memory vector and the new hidden vector generated by the 2-LSTM block256along the d dimension are provided as the input memory and hidden vectors along the d dimension to a 2-LSTM block258, while the new memory vector and the new hidden vector generated by the 2-LSTM block256along the t dimension are provided as the input memory and hidden vectors along the t dimension to the 2-LSTM block260.

Similarly, a 2-LSTM block262receives as input hidden and memory vectors along the d dimension the new memory and hidden vectors generated by the 2-LSTM block260for the d dimension and as input memory and hidden vectors along the t dimension the new memory and hidden vectors generated by the 2-LSTM block258for the t dimension. As an example, the new hidden and memory vectors generated by the 2-LSTM block262along the d dimension may be used by an output layer to generate the neural network output132for the neural network input102, while the remainder of the new hidden and memory vectors are discarded.

In particular, the 3D Grid LSTM neural network300is configured to process neural network inputs having multiple inputs along an x dimension, e.g., neural network inputs A302and neural network inputs B304, and multiple inputs along a y dimension, e.g., neural network inputs C306and neural network inputs D308, with computation depth d. Thus, the 3-dimensional grid is a 2×2×2 grid, having two 3-LSTM blocks along the dimension x, two 3-LSTM blocks along the dimension y, and two 3-LSTM blocks along the dimension d. While the numbers of blocks along each dimension is shown as equal in the example ofFIG. 2C, different number of blocks can be arranged along the different dimensions.

Accordingly, in the example ofFIG. 2C, a 3-LSTM Block310is configured to receive the neural network input A302as the input hidden vector, the input memory vector, or both to the block along they dimension, and the neural network input C306as the input hidden vector, the input memory vector, or both to the block along the x dimension. Similarly, the 3-LSTM block320is configured to receive the neural network input D308as the input hidden vector, the input memory vector, or both to the block along the x dimension, while the 3-LSTM block360is configured to receive the neural network input B304as the input hidden vector, the input memory vector, or both to the block along they dimension.

FIG. 3is a flow diagram of an example process350for processing an input to an N-LSTM block For convenience, the process350will be described as being performed by an N-LSTM block implemented by a system of one or more computers located in one or more locations. For example, an N-LSTM block in a neural network system, e.g., the N-LSTM block120of the neural network system100ofFIG. 1, appropriately programmed in accordance with this specification, can perform the process350.

The N-LSTM block receives a respective input hidden vector for each of the N dimensions (step352). That is, the N-LSTM block receives N input hidden vectors, with each hidden vector corresponding to a different one of the N dimensions. Depending on the position of the N-LSTM block within the grid LSTM neural network, the input hidden vector for a given dimension may be a hidden vector generated by another N-LSTM block, some or all of the neural network input, or a placeholder input hidden vector that is predetermined or learned through training.

The N-LSTM block receives a respective input memory vector for each of the N dimensions (step354). That is, the N-LSTM block receives N input memory vectors, with each memory vector corresponding to a different one of the N dimensions. Depending on the position of the N-LSTM block within the grid LSTM neural network, the input memory vector for a given dimension may be a memory vector generated by another N-LSTM block, some or all of the neural network input, or a placeholder input memory vector that is predetermined or learned through training.

For each dimension, the N-LSTM block applies a transform to the input hidden vector and the input memory vector for the dimension to generate a new hidden vector and a new memory vector for the dimension (step356).

Generally, the N-LSTM block generates a concatenated hidden vector for each dimension and applies the transform for the dimension to the concatenated hidden vector and the input memory vector for the dimension.

For at least some of the dimensions, the N-LSTM block generates the concatenated hidden vector by concatenating the N input hidden vectors. In some implementations, however, for a predetermined one of the dimensions, the N-LSTM block is configured to generate the concatenated hidden vector by concatenating the input hidden vector for the dimension and the new hidden vectors for each of the other dimensions. That is, the N-LSTM block is configured to first apply the transform in each dimension other than the predetermined dimension to generate the new hidden vectors for the other dimensions and then generate the concatenated hidden vector for the predetermined dimension using the input hidden vector for the dimension and the generated new hidden vectors for the other dimensions.

In some implementations, the transform for each dimension is an LSTM transform. Applying an LSTM transform to the concatenated hidden vector and the input memory vector for a given dimension is described in more detail below with reference toFIG. 4. In some other implementations, for one or more predetermined dimensions, rather than apply an LSTM transform, the N-LSTM block is configured to instead apply a non-LSTM transform to the concatenated hidden vector for the predetermined dimension. For example, the non-LSTM transform may consist of processing the concatenated hidden vector using a conventional fully-connected neural network layer with or without an activation function to generate the new hidden vector for the predetermined dimension.

Depending on the position of the N-LSTM block within the grid LSTM neural network, the new memory vector and the new hidden vector generated by the N-LSTM block for a given dimension can either be discarded, used as some or all of the neural network output, provided to an output layer, or provided to another N-LSTM block.

In particular, if the N-LSTM block is not the last N-LSTM block in the grid along a given dimension, the N-LSTM block provides the new hidden vector and the new memory vector for the given dimension to the next N-LSTM block along the given dimension for use as the input hidden vector and the input memory vector for that N-LSTM block for the given dimension.

If the N-LSTM block is the last N-LSTM block in the grid along a given dimension, the N-LSTM block can be configured to (i) discard the new hidden vector and the new memory vector, i.e., if the vectors generated for the given dimension are not to be used in generating the neural network output, (ii) provide the new hidden vector, the new memory vector, or both as some or all of the neural network output, (iii) provide the new hidden vector, the new memory vector or both, to an output layer of the grid LSTM neural network for use in generating the neural network output, or (iv) provide the new hidden vector and the new memory vector as input to another N-LSTM block along a different dimension. In some implementations, when the grid LSTM neural network includes an output layer, the output layer is configured to receive as input not only the new hidden vector, the new memory vector or both generated by the last N-LSTM block in the grid along a given dimension, but also the new hidden vector, the new memory vector or both for the given dimension generated by one or more other N-LSTM blocks that are not last in the grid along the given dimension.

FIG. 4is a flow diagram of an example process400for applying an LSTM transform to a hidden vector and a memory vector for a given dimension. For convenience, the process400will be described as being performed by an N-LSTM block implemented by a system of one or more computers located in one or more locations. For example, an N-LSTM block in a neural network system, e.g., the N-LSTM block120of the neural network system100ofFIG. 1, appropriately programmed in accordance with this specification, can perform the process400.

The N-LSTM block generates a forget gate vector from the concatenated hidden vector for the dimension (step402). Generating a gate vector from a concatenated hidden vector will be described in more detail below with reference toFIG. 5.

The N-LSTM block generates an input gate vector from the concatenated hidden vector for the dimension (step404). Generating a gate vector from a concatenated hidden vector will be described in more detail below with reference toFIG. 5.

In some implementations, instead of generating an input gate vector separately from generating the forget gate vector, the system uses a forget gate vector that is equal to one minus the input gate vector. That is, in some implementations, the input gate is the same as the forget gate.

The N-LSTM block generates an output gate vector from the concatenated hidden vector for the dimension (step406). Generating a gate vector from a concatenated hidden vector will be described in more detail below with reference toFIG. 5.

The N-LSTM block generates an intermediate memory vector update vector from the concatenated hidden vector (step408). In particular, the N-LSTM block processes the concatenated hidden vector using a neural network layer having an activation function that is a squashing function to generate the intermediate memory vector update vector.

Generally, a squashing function is a function that maps received inputs to a range of −1 to 1, exclusive. For example, the squashing function may be the hyperbolic tangent function. For example, the neural network layer may be a layer that performs a matrix multiplication between a parameter matrix and the concatenated hidden vector, optionally adds a bias vector to the product, and then applies the saturating squashing function to each component of either the resulting vector or of the product if no bias vector is applied.

The N-LSTM block combines the intermediate memory vector update vector and the input gate vector to generate a final memory vector update vector (step410). In particular, the N-LSTM block computes a point-wise multiplication between the intermediate memory vector update vector and the input gate vector to generate the final memory vector update vector.

The N-LSTM block combines the input memory vector for the dimension and the forget gate vector to generate an intermediate new memory vector (step412). In particular, the N-LSTM block computes a point-wise multiplication between the input memory vector for the dimension and the forget vector to generate the intermediate new memory vector.

The N-LSTM block combines, e.g., sums, the intermediate new memory vector and the final memory vector update vector to generate a final new memory vector for the dimension (step414).

The N-LSTM block generates a new hidden vector for the dimension from the final new memory vector for the dimension (step416).

To generate the new hidden vector, the N-LSTM block combines the output gate vector and the final new memory vector for the dimension to generate an intermediate new hidden vector. In particular, the N-LSTM block performs a pointwise multiplication between the output gate vector and the final new memory vector to generate the intermediate new hidden vector. The N-LSTM block then applies a squashing function to the intermediate new hidden vector to generate the final new hidden vector for the dimension.

FIG. 5is a flow diagram of an example process500for generating a gate vector from a concatenated hidden vector. For convenience, the process500will be described as being performed by an N-LSTM block implemented by a system of one or more computers located in one or more locations. For example, an N-LSTM block in a neural network system, e.g., the N-LSTM block120of the neural network system100ofFIG. 1, appropriately programmed in accordance with this specification, can perform the process500.

The N-LSTM block determines the concatenated hidden vector for the dimension, e.g., as described above (step502).

The N-LSTM block generates a respective intermediate gate vector from the concatenated hidden vector in accordance with a set of parameters (step504). In some implementations, the N-LSTM block performs a matrix multiplication between a parameter matrix and the concatenated hidden vector and then optionally adds a bias vector to the output of the matrix multiplication to generate the intermediate gate vector, with each of the gates having different parameter matrices and bias vectors. That is, in implementations where the N-LSTM block has a distinct input gate, forget gate, and output gate, each of these gates will have different parameter matrices and bias vectors from each other gate.

Additionally, each dimension generally has different parameter matrices from the other dimensions for a given N-LSTM block. In some implementations, however, constraints can be imposed so that parameter matrices are shared along one or more of the dimensions across multiple N-LSTM blocks in the grid LSTM neural network, e.g., to induce invariance in the computation along that dimension.

The N-LSTM block applies a gating function to each component of the respective intermediate gate vector to generate a final gate vector (step506).

Generally, a gating function is a function that maps received inputs to a range of 0 to 1, exclusive. For example, a gating function may be the sigmoid function.