A neural machine translation system implemented by one or more computers, the neural machine translation system configured to receive an input sequence of input tokens representing a first sequence of words in a first natural language and to generate an output sequence of output tokens representing a second sequence of words that is a translation of the first sequence into a second natural language, the neural machine translation system comprising: an encoder neural network comprising: an input forward long short-term memory (LSTM) layer configured to process each input token in the input sequence in a forward order to generate a respective forward representation of each input token, an input backward LSTM layer configured to process each input token in the input sequence in a backward order to generate a respective backward representation of each input token, a combining layer configured to, for each input token in the input sequence, combine the forward representation of the input token and the backward representation of the input token to generate a combined representation of the input token, and a plurality of hidden LSTM layers configured to process each combined representation in the forward order to generate a respective encoded representation of each of the input tokens in the input sequence, wherein the plurality of hidden LSTM layers are configured to perform computations in parallel, the plurality of hidden LSTM layers being unidirectional and arranged in a stack one after the other; and a decoder subsystem configured to receive the respective encoded representation of each of the input tokens in the input sequence and to process the encoded representations to generate the output sequence. an encoder neural network comprising: an input forward long short-term memory (LSTM) layer configured to process each input token in the input sequence in a forward order to generate a respective forward representation of each input token, an input backward LSTM layer configured to process each input token in the input sequence in a backward order to generate a respective backward representation of each input token, a combining layer configured to, for each input token in the input sequence, combine the forward representation of the input token and the backward representation of the input token to generate a combined representation of the input token, and a plurality of hidden LSTM layers configured to process each combined representation in the forward order to generate a respective encoded representation of each of the input tokens in the input sequence, wherein the plurality of hidden LSTM layers are configured to perform computations in parallel, the plurality of hidden LSTM layers being unidirectional and arranged in a stack one after the other; and an input forward long short-term memory (LSTM) layer configured to process each input token in the input sequence in a forward order to generate a respective forward representation of each input token, an input backward LSTM layer configured to process each input token in the input sequence in a backward order to generate a respective backward representation of each input token, a combining layer configured to, for each input token in the input sequence, combine the forward representation of the input token and the backward representation of the input token to generate a combined representation of the input token, and a plurality of hidden LSTM layers configured to process each combined representation in the forward order to generate a respective encoded representation of each of the input tokens in the input sequence, wherein the plurality of hidden LSTM layers are configured to perform computations in parallel, the plurality of hidden LSTM layers being unidirectional and arranged in a stack one after the other; and a decoder subsystem configured to receive the respective encoded representation of each of the input tokens in the input sequence and to process the encoded representations to generate the output sequence.