A system comprising: a recurrent neural network implemented by one or more computers, wherein the recurrent neural network is configured to receive a respective neural network input at each of a plurality of time steps and to generate a respective neural network output at each of the plurality of time steps, wherein the neural network input comprises an image or features extracted from an image and the neural network output generated by the recurrent neural network is a score for each of a set of object categories, with each score representing an estimated likelihood that the image contains an image of an object belonging to the category, wherein optionally the set of object categories are a set of pieces of text, or wherein the neural network input comprises features of a spoken utterance and the neural network output generated by the recurrent neural network is a score for each of a set of pieces of text, each score representing an estimated likelihood that the piece of text is the correct transcription for the utterance, wherein the recurrent neural network includes an associative long-short term memory, LSTM, layer, wherein the associative LSTM layer is configured to maintain N copies of an internal state for the associative LSTM layer, N being an integer greater than one, and each copy of the internal state is a different representation of the internal state; and wherein the associative LSTM layer is further configured to, at each of the plurality of time steps: receive a layer input for the time step, update each of the N copies of the internal state using the layer input for the time step and a layer output generated by the associative LSTM layer for a preceding time step, and generate a layer output for the time step using the N updated copies of the internal state. a recurrent neural network implemented by one or more computers, wherein the recurrent neural network is configured to receive a respective neural network input at each of a plurality of time steps and to generate a respective neural network output at each of the plurality of time steps, wherein the neural network input comprises an image or features extracted from an image and the neural network output generated by the recurrent neural network is a score for each of a set of object categories, with each score representing an estimated likelihood that the image contains an image of an object belonging to the category, wherein optionally the set of object categories are a set of pieces of text, or wherein the neural network input comprises features of a spoken utterance and the neural network output generated by the recurrent neural network is a score for each of a set of pieces of text, each score representing an estimated likelihood that the piece of text is the correct transcription for the utterance, wherein the neural network input comprises an image or features extracted from an image and the neural network output generated by the recurrent neural network is a score for each of a set of object categories, with each score representing an estimated likelihood that the image contains an image of an object belonging to the category, wherein optionally the set of object categories are a set of pieces of text, or wherein the neural network input comprises features of a spoken utterance and the neural network output generated by the recurrent neural network is a score for each of a set of pieces of text, each score representing an estimated likelihood that the piece of text is the correct transcription for the utterance, wherein the recurrent neural network includes an associative long-short term memory, LSTM, layer, wherein the associative LSTM layer is configured to maintain N copies of an internal state for the associative LSTM layer, N being an integer greater than one, and each copy of the internal state is a different representation of the internal state; and wherein the associative LSTM layer is further configured to, at each of the plurality of time steps: receive a layer input for the time step, update each of the N copies of the internal state using the layer input for the time step and a layer output generated by the associative LSTM layer for a preceding time step, and generate a layer output for the time step using the N updated copies of the internal state. receive a layer input for the time step, update each of the N copies of the internal state using the layer input for the time step and a layer output generated by the associative LSTM layer for a preceding time step, and generate a layer output for the time step using the N updated copies of the internal state.