A recurrent neural network-transducer (RNN-T) model comprising: an audio encoder, a prediction network, a joint network and a final Softmax layer; the prediction network configured to, at each of a plurality of time steps subsequent to an initial time step: receive, as input, a sequence of non-blank symbols output by the final Softmax layer; for each non-blank symbol in the sequence of non-blank symbols received as input at the corresponding time step: generate, using a shared embedding matrix, an embedding of the corresponding non-blank symbol; assign a respective position vector to the corresponding non-blank symbol; and weight the embedding proportional to a similarity between the embedding and the respective position vector; and generate, as output, a single embedding vector at the corresponding time step, the single embedding vector based on a weighted average of the weighted embeddings; the audio encoder configured to: receive, as input, a sequence of acoustic frames; and generate, at each of the plurality of time steps, a higher order feature representation for a corresponding acoustic frame in the sequence of acoustic frames; the joint network configured to, at each of the plurality of time steps subsequent to the initial time step: receive, as input, the single embedding vector generated as output from the prediction network at the corresponding time step; receive the higher order feature representation generated by the audio encoder at the corresponding time step as input; generate a probability distribution over possible speech recognition hypotheses at the corresponding time step; and the final Softmax layer configured to select, based on the probability distribution over possible speech recognition hypotheses at the corresponding time step, a non-blank symbol with the highest probability in the distribution, to output by the final Softmax layer, as a next non-blank symbol in the sequence of non-blank symbols. the prediction network configured to, at each of a plurality of time steps subsequent to an initial time step: receive, as input, a sequence of non-blank symbols output by the final Softmax layer; for each non-blank symbol in the sequence of non-blank symbols received as input at the corresponding time step: generate, using a shared embedding matrix, an embedding of the corresponding non-blank symbol; assign a respective position vector to the corresponding non-blank symbol; and weight the embedding proportional to a similarity between the embedding and the respective position vector; and generate, using a shared embedding matrix, an embedding of the corresponding non-blank symbol; assign a respective position vector to the corresponding non-blank symbol; and weight the embedding proportional to a similarity between the embedding and the respective position vector; and generate, as output, a single embedding vector at the corresponding time step, the single embedding vector based on a weighted average of the weighted embeddings; the audio encoder configured to: receive, as input, a sequence of acoustic frames; and generate, at each of the plurality of time steps, a higher order feature representation for a corresponding acoustic frame in the sequence of acoustic frames; receive, as input, a sequence of acoustic frames; and generate, at each of the plurality of time steps, a higher order feature representation for a corresponding acoustic frame in the sequence of acoustic frames; the joint network configured to, at each of the plurality of time steps subsequent to the initial time step: receive, as input, the single embedding vector generated as output from the prediction network at the corresponding time step; receive the higher order feature representation generated by the audio encoder at the corresponding time step as input; generate a probability distribution over possible speech recognition hypotheses at the corresponding time step; and the final Softmax layer configured to select, based on the probability distribution over possible speech recognition hypotheses at the corresponding time step, a non-blank symbol with the highest probability in the distribution, to output by the final Softmax layer, as a next non-blank symbol in the sequence of non-blank symbols. receive, as input, the single embedding vector generated as output from the prediction network at the corresponding time step; receive the higher order feature representation generated by the audio encoder at the corresponding time step as input; generate a probability distribution over possible speech recognition hypotheses at the corresponding time step; and the final Softmax layer configured to select, based on the probability distribution over possible speech recognition hypotheses at the corresponding time step, a non-blank symbol with the highest probability in the distribution, to output by the final Softmax layer, as a next non-blank symbol in the sequence of non-blank symbols.