A computer-implemented method when executed on data processing hardware causes the data processing hardware to perform operations comprising: receiving a training example for a second pass decoder of a two-pass neural network model, the training example comprising audio data representing a spoken utterance of one or more words and a corresponding ground truth transcription of the spoken utterance; for each word in the spoken utterance: inserting a placeholder symbol before the respective word; identifying a respective ground truth alignment (312, 314) for a beginning of the respective word and an end of the respective word; determining a beginning word piece of the respective word and an ending word piece of the respective word; and generating a first constrained alignment for the beginning word piece of the respective word and a second constrained alignment for the ending word piece of the respective word, the first constrained alignment aligned with the ground truth alignment (312, 314) for the beginning of the respective word, the second constrained alignment aligned with the ground truth alignment (312, 314) for the ending of the respective word; and constraining an attention head of the second pass decoder of the two-pass neural network model by applying the training example comprising all of the first constrained alignments and the second constrained alignments for each word of the training example. receiving a training example for a second pass decoder of a two-pass neural network model, the training example comprising audio data representing a spoken utterance of one or more words and a corresponding ground truth transcription of the spoken utterance; for each word in the spoken utterance: inserting a placeholder symbol before the respective word; identifying a respective ground truth alignment (312, 314) for a beginning of the respective word and an end of the respective word; determining a beginning word piece of the respective word and an ending word piece of the respective word; and generating a first constrained alignment for the beginning word piece of the respective word and a second constrained alignment for the ending word piece of the respective word, the first constrained alignment aligned with the ground truth alignment (312, 314) for the beginning of the respective word, the second constrained alignment aligned with the ground truth alignment (312, 314) for the ending of the respective word; and inserting a placeholder symbol before the respective word; identifying a respective ground truth alignment (312, 314) for a beginning of the respective word and an end of the respective word; determining a beginning word piece of the respective word and an ending word piece of the respective word; and generating a first constrained alignment for the beginning word piece of the respective word and a second constrained alignment for the ending word piece of the respective word, the first constrained alignment aligned with the ground truth alignment (312, 314) for the beginning of the respective word, the second constrained alignment aligned with the ground truth alignment (312, 314) for the ending of the respective word; and constraining an attention head of the second pass decoder of the two-pass neural network model by applying the training example comprising all of the first constrained alignments and the second constrained alignments for each word of the training example.