A computer-implemented method when executed on data processing hardware causes the data processing hardware to perform operations comprising: receiving a training example for a second pass decoder of a two-pass neural network model, the training example comprising audio data representing a spoken utterance of one or more words and a corresponding ground truth transcription of the spoken utterance ; for each word in the spoken utterance : inserting a placeholder symbol before the respective word ; identifying a respective ground truth alignment for a beginning of the respective word and an end of the respective word ; determining a beginning word piece of the respective word and an ending word piece of the respective word ; and generating a first constrained alignment for the beginning word piece of the respective word and a second constrained alignment for the ending word piece of the respective word , the first constrained alignment aligned with the ground truth alignment for the beginning of the respective word , the second constrained alignment aligned with the ground truth alignment for the ending of the respective word ; and constraining an attention head of the second pass decoder of the two-pass neural network model by applying the training example comprising all of the first constrained alignments and the second constrained alignments for each word of the training example . receiving a training example for a second pass decoder of a two-pass neural network model, the training example comprising audio data representing a spoken utterance of one or more words and a corresponding ground truth transcription of the spoken utterance ; for each word in the spoken utterance : inserting a placeholder symbol before the respective word ; identifying a respective ground truth alignment for a beginning of the respective word and an end of the respective word ; determining a beginning word piece of the respective word and an ending word piece of the respective word ; and generating a first constrained alignment for the beginning word piece of the respective word and a second constrained alignment for the ending word piece of the respective word , the first constrained alignment aligned with the ground truth alignment for the beginning of the respective word , the second constrained alignment aligned with the ground truth alignment for the ending of the respective word ; and inserting a placeholder symbol before the respective word ; identifying a respective ground truth alignment for a beginning of the respective word and an end of the respective word ; determining a beginning word piece of the respective word and an ending word piece of the respective word ; and generating a first constrained alignment for the beginning word piece of the respective word and a second constrained alignment for the ending word piece of the respective word , the first constrained alignment aligned with the ground truth alignment for the beginning of the respective word , the second constrained alignment aligned with the ground truth alignment for the ending of the respective word ; and constraining an attention head of the second pass decoder of the two-pass neural network model by applying the training example comprising all of the first constrained alignments and the second constrained alignments for each word of the training example . <EOS>