A method comprising: receiving, at data processing hardware , audio data corresponding to an utterance spoken by a user ; receiving, at the data processing hardware , video data representing motion of lips of the user while the user was speaking the utterance ; obtaining, by the data processing hardware , multiple candidate transcriptions for the utterance based on the audio data ; characterized by : for each candidate transcription of the multiple candidate transcriptions for the utterance : generating, by the data processing hardware , a synthesized speech representation of the corresponding candidate transcription ; and determining, by the data processing hardware , an agreement score indicating a likelihood that the synthesized speech representation of the corresponding candidate transcription matches the motion of the lips of the user while the user speaks the utterance , the agreement score being determined based on the synthesized speech representation and the video data ; and selecting, by the data processing hardware , one of the multiple candidate transcriptions for the utterance as a speech recognition output based on the agreement scores determined for the multiple candidate transcriptions for the utterance . receiving, at data processing hardware , audio data corresponding to an utterance spoken by a user ; receiving, at the data processing hardware , video data representing motion of lips of the user while the user was speaking the utterance ; obtaining, by the data processing hardware , multiple candidate transcriptions for the utterance based on the audio data ; characterized by : for each candidate transcription of the multiple candidate transcriptions for the utterance : generating, by the data processing hardware , a synthesized speech representation of the corresponding candidate transcription ; and determining, by the data processing hardware , an agreement score indicating a likelihood that the synthesized speech representation of the corresponding candidate transcription matches the motion of the lips of the user while the user speaks the utterance , the agreement score being determined based on the synthesized speech representation and the video data ; and selecting, by the data processing hardware , one of the multiple candidate transcriptions for the utterance as a speech recognition output based on the agreement scores determined for the multiple candidate transcriptions for the utterance . for each candidate transcription of the multiple candidate transcriptions for the utterance : generating, by the data processing hardware , a synthesized speech representation of the corresponding candidate transcription ; and determining, by the data processing hardware , an agreement score indicating a likelihood that the synthesized speech representation of the corresponding candidate transcription matches the motion of the lips of the user while the user speaks the utterance , the agreement score being determined based on the synthesized speech representation and the video data ; and generating, by the data processing hardware , a synthesized speech representation of the corresponding candidate transcription ; and determining, by the data processing hardware , an agreement score indicating a likelihood that the synthesized speech representation of the corresponding candidate transcription matches the motion of the lips of the user while the user speaks the utterance , the agreement score being determined based on the synthesized speech representation and the video data ; and selecting, by the data processing hardware , one of the multiple candidate transcriptions for the utterance as a speech recognition output based on the agreement scores determined for the multiple candidate transcriptions for the utterance . <EOS>