A method comprising: receiving, at data processing hardware, audio data corresponding to an utterance spoken by a user; receiving, at the data processing hardware, video data representing motion of lips of the user while the user was speaking the utterance; obtaining, by the data processing hardware, multiple candidate transcriptions for the utterance based on the audio data; characterized by: for each candidate transcription of the multiple candidate transcriptions for the utterance: generating, by the data processing hardware, a synthesized speech representation of the corresponding candidate transcription; and determining, by the data processing hardware, an agreement score indicating a likelihood that the synthesized speech representation of the corresponding candidate transcription matches the motion of the lips of the user while the user speaks the utterance, the agreement score being determined based on the synthesized speech representation and the video data; and selecting, by the data processing hardware, one of the multiple candidate transcriptions for the utterance as a speech recognition output based on the agreement scores determined for the multiple candidate transcriptions for the utterance. receiving, at data processing hardware, audio data corresponding to an utterance spoken by a user; receiving, at the data processing hardware, video data representing motion of lips of the user while the user was speaking the utterance; obtaining, by the data processing hardware, multiple candidate transcriptions for the utterance based on the audio data; characterized by: for each candidate transcription of the multiple candidate transcriptions for the utterance: generating, by the data processing hardware, a synthesized speech representation of the corresponding candidate transcription; and determining, by the data processing hardware, an agreement score indicating a likelihood that the synthesized speech representation of the corresponding candidate transcription matches the motion of the lips of the user while the user speaks the utterance, the agreement score being determined based on the synthesized speech representation and the video data; and selecting, by the data processing hardware, one of the multiple candidate transcriptions for the utterance as a speech recognition output based on the agreement scores determined for the multiple candidate transcriptions for the utterance. for each candidate transcription of the multiple candidate transcriptions for the utterance: generating, by the data processing hardware, a synthesized speech representation of the corresponding candidate transcription; and determining, by the data processing hardware, an agreement score indicating a likelihood that the synthesized speech representation of the corresponding candidate transcription matches the motion of the lips of the user while the user speaks the utterance, the agreement score being determined based on the synthesized speech representation and the video data; and generating, by the data processing hardware, a synthesized speech representation of the corresponding candidate transcription; and determining, by the data processing hardware, an agreement score indicating a likelihood that the synthesized speech representation of the corresponding candidate transcription matches the motion of the lips of the user while the user speaks the utterance, the agreement score being determined based on the synthesized speech representation and the video data; and selecting, by the data processing hardware, one of the multiple candidate transcriptions for the utterance as a speech recognition output based on the agreement scores determined for the multiple candidate transcriptions for the utterance.