A computer-implemented method comprising: receiving, by a computing device, audio data of an utterance; generating, by the computing device using (i) a neural network or (ii) an acoustic model and a language model, a word lattice that includes multiple candidate transcriptions of the utterance and that includes transcription confidence scores that each reflect a likelihood that a respective candidate transcription is a match for the utterance; determining, by the computing device, a context of the computing device; based on the context of the computing device, identifying, by the computing device, grammars that correspond to the multiple candidate transcriptions; based on the context, determining, by the computing device and for each of the multiple candidate transcriptions, grammar confidence scores that reflect a likelihood that a respective grammar is a match for a respective candidate transcription; determining that two or more of the grammars correspond to one of the candidate transcriptions; based on determining that two or more of the grammars correspond to one of the candidate transcriptions, adjusting the grammar confidence scores for the two or more grammars by increasing the grammar confidence scores for the two or more grammars; based on the transcription confidence scores and the adjusted grammar confidence scores, selecting, by the computing device and from among the candidate transcriptions, a candidate transcription; and providing, for output by the computing device, the selected candidate transcription as a transcription of the utterance, wherein adjusting the grammar confidence scores for the two or more grammars comprises: increasing each of the grammar confidence scores for each of the two or more grammars by a factor, wherein a product of the factor and the highest grammar confidence score of the grammar confidence scores is 1.0. receiving, by a computing device, audio data of an utterance; generating, by the computing device using (i) a neural network or (ii) an acoustic model and a language model, a word lattice that includes multiple candidate transcriptions of the utterance and that includes transcription confidence scores that each reflect a likelihood that a respective candidate transcription is a match for the utterance; determining, by the computing device, a context of the computing device; based on the context of the computing device, identifying, by the computing device, grammars that correspond to the multiple candidate transcriptions; based on the context, determining, by the computing device and for each of the multiple candidate transcriptions, grammar confidence scores that reflect a likelihood that a respective grammar is a match for a respective candidate transcription; determining that two or more of the grammars correspond to one of the candidate transcriptions; based on determining that two or more of the grammars correspond to one of the candidate transcriptions, adjusting the grammar confidence scores for the two or more grammars by increasing the grammar confidence scores for the two or more grammars; based on the transcription confidence scores and the adjusted grammar confidence scores, selecting, by the computing device and from among the candidate transcriptions, a candidate transcription; and providing, for output by the computing device, the selected candidate transcription as a transcription of the utterance, wherein adjusting the grammar confidence scores for the two or more grammars comprises: increasing each of the grammar confidence scores for each of the two or more grammars by a factor, wherein a product of the factor and the highest grammar confidence score of the grammar confidence scores is 1.0.