A method performed by one or more computers, the method comprising: receiving a voice query from a user that includes (i) audio data corresponding to an utterance of the user, and (ii) context data associated with at least one of the audio data or the user; in response to receiving the voice query, generating one or more class models that collectively identify (i) a first set of terms determined based on the context data associated with the at least one of the audio data or the user and (ii) for each respective term in the first set of terms, a respective class to which the respective term is assigned, wherein at least some terms in the first set of terms are assigned to different ones of a plurality of classes; accessing a language model that includes a residual unigram representing a second set of terms; processing the language model, including for each respective class among the plurality of classes, inserting, at each instance of the residual unigram that occurs within the language model, a respective class symbol associated with the respective class; and generating a transcription of the utterance of the user using the modified language model, wherein generating the transcription of the utterance of the user comprises: accessing one or more class-based language models that correspond to respective classes to which one or more terms within the first set of terms are assigned; determining, for each respective class symbol that was inserted into the modified language model and based at least on probabilities indicated by a particular one of the class-based language models that corresponds to the respective class indicated by the respective class symbol, a respective probability that a term within the respective class occurs within a candidate language sequence at a position in the candidate language sequence that corresponds to a position of the respective class symbol; and generating the transcription of the utterance of the user using the determined probabilities. receiving a voice query from a user that includes (i) audio data corresponding to an utterance of the user, and (ii) context data associated with at least one of the audio data or the user; in response to receiving the voice query, generating one or more class models that collectively identify (i) a first set of terms determined based on the context data associated with the at least one of the audio data or the user and (ii) for each respective term in the first set of terms, a respective class to which the respective term is assigned, wherein at least some terms in the first set of terms are assigned to different ones of a plurality of classes; accessing a language model that includes a residual unigram representing a second set of terms; processing the language model, including for each respective class among the plurality of classes, inserting, at each instance of the residual unigram that occurs within the language model, a respective class symbol associated with the respective class; and generating a transcription of the utterance of the user using the modified language model, wherein generating the transcription of the utterance of the user comprises: accessing one or more class-based language models that correspond to respective classes to which one or more terms within the first set of terms are assigned; determining, for each respective class symbol that was inserted into the modified language model and based at least on probabilities indicated by a particular one of the class-based language models that corresponds to the respective class indicated by the respective class symbol, a respective probability that a term within the respective class occurs within a candidate language sequence at a position in the candidate language sequence that corresponds to a position of the respective class symbol; and generating the transcription of the utterance of the user using the determined probabilities. accessing one or more class-based language models that correspond to respective classes to which one or more terms within the first set of terms are assigned; determining, for each respective class symbol that was inserted into the modified language model and based at least on probabilities indicated by a particular one of the class-based language models that corresponds to the respective class indicated by the respective class symbol, a respective probability that a term within the respective class occurs within a candidate language sequence at a position in the candidate language sequence that corresponds to a position of the respective class symbol; and generating the transcription of the utterance of the user using the determined probabilities.