A method for improving a speech recognition system comprising the steps of: initiating a speech dialog with at least one point in the dialog where there is a grammar of possible responses and a set of at least one expected response, wherein the set is a subset of the grammar, the set includes the response or responses expected to be uttered by a user at the at least one point in the speech dialog, and the set of at least one expected response for the at least one point is known in the speech recognition system before receiving input speech from the user; progressing through the speech dialog until arriving at the at least one point; receiving input speech from the user; generating acoustic features of the input speech using an apparatus with at least one hardware-implemented processor; comparing the generated input speech acoustic features to acoustic models associated with words in the grammar to generate a hypothesis; comparing the hypothesis with at least one expected response in the set to determine if the hypothesis matches the at least one expected response in the set; if the hypothesis matches the at least one expected response in the set, adapting at least one acoustic model corresponding to the matched expected response using the acoustic features of the input speech to use the at least one adapted model with future input speech in the speech recognition system, otherwise, not adapting the at least one acoustic model corresponding to the expected response. initiating a speech dialog with at least one point in the dialog where there is a grammar of possible responses and a set of at least one expected response, wherein the set is a subset of the grammar, the set includes the response or responses expected to be uttered by a user at the at least one point in the speech dialog, and the set of at least one expected response for the at least one point is known in the speech recognition system before receiving input speech from the user; the set is a subset of the grammar, the set includes the response or responses expected to be uttered by a user at the at least one point in the speech dialog, and the set of at least one expected response for the at least one point is known in the speech recognition system before receiving input speech from the user; progressing through the speech dialog until arriving at the at least one point; receiving input speech from the user; generating acoustic features of the input speech using an apparatus with at least one hardware-implemented processor; comparing the generated input speech acoustic features to acoustic models associated with words in the grammar to generate a hypothesis; comparing the hypothesis with at least one expected response in the set to determine if the hypothesis matches the at least one expected response in the set; if the hypothesis matches the at least one expected response in the set, adapting at least one acoustic model corresponding to the matched expected response using the acoustic features of the input speech to use the at least one adapted model with future input speech in the speech recognition system, otherwise, not adapting the at least one acoustic model corresponding to the expected response.