A method implemented using one or more processors, comprising: receiving natural language input from a user as part of an ongoing human-to-computer dialog between the user and an automated assistant operated by one or more of the processors, wherein one or more topics raised previously during the ongoing human-to-computer dialog are stored in memory as part of a contextual data structure associated with the ongoing human-to-computer dialog; contextually filtering a subset of one or more grammars associated with the previously-raised one or more topics from a superset of grammars associated with a plurality of respective topics, wherein the contextually filtering is based on one or more respective measures of relevance of the previously-raised one or more topics to the ongoing human-to-computer dialog, wherein the measure of relevance associated with each given topic of the one or more topics is determined based at least in part on a count of turns of the ongoing human-to-computer dialog since the given topic was last raised, wherein the count of turns since the given topic was last raised is inversely related to relevance of the given topic to the ongoing human-to-computer dialog; subsequent to the contextually filtering, parsing the natural language input based on the contextually-filtered one or more grammars to generate one or more parses; generating, based on one or more of the parses, a natural language response; outputting the natural language response to the user using one or more output devices; identifying one or more new topics raised by one or more of the parses or the natural language response; adding the identified one or more new topics to the contextual data structure; and elevating the measure of relevance associated with a given topic of the previously-raised topics to the ongoing human-to-computer dialog based on a measure of semantic relatedness between the given topic and the one or more new topics. receiving natural language input from a user as part of an ongoing human-to-computer dialog between the user and an automated assistant operated by one or more of the processors, wherein one or more topics raised previously during the ongoing human-to-computer dialog are stored in memory as part of a contextual data structure associated with the ongoing human-to-computer dialog; contextually filtering a subset of one or more grammars associated with the previously-raised one or more topics from a superset of grammars associated with a plurality of respective topics, wherein the contextually filtering is based on one or more respective measures of relevance of the previously-raised one or more topics to the ongoing human-to-computer dialog, wherein the measure of relevance associated with each given topic of the one or more topics is determined based at least in part on a count of turns of the ongoing human-to-computer dialog since the given topic was last raised, wherein the count of turns since the given topic was last raised is inversely related to relevance of the given topic to the ongoing human-to-computer dialog; subsequent to the contextually filtering, parsing the natural language input based on the contextually-filtered one or more grammars to generate one or more parses; generating, based on one or more of the parses, a natural language response; outputting the natural language response to the user using one or more output devices; identifying one or more new topics raised by one or more of the parses or the natural language response; adding the identified one or more new topics to the contextual data structure; and elevating the measure of relevance associated with a given topic of the previously-raised topics to the ongoing human-to-computer dialog based on a measure of semantic relatedness between the given topic and the one or more new topics.