A computer-implemented method, comprising: receiving natural language input from a user as part of an ongoing human-to-computer dialog between the user and an automated assistant operated by one or more processors; selecting one or more grammars to parse the natural language input, wherein the selecting is based on one or more topics stored in memory as part of a contextual data structure associated with the ongoing human-to-computer dialog, wherein each topic stored in the memory as part of the contextual data structure is associated with a measure of relevance of the topic to the ongoing human-to-computer dialog, the contextual data structure comprises an undirected graph comprising a plurality of nodes and a plurality of edges connecting the plurality of nodes, each node of the undirected graph represents a given topic of the one or more topics stored as part of the contextual data structure, and a count of turns of the ongoing human-to-computer dialog since the given topic was last raised, each edge connecting two nodes represents a measure of relatedness between two topics represented by the two nodes, respectively, the measure of relevance associated with each topic is determined based at least in part on the a count of turns and the measure of relatedness; parsing the natural language input based on the selected one or more grammars to generate one or more parses; generating , based on one or more of the parses, a natural language response; outputting the natural language response to the user using one or more output devices; identifying one or more topics raised by one or more of the parses or the natural language response; and adding the identified one or more topics to the contextual data structure. receiving natural language input from a user as part of an ongoing human-to-computer dialog between the user and an automated assistant operated by one or more processors; selecting one or more grammars to parse the natural language input, wherein the selecting is based on one or more topics stored in memory as part of a contextual data structure associated with the ongoing human-to-computer dialog, wherein each topic stored in the memory as part of the contextual data structure is associated with a measure of relevance of the topic to the ongoing human-to-computer dialog, the contextual data structure comprises an undirected graph comprising a plurality of nodes and a plurality of edges connecting the plurality of nodes, each node of the undirected graph represents a given topic of the one or more topics stored as part of the contextual data structure, and a count of turns of the ongoing human-to-computer dialog since the given topic was last raised, each edge connecting two nodes represents a measure of relatedness between two topics represented by the two nodes, respectively, the measure of relevance associated with each topic is determined based at least in part on the a count of turns and the measure of relatedness; parsing the natural language input based on the selected one or more grammars to generate one or more parses; generating , based on one or more of the parses, a natural language response; outputting the natural language response to the user using one or more output devices; identifying one or more topics raised by one or more of the parses or the natural language response; and adding the identified one or more topics to the contextual data structure. <EOS>