A method implemented by one or more processors, comprising: receiving textual input, the textual input being based on user interface input generated by a user via one or more user interface input devices of a computing device of the user, wherein the user interface input is generated by the user as part of a dialog between the user and an automated assistant implemented by one or more of the processors, whereby the automated assistant is interacted with by the user; determining user state information for the user , wherein the user state information identifies a state of the user, is in addition to the textual input, and is based on sensor data generated by the computing device or an additional computing device of the user; generating reply content to the textual input , the reply content based on both the textual input and the user state information; and providing the reply content as part of the dialog between the user and the automated assistant in response to the user interface input , wherein the reply content is provided for inclusion in the dialog in response to the textual input, and wherein the reply content is provided for presentation via one or more user interface output devices, where generating the reply content based on both the textual input and the user state information comprises: providing the textual input to a text generation engine ; receiving an initial textual output from the text generation engine; and generating the reply content by modifying the initial textual output based on the user state information. receiving textual input, the textual input being based on user interface input generated by a user via one or more user interface input devices of a computing device of the user, wherein the user interface input is generated by the user as part of a dialog between the user and an automated assistant implemented by one or more of the processors, whereby the automated assistant is interacted with by the user; determining user state information for the user , wherein the user state information identifies a state of the user, is in addition to the textual input, and is based on sensor data generated by the computing device or an additional computing device of the user; generating reply content to the textual input , the reply content based on both the textual input and the user state information; and providing the reply content as part of the dialog between the user and the automated assistant in response to the user interface input , wherein the reply content is provided for inclusion in the dialog in response to the textual input, and wherein the reply content is provided for presentation via one or more user interface output devices, where generating the reply content based on both the textual input and the user state information comprises: providing the textual input to a text generation engine ; receiving an initial textual output from the text generation engine; and generating the reply content by modifying the initial textual output based on the user state information. providing the textual input to a text generation engine ; receiving an initial textual output from the text generation engine; and generating the reply content by modifying the initial textual output based on the user state information. <EOS>