A method implemented by one or more processors, the method comprising: causing a portable computing device to operate in a first operating mode, wherein, while the portable computing device is operating in the first operating mode, an automated assistant is invoked by the portable computing device in response to the detection of a spoken trigger phrase, and the usage of the output of a proximity sensor is restricted from being used by the portable computing device to invoke the automated assistant; processing contextual data that characterizes a current context of the portable computing device, wherein the contextual data is generated based on sensor output from one or more sensors of the portable computing device, comprising determining, based on processing the contextual data, that an environment of the portable computing device is receiving audio that is being projected by another computing device, wherein the automated assistant is accessible via the other computing device; causing the portable computing device to operate in a second operating mode in response to the processing of the contextual data satisfying one or more criteria, comprising determining that the environment of the portable computing device is receiving audio that is being projected by the other computing device, wherein, while the portable computing device is operating in the second operating mode, the automated assistant is invoked by the portable computing device in response to the proximity sensor indicating that a physical gesture was performed by a user; and causing, in response to the proximity sensor indicating that the physical gesture was performed by the user, a touch screen display of the portable computing device to provide an interactive graphical element that includes natural language text, wherein the interactive graphical element is configured to modify an operation of the other computing device in response to the user providing a different gesture for selecting the interactive graphical element. causing a portable computing device to operate in a first operating mode, wherein, while the portable computing device is operating in the first operating mode, an automated assistant is invoked by the portable computing device in response to the detection of a spoken trigger phrase, and the usage of the output of a proximity sensor is restricted from being used by the portable computing device to invoke the automated assistant; processing contextual data that characterizes a current context of the portable computing device, wherein the contextual data is generated based on sensor output from one or more sensors of the portable computing device, comprising determining, based on processing the contextual data, that an environment of the portable computing device is receiving audio that is being projected by another computing device, wherein the automated assistant is accessible via the other computing device; causing the portable computing device to operate in a second operating mode in response to the processing of the contextual data satisfying one or more criteria, comprising determining that the environment of the portable computing device is receiving audio that is being projected by the other computing device, wherein, while the portable computing device is operating in the second operating mode, the automated assistant is invoked by the portable computing device in response to the proximity sensor indicating that a physical gesture was performed by a user; and causing, in response to the proximity sensor indicating that the physical gesture was performed by the user, a touch screen display of the portable computing device to provide an interactive graphical element that includes natural language text, wherein the interactive graphical element is configured to modify an operation of the other computing device in response to the user providing a different gesture for selecting the interactive graphical element. wherein, while the portable computing device is operating in the second operating mode, the automated assistant is invoked by the portable computing device in response to the proximity sensor indicating that a physical gesture was performed by a user; and causing, in response to the proximity sensor indicating that the physical gesture was performed by the user, a touch screen display of the portable computing device to provide an interactive graphical element that includes natural language text, wherein the interactive graphical element is configured to modify an operation of the other computing device in response to the user providing a different gesture for selecting the interactive graphical element.