A method implemented by one or more processors, the method comprising: receiving image data corresponding to an output of a camera that is integral to a portable client device and is directed at an environment that includes multiple controllable electronic devices (106, 108) when the camera provided the output, wherein the portable client device includes an automated assistant interface that is configured to provide an interface between a user and an automated assistant; determining that a user input was received at the automated assistant interface, wherein the user input is associated with an action for modifying an adjustable setting of a target controllable electronic device of the multiple controllable electronic devices; determining multiple metrics based on processing of the image data, wherein each metric of the multiple metrics characterizes a probability that a corresponding controllable electronic device of the multiple controllable electronic devices is the target controllable electronic device that the user is intending to control via the automated assistant interface; and when a particular metric of the multiple metrics indicates that a particular controllable electronic device of the multiple controllable electronic devices is the target controllable electronic device: determining, based at least on the image data, the adjustable setting of the particular controllable electronic device, and causing, in accordance with the user input received at the automated assistant interface, the adjustable setting of the particular controllable electronic device to be modified. receiving image data corresponding to an output of a camera that is integral to a portable client device and is directed at an environment that includes multiple controllable electronic devices (106, 108) when the camera provided the output, wherein the portable client device includes an automated assistant interface that is configured to provide an interface between a user and an automated assistant; determining that a user input was received at the automated assistant interface, wherein the user input is associated with an action for modifying an adjustable setting of a target controllable electronic device of the multiple controllable electronic devices; determining multiple metrics based on processing of the image data, wherein each metric of the multiple metrics characterizes a probability that a corresponding controllable electronic device of the multiple controllable electronic devices is the target controllable electronic device that the user is intending to control via the automated assistant interface; and when a particular metric of the multiple metrics indicates that a particular controllable electronic device of the multiple controllable electronic devices is the target controllable electronic device: determining, based at least on the image data, the adjustable setting of the particular controllable electronic device, and causing, in accordance with the user input received at the automated assistant interface, the adjustable setting of the particular controllable electronic device to be modified. determining, based at least on the image data, the adjustable setting of the particular controllable electronic device, and causing, in accordance with the user input received at the automated assistant interface, the adjustable setting of the particular controllable electronic device to be modified.