A method implemented by one or more processors of a vision-sensor equipped assistant device to facilitate touch-free invocation of an automated assistant, the method comprising: whenever the assistant device is moved, performing an excluded region classification process to classify one or more regions (360, 362) of a field of view of the vision sensor as being likely to contain visual noise and/or unlikely to contain user-provided visual cues, wherein the region classification process includes: obtaining one or more image frames from the vision sensor; analyzing the one or more image frames; and based on the analyzing, classifying the one or more regions of the field of view of the vision sensor as likely to contain visual noise; obtaining one or more subsequent image frames from the one or more vision sensors; analyzing, based on the previously performed excluded region classification process, the one or more subsequent image frames to detect one or more actionable visual cues provided by one or more users, wherein the analyzing the one or more subsequent image frames comprises: weighting a first candidate visual cue detected in the one or more classified regions of the one or more subsequent image frames less heavily than a second candidate visual cue detected outside of the one or more classified regions, or refraining from analyzing the one or more classified regions of the subsequent image frames obtained from the vision sensor; and causing the automated assistant to take one or more responsive actions based on one or more of the detected actionable visual cues. whenever the assistant device is moved, performing an excluded region classification process to classify one or more regions (360, 362) of a field of view of the vision sensor as being likely to contain visual noise and/or unlikely to contain user-provided visual cues, wherein the region classification process includes: obtaining one or more image frames from the vision sensor; analyzing the one or more image frames; and based on the analyzing, classifying the one or more regions of the field of view of the vision sensor as likely to contain visual noise; obtaining one or more image frames from the vision sensor; analyzing the one or more image frames; and based on the analyzing, classifying the one or more regions of the field of view of the vision sensor as likely to contain visual noise; obtaining one or more subsequent image frames from the one or more vision sensors; analyzing, based on the previously performed excluded region classification process, the one or more subsequent image frames to detect one or more actionable visual cues provided by one or more users, wherein the analyzing the one or more subsequent image frames comprises: weighting a first candidate visual cue detected in the one or more classified regions of the one or more subsequent image frames less heavily than a second candidate visual cue detected outside of the one or more classified regions, or refraining from analyzing the one or more classified regions of the subsequent image frames obtained from the vision sensor; and weighting a first candidate visual cue detected in the one or more classified regions of the one or more subsequent image frames less heavily than a second candidate visual cue detected outside of the one or more classified regions, or refraining from analyzing the one or more classified regions of the subsequent image frames obtained from the vision sensor; and causing the automated assistant to take one or more responsive actions based on one or more of the detected actionable visual cues.