A computing device comprising: a memory containing machine readable medium storing machine executable code; and one or more processors coupled to the memory and configurable to execute the machine executable code to cause the one or more processors to: receive a navigation instruction in the form of text or speech for instructing a mobile automated system to navigate an environment in which the mobile automated system is located; receive visual information for the environment, the visual information comprising one or more images observed for the environment as the mobile automated system is navigated therethrough; generate an instruction grounding based at least in part on the navigation instruction, the instruction grounding identifying which part of the navigation instruction has been completed by the mobile automated system and which part of the navigation instruction is outstanding; generate a visual grounding based at least in part on the visual information, the visual grounding identifying a direction in which the mobile automated system should proceed by determining the part of the instruction that corresponds to the observed images; and using the instruction grounding and the visual grounding, generate an action for the mobile automated system to perform for navigating the environment. a memory containing machine readable medium storing machine executable code; and one or more processors coupled to the memory and configurable to execute the machine executable code to cause the one or more processors to: receive a navigation instruction in the form of text or speech for instructing a mobile automated system to navigate an environment in which the mobile automated system is located; receive visual information for the environment, the visual information comprising one or more images observed for the environment as the mobile automated system is navigated therethrough; generate an instruction grounding based at least in part on the navigation instruction, the instruction grounding identifying which part of the navigation instruction has been completed by the mobile automated system and which part of the navigation instruction is outstanding; generate a visual grounding based at least in part on the visual information, the visual grounding identifying a direction in which the mobile receive a navigation instruction in the form of text or speech for instructing a mobile automated system to navigate an environment in which the mobile automated system is located; receive visual information for the environment, the visual information comprising one or more images observed for the environment as the mobile automated system is navigated therethrough; generate an instruction grounding based at least in part on the navigation instruction, the instruction grounding identifying which part of the navigation instruction has been completed by the mobile automated system and which part of the navigation instruction is outstanding; generate a visual grounding based at least in part on the visual information, the visual grounding identifying a direction in which the mobile automated system should proceed by determining the part of the instruction that corresponds to the observed images; and using the instruction grounding and the visual grounding, generate an action for the mobile automated system to perform for navigating the environment.