A computing device comprising: a memory containing machine readable medium storing machine executable code; and one or more processors coupled to the memory and configurable to execute the machine executable code to cause the one or more processors to: generate, by a textual grounding module, an instruction grounding from a training navigation instruction sample for instructing a mobile automated system to navigate an environment in which the mobile automated system is located; generate, by a visual grounding module, a visual grounding from one or more training images observed for the environment as the mobile automated system is navigating therethrough; generate, by a neural model, a progress monitor probability in response to an input of the instruction grounding and the visual grounding; compute an action probability of navigation directions based on a hidden state of the neural model and the instruction grounding; compute a loss objective based at least in part on a first cross entropy between the progress monitor probability and a ground-truth progress label, and a second cross-entropy between the action probability and a ground-truth navigable direction, wherein the loss objective is computed as a linear combination of the first cross-entropy and the second cross-entropy with a weighting parameter adjusting a ratio between the first cross-entropy and the second cross-entropy; and update the neural model based on the loss objective. a memory containing machine readable medium storing machine executable code; and one or more processors coupled to the memory and configurable to execute the machine executable code to cause the one or more processors to: generate, by a textual grounding module, an instruction grounding from a training navigation instruction sample for instructing a mobile automated system to navigate an environment in which the mobile automated system is located; generate, by a visual grounding module, a visual grounding from one or more training images observed for the environment as the mobile automated system is navigating therethrough; generate, by a neural model, a progress monitor probability in response to an input of the instruction grounding and the visual grounding; compute an action probability of navigation directions based on a hidden state of the neural model and the instruction grounding; compute a loss objective based at least in part on a first cross entropy between the progress monitor probability and a ground-truth progress label, and a second cross-entropy between the action probability and a ground-truth navigable direction, wherein the loss objective is computed as a linear combination of the first cross-entropy and the second cross-entropy with a weighting parameter adjusting a ratio between the first cross-entropy and the second cross-entropy; and update the neural model based on the loss objective. generate, by a textual grounding module, an instruction grounding from a training navigation instruction sample for instructing a mobile automated system to navigate an environment in which the mobile automated system is located; generate, by a visual grounding module, a visual grounding from one or more training images observed for the environment as the mobile automated system is navigating therethrough; generate, by a neural model, a progress monitor probability in response to an input of the instruction grounding and the visual grounding; compute an action probability of navigation directions based on a hidden state of the neural model and the instruction grounding; compute a loss objective based at least in part on a first cross entropy between the progress monitor probability and a ground-truth progress label, and a second cross-entropy between the action probability and a ground-truth navigable direction, wherein the loss objective is computed as a linear combination of the first cross-entropy and the second cross-entropy with a weighting parameter adjusting a ratio between the first cross-entropy and the second cross-entropy; and update the neural model based on the loss objective.