Question: Why is the explored area considered an extra modality?
Evidence: Affordance-aware navigation is a major challenge in solving complex and long-horizon indoor tasks such as ALFRED with both navigation and object interactions. Specifically, given each object of interest in the scene, the agent is required to not only find and approach it but also end up at a pose (x, y, r, h), that is feasible for subsequent interactions with the object. For instance, to open a fridge, the robot should approach the fridge closely enough (so the door is within reach), look at it (so that the fridge is in the field of view), and leave enough room to open the door. To solve a long-horizon task involving multiple navigation and object interaction subgoals, it is natural to use an explicit semantic map, either 2D or 3D, of the environment (similar to Neural Active SLAM Chaplot et al. (2020a)), together with model-based planning (e.g. as in). This line of work tends to generalize better than models that directly learn mappings from human instructions to navigation & interaction actions (e.g., E.T.). With perfect knowledge of the environment, it is possible to achieve (nearly) perfect performance. In practice, however, the semantic map acquired at inference time is usually far from ideal, primarily due to Incompleteness (missing information due to insufficient exploration of the scene) and Inaccuracy (erroneous object location prediction on the map, especially for small objects).
To improve exploration performance, we propose a multimodal module that, at each step, predicts an exploration action a ∈ {MoveAhead, RotateLeft, RotateRight} by taking visual observations & actions in the past, step-by-step language instructions, and the explored area map which indicates where the agent has visited. We show that, compared to existing model-based approaches on ALFRED (e.g., which applies random exploration), our use of lowlevel language instructions leads to more efficient exploration. The proposed exploration module operates at the subgoal level and only predicts exploration actions (in contrast to E.T. which directly predicts actions for the entire task). The extra modality (the explored area) facilitates exploration by providing the agent with explicit spatial information. We illustrate the exploration module in Figure, elaborate its details in Section 4.3, and empirically demonstrate its advantages in Section 5.
To improve exploration performance, we propose a multimodal module that, at each step, predicts an exploration action a ∈ {MoveAhead, RotateLeft, RotateRight} by taking visual observations & actions in the past, step-by-step language instructions, and the explored area map which indicates where the agent has visited.****The Extra Modality: Explored Area The extra modality introduced in our multimodal exploration module essentially tracks the action history explicitly and geometrically.
Answer: The “explored area” used as input to the Transformer module for the exploration phase helps to keep track of the area visited by the agent in a geometric and explicit way. The ablation studies (see Sec. 5.4 & Tab. 4) empirically show the advantage of this extra modality.