Question: The representation used in Chaplot et al. 2020b and HLSM also involves object positions, heights, and so on. What makes the proposed method different so that it claims to be affordance-aware?
Evidence: Recent progress in Embodied Artificial Intelligence, spans both simulation environments;;;; and sophisticated tasks;;. Our work is most closely related to research in language-guided task completion, Neural SLAM, and exploration.
Language-Guided Task Completion. is a benchmark that enables a learning agent to follow natural language descriptions to complete complex household tasks. The agent's goal is to learn a mapping from natural language instructions to a sequence of actions for task completion in a simulated 3D environment. Various modeling approaches have been proposed falling into roughly two families of methods. The first focuses on learning large end-to-end models that directly translate instructions to low-level agent actions;. However, these agents typically suffer from poor generalization performance, and are difficult to interpret. Recently, hierarchical approaches Zhang & Chai (2021); have attracted attention due to their better generalization and interpretability. We also adopt a hierarchical structure, focusing on affordance-aware navigation thereby achieving significantly better generalization than all existing approaches.
Neural SLAM and Affordance-aware Semantic Representation. Neural SLAM, constructs an environment semantic representation enabling map-based long-horizon planning. However, these are tested in pure navigation tasks instead of complex household tasks, and does not consider affordance;, which is required for tasks involving both navigation and manipulations. In, the authors utilize SLAM for 3D environment reconstruction in language-guided task completion. Their approach relies heavily on accurate depth prediction (less robust in unseen environments). Instead, we propose a waypoint-oriented representation which associates each object with the locations on the floor from where the agent can interact with the object. Furthermore, different from the 2D affordance map in that directly predicts affordance type, our semantic representation supports more fine-grained control of the robot's position and pose, which facilitates significantly better generalization. The approach in assumes direct access to the ground truth depth information (not available in our setup) and the method in only focuses on pure navigation problems. approaches reward visiting all navigable areas by searching in a task-agnostic manner. In contrast, we propose a multimodal exploration approach utilizing egocentric visual input, language instructions, and memory of explored areas to reduce task-specific uncertainty of points of interest (areas important to complete the task). We show this to be more efficient, leading to more effective map prediction and robust planning.
However, these are tested in pure navigation tasks instead of complex household tasks, and does not consider affordance Qi et al. (2019); Nagarajan & Grauman (2020); Xu et al. (2020), which is required for tasks involving both navigation and manipulations. In Blukis et al. (2021), the authors utilize SLAM for 3D environment reconstruction in language-guided task completion. Their approach relies heavily on accurate depth prediction (less robust in unseen environments). Instead, we propose a waypoint-oriented representation which associates each object with the locations on the floor from where the agent can interact with the object.****Figure 2: Illustration of the affordance-aware semantic representation used in our framework. (left) A top-down view of an indoor scene used in ALFRED (this view is not available to the agent at test time). (middle) A visualization of the corresponding semantic map; only shown are the side table (in green), counter top (in blue) and the navigable area (in red). Two waypoints (drawn in white stars and arrows) are displayed for side table (∈ large object) and lettuce (∈ small object), respectively. (right) While the waypoint for side table is computed from the predicted map, the waypoint for lettuce is obtained by searching among all exploration steps. The visual observation and its mask prediction on the waypoint for lettuce are shown. For reference, the high-level goal description of the task used in this example is “pick up the lettuce and place it on a table”.****ALFRED supports more than 100 types of objects (one way in which it mimicks real-world complexity), and we propose to handle small and large objects differently. We detail our design below and give an example in Figure 2. Further implementation details are available in the Appendix.
Answer: Our major difference is that we devise a waypoint-oriented representation instead of trying to estimate the exact 3D coordinates of each object in the scene. Each object in our representation is constructed by the associated location from where the agent can interact with it (illustrated in Figure 2). This is easier to obtain and better suited for the downstream object interaction tasks than directly estimating the locations of all objects. To acquire such a map we handle large and small objects differently as described in Sec. 4.2.