Narrowing the Gap between Vision and Action in Navigation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-06 18:25:13

Abstract: The existing methods for Vision and Language Navigation in the Continuous Environment (VLN-CE) commonly incorporate a waypoint predictor to discretize the environment. This simplifies the navigation actions into a view selection task and improves navigation performance significantly compared to direct training using low-level actions. 
However, the VLN-CE agents are still far from the real robots since there are gaps between their visual perception and executed actions. 
First, VLN-CE agents that discretize the visual environment are primarily trained with high-level view selection, which causes them to ignore crucial spatial reasoning within the low-level action movements. Second, in these models, the existing waypoint predictors neglect object semantics and their attributes related to passibility, which can be informative in indicating the feasibility of actions.
To address these two issues, we introduce a low-level action decoder jointly trained with high-level action prediction, enabling the current VLN agent to learn and ground the selected visual view to the low-level controls. Moreover, we enhance the current waypoint predictor by utilizing visual representations containing rich semantic information and explicitly masking obstacles based on humans' prior knowledge about the feasibility of actions.  Empirically, our agent can improve navigation performance metrics compared to the strong baselines on both high-level and low-level actions.

Main paper: 
 introduction: 
 Vision and Language Navigation (VLN) is a problem setting that requires the agent to navigate in a photo-realistic environment following natural language instructions. Most VLN-related works model the navigation action space as high-level viewpoint selection by discretizing the visual environment into images based on an underlying navigability graph, named VLN-DE [5, 11] . Subsequently, the navigation task is extended to the continuous environment (VLN-CE) [23] where the agent navigates with low-level controls (LEFT/RIGHT/FORWARD/STOP). The VLN-CE agents are Instruction: Walk towards the pool table. closer to the challenges encountered by real-world robots, although their performance is inferior to the VLN-DE agents.InitialTo improve the navigation performance, existing VLN-CE agents employ an offline-trained waypoint predictor, which takes panoramic RGB and depth images as input and generates a local navigability graph centered around the agent at each navigation step [4, 15, 20] . The generated graphs have similar functionality to the connectivity graph in the VLN-DE setting, providing navigable viewpoints. In this case, VLN-CE agents can also use high-level actions (viewpoint selection) to enhance navigation performance. Nevertheless, despite the waypoint predictor's crucial role in existing VLN-CE agents, we have observed two limitations that hinder the navigation agent's understanding of the spatial environment. First, the current VLN-CE navigation agents, equipped with the waypoint predictor, are primarily trained to focus on high-level viewpoint selection and rely on an offline controller for low-level action movements within the environment. However, this approach overlooks including low-level actions as part of the training signal. Consequently, the navigation agent misses spatial information embedded in low-level actions, thereby affecting the grounding of different modalities of textual instructions, visual images, and physical spatial motions. Second, existing waypoint predictors mainly focus on visual information such as RGB and depth images, overlooking a thorough exploration of object semantic attributes, which are important for assessing the feasibility of the physical actions, such as recognizing that walls are impassable.To address the two above-mentioned issues in the VLN-CE agents with a waypoint predictor, we first introduce a dual-action module in which the agent selects high-level viewpoints while generating low-level action sequences simultaneously. The high-level actions serve as guidance, facilitating the agent's understanding of the relationships between low-level actions and navigable areas indicated by high-level actions. This enhances the agent's spatial grounding ability to connect action with visual perception and language understanding. Second, to address the issue of the waypoint predictor neglecting object semantic information, we incorporate visual representations with rich object semantics and explicit obstacle masking based on prior knowledge of object passability. As a result, the waypoint predictor will generate more effective viewpoints by leveraging comprehensive pre-training and prior knowledge.Specifically, we incorporate a Transformer-based decoder into the VLN-CE agent to generate low-level action sequences at each navigation step. We formulate the low-level action generation as a text sequence generation task. As shown in Fig. 1 (c), the agent is flexible in selecting a navigable viewpoint (green circle) from the visual environment or navigating through a low-level action sequence (path with black circle). The low-level action decoder and high-level action selection are jointly trained. This approach encourages the agent to ground the selected view to action planning, bridging the learning gap between low-level and high-level actions, eventually benefiting the navigation performance on both levels. Moreover, to improve the waypoint predictor, we use visual representations obtained from Vision-Language Models Pretrained (VLPMs) [32, 33, 43] containing comprehensive object semantics from different modalities. Besides, we mask obstacle areas in images (such as "sofa", "table", and "fireplace". in Fig. 1 In summary, our contributions are as follows. 1. We introduce a dual-action module for the VLN-CE agents, grounding high-level visual perception into low-level spatial actions. This design empowers the agent with the flexibility to select high-level viewpoints and generate low-level action sequences. 2. We enhance the waypoint predictor with visual representations containing rich object semantics and explicit prior knowledge about objects' passability attributes. 3. We adapt our method on several VLN-CE agents. The experimental results demonstrate the effectiveness of our approach in both waypoint predictor, high-level as well as low-level navigation performance. 
related works 2.1 vision and language navigation: 
 Following instructions to navigate in a simulated environment has attracted significant research interest in recent years. There are many navigation datasets introduced, such as indoor navigation R2R [5] , R4R [19] , RXR [24] , outdoor navigation Touchdown [7] , object-finding REVERIE [30] , and dialogue-based navigation CVDN [40] .Early methods formulate the navigation agent using a sequenceto-sequence framework and apply attention mechanism to improve learning across text and vision modalities [5, 27] . Data augmentation methods are proposed to strengthen the agent generalization ability, such as image editing [25, 26] or instruction generation [11, 49] . The Transformer-based models are the recent main trend for the VLN agent focusing on multi-modal representation learning [16, 44] . In this framework, map representation learning [2, 8, 17] , graph-based exploration [10, 42, 50] , and auxiliary pre-training [9, 13, 31, 48, 51] techniques are introduced. However, these agents formulate the navigation in a discrete environment, ignoring the low-level control challenges a real-world robot faces. Our work mainly focuses on improving the agent's performance in the continuous setting.Several continuous environments have been proposed in the realm of embodied AI to simulate photo-realistic scenes, such as Gibson [47] , House3D [46] , and Habitat [36] . R2R-CE [22] dataset is constructed by transferring discrete trajectories in R2R [5] dataset to continuous trajectories using Habitat simulator [36] . Compared to discrete environment navigation, VLN-CE is much closer to realworld navigation since it relies on a limited field of view to infer low-level controls rather than a pre-defined connectivity graph with nodes distributed across accessible spaces within a discrete environment.The VLN-CE is pioneered by [22] , and they introduce a seq2seq baseline model, mapping a set of images during navigation to a sequence of low-level actions. LAW [34] improves the baseline using language-aligned waypoint supervisions. The methods applying semantic map [8, 12, 17] are also explored. For example, WS-MGMap [8] proposes a multi-granularity map and introduces a weakly-supervised auxiliary task to learn a better map representation that reveals object information. However, these methods still demonstrate a huge performance gap compared to the navigation agents in the discrete environments. To bridge the gap between VLN-DE and VLN-CE, waypoint models are proposed for hierarchical visual navigation [4, 20] . Specifically, CWP [15] uses a waypoint predictor to provide accessible directions for the agent to act, and then they apply the efficient models in the VLN-DE to the continuous environments. The recent work significantly improves the waypoint models using better visual representations pre-trained in a larger dataset [43, 44] , graph-based modeling [1, 2] or extra auxiliary tasks [17] . However, there are still gaps between the visual perception and physical action within the current VLN-CE agent. For example, with the waypoint predictor, the navigation agent is primarily trained based on view selection. The navigation performance drops significantly when modeling low-level actions [15] . In addition, the waypoint predictor mainly focuses on visual features, neglecting rich object semantics within images. Our study focuses on enhancing the VLN-CE agent by addressing these two aspects to narrow the gap by grounding visual perception into spatial actions. 
backgrounds 3.1 problem statement: 
 In the VLN-CE problem setting, the navigation agent is required to reach the target location within indoor environments following natural language instructions. Given an instruction, the agent navigates on a 3D mesh of an environment with low-level actions, including RIGHT (15 We also enhance the RGB visual encoder with pre-trained VL representations.action or a pre-defined maximum number of navigation steps have been reached.Our method is designed to be model-agnostic, and we experiment with different VLN-CE agents. In this section, we utilize History Aware Multimodal Transformer (HAMT) in Continuous Environment [9, 43] as the backbone architecture to introduce our method.In the following sections, we first introduce the text and vision encoders. We then present two primary components in the backbone: the waypoint predictor and the navigator. The waypoint predictor is trained offline to generate navigable viewpoints, which are applied to the navigator for view selection. Vision Encoder. At each navigation step, the agent observes 12 RGB images and 12 depth images, which are represented as𝐼 𝑟𝑔𝑏 = {𝐼 𝑟𝑔𝑏 1 , 𝐼 𝑟𝑔𝑏 2 , • • • , 𝐼 𝑟𝑔𝑏 12 } and 𝐼 𝑑 = {𝐼 𝑑 1 , 𝐼 𝑑 2 , • • • , 𝐼 𝑑 12 }, respectively. In the baseline, different RGB vision encoders are used for the waypoint predictor and navigator. The waypoint predictor utilizes ResNet-152 [14] as its vision encoder, pre-trained on the ImageNet dataset [35] , Meanwhile, the navigator uses the InternVideo [43] as the vision encoder, pre-trained on large video-text datasets. Formally, the obtained visual representations of RGB images are denoted as𝑣 𝑟𝑔𝑏 = {𝑣 𝑟𝑔𝑏 1 , 𝑣 𝑟𝑔𝑏 2 , • • • , 𝑣 𝑟𝑔𝑏 12 }.The depth images are fed into DD-PPO ResNet-50 [45] , which is trained for point-goal navigation, represented as𝑣 𝑑 = {𝑣 𝑑 1 , 𝑣 𝑑 2 , • • • , 𝑣 𝑑 12 }. Text Encoder.The textual instructions are input to the navigator to guide the agent to complete the task. The instruction is denoted as𝑤 = {𝑤 1 , 𝑤 2 , • • • , 𝑤 𝑙 },where 𝑙 is the length of the instruction tokens. We use BERT [41] to obtain initial text representation of instruction 𝑤, denoted as𝑋 = [𝑥 1 , 𝑥 2 , • • • , 𝑥 𝑙 ].Waypoint Predictor. We follow the waypoint predictor designed in [15] , which is a multi-layer Transformer with a non-linear classifier. All 12 RGB image representations 𝑣 𝑟𝑔𝑏 and depth image representations 𝑣 𝑑 are concatenated and then input into the waypoint predictor to predict a heatmap of 120 angles-by-12 distances.Each angle is 3 degrees, and distances range from 0.25 to 3.00 meters with an interval of 0.25 meters. The heatmap is represented as a Gaussian distribution with a variance of 1.75m and 15 • to expand the prediction range. The waypoint predictor is pre-trained based on the navigable connectivity graph from MP3D [6] . During inference, non-maximum suppression (NMS) is used to sample 𝐾 neighboring waypoints, which are utilized as the candidate views for the following navigator.Navigator. HAMT is a multimodal Transformer-based navigator that can memorize historical information. HAMT is initially proposed as a VLN-DE agent. With the introduction of the waypoint predictor, its application is extended to the continuous environment [43, 44] . Specifically, HAMT takes textual instruction, panoramic visual images from the current step, and a sequence of historical panoramic images along the navigation trajectory as input. The output is a selected navigable viewpoint. Formally, at each navigation step 𝑡, the waypoint predictor generates 𝐾 candidate views, and their observation representations are denoted as 𝑂 𝑡 . The history representation is denoted as 𝐻 𝑡 . HAMT concatenates history and observation as the vision modality and uses a cross-modal transformer to learn the connection between text presentation 𝑋 and visual representation [𝐻 𝑡 ; 𝑂 𝑡 ]. The model is trained using highlevel viewpoint selection by selecting the highest similarity score between observation encoding 𝑂 𝑡 and <CLS> token of the Transformer, which contains global instruction-trajectory information. After the agent selects a viewpoint, an offline controller is applied to guide the agent to the corresponding position. 
proposed approach: 
 Built upon the above-introduced backbone architecture, we improve the current VLN-CE agent by introducing an obstacle-aware waypoint predictor and dual-action module for the navigator. Fig. 3 shows the main architecture.Object semantics in the visual environment is expected to play an important role in predicting navigable viewpoints, especially in the sense of open and obstacle areas. Objects have attributes that determine whether they should be labeled as passable or impassable. For example, the agent is not supposed to traverse beneath a "table" or on the "bed". However, the current methods mainly leverage visual information from RGB and depth images and neglect further exploration of the object semantics and their attributes related to passibility.To overcome this limitation, we first enhance the current waypoint predictor with vision representation from Vision and Language Pre-trained Models (VLPMs) [32, 33, 43] , which contain much more comprehensive object semantics than ResNet used in the baseline's waypoint predictor. We employ vision representations from different VLPMs to assess their influence on the waypoint predictor's performance. Our results demonstrate that CLIP [32] visual representation can achieve the best result. Please see Table 3 for our detailed analysis. Second, we introduce an obstacle mask mechanism based on semantic segmentation within the visual environment and our prior knowledge about impassable objects. We For each image, we obtain the corresponding obstacle mask based on semantic segmentation. We assign a label of 1 to object areas in the open vocabulary and 0 otherwise. The resulting obstacle masks are represented as 𝑚 = {𝑚 1 , 𝑚 2 , • • • , 𝑚 12 }. Subsequently, we apply obstacle masks to the RGB images and obtain the masked RGB representation, 𝑣 𝑐𝑚 = 𝑣 𝑐 * 𝑚, which is then concatenated with depth visual representation 𝑣 𝑑 . This combined representation is input to the waypoint predictor to generate views at each navigation step.We train the waypoint predictor with enhanced visual representation and obstacle-masked image. Then, we employ it in the navigator for offline usage to generate navigable views. The details of the navigator are explained in the following section.There are two types of actions in the existing VLN-CE navigator to select: high-level and low-level actions. The current techniques typically model the prediction of these two types of actions independently.High-level Action is to select a view based on the similarity between the observation 𝑂 𝑡 and the hidden states from the crossmodal Transformer, which is represented as follows,EQUATIONwhere ℎ cls is the <CLS> token representation from navigator at step 𝑡, and 𝑝 ℎ 𝑡 is the probability of high-level action. Once the most similar viewpoint is selected, the agent employs an offline controller to navigate to the corresponding position. While highlevel actions effectively boost navigation performance, the training mechanism primarily focuses on view selection, neglecting the spatial information in the low-level action sequence. Additionally, it is challenging for a real-world robot to navigate to a precise angle and distance in a realistic environment. Real-world robots typically operate with very limited action sets, such as FORWARD 0.25m.Low-level Action is to generate low-level action directly. The approach is to use a non-linear classifier to predict an action class at each navigation step. Then, a controller is applied to execute action. Formally, the prediction for low-level actions is performed as follows,EQUATIONwhere 𝑊 𝑐 projects the <CLS> token representation to four low-level actions, and 𝑝 𝑙 𝑡 is the probability of low-level actions. While lowlevel actions are closer to real-world robotic behavior, directly modeling the agent to generate such actions results in a cost-training process. This is because the episodes for low-level actions are around 10 times longer than high-level actions (around 56 steps for low-level action steps, compared to 4 -6 for high-level episodes). Additionally, while there are methods [15, 17] attempting to train the navigation agent with a low-level action classifier directly, the performance drops substantially compared to the method using an offline controller.Dual-Actions. We enable the VLN-CE agent to navigate simultaneously using high-level and low-level actions. Built upon the existing VLN-CE agent that predicts high-level actions, we introduce a decoder to generate the corresponding low-level action sequence simultaneously. Specifically, instead of generating one low-level action using an action classifier at each navigation step, our agent is trained to generate a low-level action sequence. We formulate low-level action prediction as the textual sequence generation task. As shown in Fig. 4 , we introduce a Transformer-based text decoder to generate the low-level action sequences with the prompt of "low actions:". At each navigation step, the agent selects a high-level view, and, at the same time, it is trained to generate the corresponding low-level sequence of action tokens auto-repressively. We input the text decoder with <CLS> representation from the navigator and textual prompt representation. The training objective is to maximize the likelihood of the next low-level action token. The following equation shows the loss of generating 𝑗-th token in the action sequence.EQUATIONwhere 𝑚 is the length of the sequence. We jointly train low-level action decoders and high-level selection. The labels for the low-level action sequence are obtained from the heading and distance differences between the initial view and the selected views in the high-level action. When the agent selects a high-level view at each navigation step, the corresponding lowlevel sequence label is created. Specifically, we initially calculate the degree difference in headings and divide it by 15 to determine the number of rotation steps the agent takes. The direction (LEFT or RIGHT) is determined based on the smaller rotations. A similar process is applied to distance: we calculate the distance between the start point and the selected view and divide it by 0.25. We ignore the remaining if the heading or distance is not perfectly divisible. For instance, as shown in Fig. 4 , the distance between the start point and the target view is 0.33m, and the low action is just one FORWARD (0.25m). 
training and inference for navigator: 
 For the high-level action, we train the navigator with a mixture of Imitation Learning (IL) and Reinforcement Learning (RL) [39] . It consists of the cross-entropy loss of the predicted probability distribution against the ground-truth action and a sampled action from the predicted distribution to learn the designed rewards. In summary, the navigation loss is as follows,EQUATION)where 𝜆 is the hyper-parameter to balance the two components, 𝛼 ℎ 𝑡 is the teacher action for IL that is closest to the next viewpoint, and 𝛼 𝑠 𝑡 is a sample action for RL. The joint training objective for our high-level action navigator with the low-level action decoder is as follows, L = L ℎ𝑖𝑔ℎ + 𝐿 𝑙𝑜𝑤 .(EQUATIONDuring inference, in terms of navigating with high-level action, we use greedy search to select an action with the highest probability at each navigation step to generate a trajectory. For navigation with low-level action, we provide the text prompts "low actions:" to the low-level action decoder and utilize a beam search approach to generate the low action sequence. Then, we apply post-processing to separate all actions and use a controller to execute each action. For the next navigation step, the agent relies on the previous low-level action results, even though it is trained based on view selection. 
experiments 6.1 dataset and evaluation metrics: 
 VLN-CE uses the Habitat Simulator [29, 37, 38] to render environment observations based on the MP3D dataset [6] , including 61 environments for training, 11 for unseen validation and 18 for testing. The scenes in the validation unseen and test unseen sets differ from those in the training set. Three main metrics are used to evaluate navigation performance. 1) Success Rate (SR) [5] is to evaluate wayfinding performance to indicate whether the agent arrives at the destination. 2) Success Rate Weighted Path Length (SPL) [3] normalizes success rate by trajectory length. 3) normalized Dynamic Time Warping (nDTW) [18] is used to measure the fidelity between the predicted and the ground-truth trajectories.The waypoint predictor is trained on a single NVIDA RTX GPU for 20 hours, while the navigator is trained on 6 NVIDA RTX GPUs for 2.5 days. We adopt the model architecture and implementation from [9, 43] based on the PyTorch framework [28] and the Habitat Simulator [36] . The RGB image size is 224 and the depth image size is 256. CLIP ViT-B/16 is used as visual input for the waypoint predictor. We train the waypoint predictor with 300 epochs with a batch size of 8 and a learning rate of 1𝑒 -6. We train the navigator 10000 iterations with the batch size of 6 and the learning rate of 3𝑒 -5. The 𝜆 in Eq. 4 is 0.75. For the low-level action generator, we set the maximum generated length as 30 and the beam search size as 3. The end token is the comma.We evaluate our method on top of three VLN-CE agents: WP-VLN-BERT [15] , WP-HAMT [43] , and ETPNav [2] . These VLN-CE agents are all Transformer-based models and employ a waypoint predictor to discretize the visual environment for navigation in the continuous setting. WP-VLN-BERT uses an implicit state representation to store historical information, whereas WP-HAMT improves this WP-VLN-BERT by storing historical panoramic images from the High-Level Action Performance. Table 1 presents the navigation performance using high-level actions on the validation unseen and test unseen sets. All VLN-CE agents in Table 1 employ a waypoint predictor to generate navigable views and are trained with the high-level view selection. Our method improves high-level action performance for all baseline agents. Specifically, we can improve WP-VLN-BERT almost 2% on all navigation metrics on both validation and test unseen sets, as shown in row#6. WP-HAMT utilizes visual representation from InternVideo [43] to strengthen the model's performance. We mainly compare with InternVideo base weights because of the computation cost limitation. In terms of this baseline, we can improve the navigation performance of the baseline, especially 3% of the success rate on the test unseen (row#8). In addition to enhancing the standard Transformer-based navigation agent, our method can also increase the success rate of the graph-based agent ETPNav, as shown in row#10.Low-Level Action performance. Table 2 shows the navigation performance with low-level movement actions. The existing methods mainly compare the results of the validation unseen for low-level actions. The models in row#1 to row#3 are based on LSTM architecture to frame the navigation as a sequence-to-sequence task and predict low-level actions directly. The models from row#4 to row#6 are models using a waypoint predictor. They add an action classifier to the navigator and train the model to select one lowlevel action at each navigation step. Another noticeable difference between the models from row#1 to row#3 and others is that the agent in these methods can only observe the current view rather than the whole panoramic view at each navigation step. As shown in Table 2 , we observe that some Transformer-based navigators with much more powerful pre-trained visual representations and complex model architecture, such as the approaches in row#5 and row#6, their low-level action navigation performance could not compete with LSTM-based models (row#1 to row#3). Specifically, for the baseline model WP-VLN-BERT, although our method can significantly improve it (row#7), it is still far behind LSTM-based models. However, we can achieve SOTA after applying our method on WP-HAMT and ETPNav, as shown in row#9 and row#10, respectively. It is worth noting that the majority of VLN-CE agents do not report their low-level action performance. To address this, we follow the method of low-level action prediction in WP-VLN-BERT to add a non-linear classifier on top of WP-HAMT [43] to adapt it to low-level action prediction. In general, our method's performance is aligned with the VLN-CE navigator's performance. This correlation is expected, as the low-level action sequence is trained using the hidden state representation from the corresponding baseline, and stronger representations yield better performance when training low-level actions.In this section, we conduct an ablation analysis of the waypoint predictor and for waypoint predictor and the effectiveness of different components in our method. Waypoint Predictor Performance. The waypoint predictor is trained offline to generate navigable viewpoints. We enhance the baseline waypoint predictor from the aspects of stronger visual representations and explicit object masks, and Table 3 shows our results on R2R-CE validation unseen set. The main metrics to evaluate the waypoint predictor's performance are as follows: |Δ| measures the difference in the number of target waypoints and predicted waypoints. %Open is the ratio of predicted waypoints in open space. 𝑑 𝑐 and 𝑑 𝐻 are the Chamfer and Hausdorff distances, respectively, to measure the distance between point clouds. We experiment with visual representations from different Vision and Language Pre-trained Models (VLMs) and test their influence on the performance of the waypoint predictor. As shown in Table 3 , the waypoint predictor achieves the best performance when utilizing CLIP vision representations. However, we cannot conclude that more powerful vision representations lead to better waypointpredicting performance since the representations from InterVideo seem to hurt the waypoint predictor. This result suggests that different pre-trained visual encoders possess varying capacities to influence the agent's ability to recognize open and obstacle areas. The better result is achieved when both CLIP representation and our designed obstacle mask are applied. Compared to ResNet (row#1), CLIP improves the open area prediction by about 3%, demonstrating that rich semantics in visual representation in CLIP aids the waypoint predictor in learning the open and obstacle objects. After applying our designed obstacle mask, the accuracy of open area prediction gained an additional improvement of 2%, emphasizing the effectiveness of the prior knowledge in encouraging the waypoint predictor to better focus on open area spaces.Different Components. The results of the ablation study in Table 4 demonstrate the influence of each component of our proposed method on both high-level and low-level navigation. The components in our method include dual-action for the navigator and the enhanced waypoint predictor with CLIP visual representations and the obstacle mask. The analyzed navigator is WP-HAMT, which uses visual representation obtained from InternVideo base weights. We report results on the R2R-CE validation unseen dataset. In row#1, we integrate the low-level action decoder with the baseline navigator and jointly train it with high-level actions, and the low-level action navigation performance is significantly improved (about 4% on SPL). In row#2, we train the waypoint predictor with only CLIP representations and apply it to the baseline navigator without dual-action training. Notably, the enhanced waypoint predictor already contributes to better high-level navigation performance, indicating that CLIP's rich object semantic representation boosts overall navigation. Row#3 shows the effectiveness of the obstacle mask in enhancing the SPL for high-level action. In row#4, we train the waypoint predictor with both CLIP and obstacle mask, and we apply this enhanced predictor to the navigator with the dual-action module. We achieve the best results in this setting. Compared to row#3, we conclude that the spatial information incorporated into the low-level action benefits the high-level viewpoint selection. Similarly, compared to row#1, enhanced waypoint prediction not only enhances high-level viewpoint selection but also benefits low-level action generation.In this section, we provide a qualitative analysis from the perspectives of low-level actions and obstacle masks.Low-Level Actions Generation. In Fig. 5 (a), we show an example of our generated low-level action sequences that lead the agent to the destination. However, we have observed cases where the agent generates a low-level action sequence that reaches the destination but does not fully follow the instruction, as shown in Fig. 5 (b ). This issue also occurs in other VLN-CE agents when modeling low-level action predictions. We assume the reason is the inherent challenges in building the VLN-CE dataset. It transfers the instructions and trajectories from VLN-DE. When the simulator executes the low-level actions, especially rotations, there is no human evaluation process to confirm whether the low-level actions align with the instruction. The actions are generated based on minimal required rotations when the selected view is given to the simulator. Training the agent to learn low-level actions in these scenarios is challenging compared to directly training with view selection.Open-area Vocabulary. In Table 5 , we provide an analysis of object semantics and their relation with the performance of the waypoint predictor. We select open-area vocabularies based on our prior knowledge. The baseline we used is the waypoint predictor trained with CLIP visual representations. Given the semantic segmentation from the simulated environment, we mask other object areas except the semantic areas in open-area vocabularies. Then, we input the masked image into the waypoint predictor. The experimental results demonstrate that different object semantics show varying influences on the waypoint predictor. For instance, the %open is low when we mask objects other than "door", indicating the presence of closed or blocked doors (row#3). We can get the best results when our open-area vocabularies contain "floor", "stairs", and "doors" (row#4).Qualitative Examples for Obstacle Mask. In Fig. 6 , we show an example to demonstrate the different generated waypoint heatmaps between an RGB image with obstacle mask (Fig. 6 (2)) and without it (Fig. 6 4 , the obstacle mask ultimately contributes to navigation performance. 
conclusion: 
 In this work, we narrow the gap between the vision and action of the current VLN-CE agents mainly in two aspects. we introduce a dualaction module that enables the current VLN-CE agent, equipped with the waypoint predictor, to jointly train for both high-level and low-level actions simultaneously. This joint optimization encourages the agent to learn to ground the high-level visual perception and view selection into physical actions and spatial motions. Second, we enhance the existing waypoint predictor by incorporating rich object semantic representations and knowledge about object attributes. This helps the model to understand the feasibility of actions better. We conduct comprehensive experiments and analyses to demonstrate the effectiveness of our proposed method. 
 
 Appendix: We mainly summarize the following limitations. First, integrating the low-level action decoder into the current VLN-CE agent leads to increased computational costs compared to the baseline. However, the significant improvement in low-level action performance partially justifies this tradeoff. Second, although our designed dualaction module improves the low-level action navigation, the performance gap still remains compared to high-level action performance. 


Decision: This paper proposes to tackle the issues of existing vision and language navigation methods in ignoring low-level spatial reasoning and high-level object semantics. To this end, an obstacle-aware waypoint predictor and a dual-action module for selecting high-level viewpoint and low-level action sequences are introduced. These modules are combined with several agents for evaluation and the experimental results show they indeed bring performance gains.

The pre-rebuttal scores of this paper were mixed, including one weak reject, one borderline accept, and one weak accept. The major concern from the reviewers was about the clarification of the proposed method, which was addressed in the rebuttal. The post-rebuttal ratings were increased to two weak accepts and one borderline accept. The AC also agrees with the reviewers’ assessments and recommends accepting the paper.

Reviews: review: paper_summary: paper_summary: This paper focuses on solving the gap between visual perception and executed actions in VLN-CE tasks. By introducing a dual-action module for the VLN-CE agents, grounding high-level visual perception into low-level spatial actions and enhancing the waypoint predictor with visual representations, this paper increases the high-level navigation performance and low-level navigation performance.
strength_weakness: strength_weakness: 1: The proposed dual-action module is novel. It views low-level actions as tokens in NLP and generate low-level action sequence from heading and distance differences between views in high-level action.
2: The paper provides an evaluation of the system with different baselines to showcase the effectiveness of the system.
limitations: limitations: Insufficient evaluation:
1: Real-time performance of the system is not evaluated, considering the paper focuses on mitigating the gap between VLN-CE agent and real robots. Analysis on inference efficiency is persuasive.

2: In Section 6.2 "Implementation Details", the author writes "The waypoint predictor is trained on a single NVIDA RTX GPU for 20 hours, while the navigator is trained on 6 NVIDA RTX GPUs for 2.5 days.". It will be more Intuitive if the author can list the GPU model, or at least the computing power.

Some points were not made clear: How the labeling process of the low-level action sequence handling the obstacles between initial view and the selected views in the high-level action? Since the labels only use the heading and distance differences between views, thus the low-level actions cannot avoid obstacles while moving.
paper_summary: paper_summary: They think the VLN-CE agents are still far from the real robots since there are gaps between their visual perception and executed actions. To address the problem in the VLN-CE agents with a waypoint predictor, They introduce a dual-action module in which the agent selects high-level viewpoints while generating low-level action sequences simultaneously.  They enhance the waypoint predictor with visual representations containing rich object semantics and explicit prior knowledge about objects’ passability attributes.
strength_weakness: strength_weakness: 1. They introduce a low-level action decoder, enabling the current VLN agent to learn and ground the selected visual view to the low-level controls.

2. They enhance the current waypoint predictor by utilizing visual representations containing rich semantic information and explicitly masking obstacles based on humans’ prior knowledge about the feasibility of actions.
limitations: limitations: 1. They believe the VLN-CE agents are still far from the real robots since there are gaps between their visual perception and executed actions. But now there are also some VLN datasets that use low-level actions, such as Touchdown, map2seq. I don't understand the significance of this paper.
paper_summary: paper_summary: This paper proposes two useful modules to narrow the gap between the vision and action of the current VLN-CE agents. The first module is Obstacle-aware Waypoint Predictor and the second module is the Dual-Action module. Based on the ablation study, both modules are effective.
strength_weakness: strength_weakness: - The design of the Low-Level Action Decoder is creative. This module connects high-level action and low-level action, which narrows the gap between VLN-DE and VLN-CE.
- The experiment results are persuasive.
limitations: limitations: - The reviewer would like to confirm if the high-level performance in Table 1 only is related to the Obstacle-aware Waypoint Predictor.
- Some typos: There is no (a) (b) in Fig 5. The Fig.5 (a) in line 785 and the Fig.5 (b) in line 790 are nonsense.
- The title of the article seems a bit too broad. This work just narrows the gap between high-level action and low-level action, instead of vision and action.