Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigation
Authors: Peihao Chen
Dongyu Ji
Kunyang Lin
Runhao Zeng
Thomas H. Li
Mingkui Tan
Chuang Gan
Venue: NeurIPS 2022 Accept (2022)
Keywords: Vision-and-Language Navigation
Map Representation
Weakly-Supervised Learning

Abstract: We address a practical yet challenging problem of training robot agents to navigate in an environment following a path described by some language instructions. The instructions often contain descriptions of objects in the environment. To achieve accurate and efficient navigation, it is critical to build a map that accurately represents both spatial location and the semantic information of the environment objects. However, enabling a robot to build a map that well represents the environment is extremely challenging as the environment often involves diverse objects with various attributes. In this paper, we propose a multi-granularity map, which contains both object fine-grained details (\eg, color, texture) and semantic classes, to represent objects more comprehensively. Moreover, we propose a weakly-supervised auxiliary task, which requires the agent to localize instruction-relevant objects on the map. Through this task, the agent not only learns to localize the instruction-relevant objects for navigation but also is encouraged to learn a better map representation that reveals object information. We then feed the learned map and instruction to a waypoint predictor to determine the next navigation goal. Experimental results show our method outperforms the state-of-the-art by 4.0% and 4.6% w.r.t. success rate both in seen and unseen environments, respectively on VLN-CE dataset. The code is available at https://github.com/PeihaoChen/WS-MGMap.

Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigation
Peihao Chen 16-
Dongyu Ji Kunyang Lin 1.2 Runhao Zeng' Thomas H. Li Mingkui Tan 17f Chuang Gan
'South China University of Technology. Paazhou Laboratory, MIT-IBM Watson AI Lab, *UMass Amherst, SShenzen University, Information Technology R&D Innovation Center of Peking University, Key Laboratory of Big Data and Intelligent Robot, Ministry of Education, (phchencs, dongyuji.jyy gmail.com, mingkuitan @scut.edu.cc
Abstract
We address a practical yet challenging problem of training robot agents to navigate in an environment following a path described by some language instructions. The instructions often contain descriptions of objects in the environment. To achieve accurate and efficient navigation, is critical to build map that accurately repre- sents both spatial location and the semantic information of the environment objects. However, enabling a robot to build a map that well represents the environment ex- tremely challenging as the environment often involves diverse objects with various attributes. In this paper, we propose multi-gran multi-granularity map, which contains both object fine-graine details (e.g., color, texture) and semantic classes, to represent objects more comprehensively. Moreover, we propose weakly-supervisee auxil- iary task, which requires the agent to localize instruction-relevant objects on the map. Through this task, the agent not only learns to localize the instruction--relevnt objects for navigation but also is encouraged to learn better map representation that reveals object information. We then feed the learned map and instruction waypoint predictor to determine the next navigation goal. Experimental results show our method outperforms the rate both in seen and unseen environments, respectively VLN-CE dataset. Code is available at https:: / github. com/PeinaoChen//S-MGG.
by 4.0% and 4.6% w.z.. success
Introduction
Developing a robot that is able to cooperate with humans is of the goals for embodied artificial intelligence. An important ability for such robot is to understand human instructions (e.g., "stop between the gray sofa and wooden table") and navigate the corresponding location. Toward this goal, Anderson ef al. [1] propose a ussio-and-langune agent follow human language instructions navigate in unseen environments. In this paper, we focus on its variant task VLN-CE [32], where the agent navigates in a continuous environment. The agent could only perform low-level actions and perceive RGB-D with limited field of view.
navigation (VLN) task, which requires an
Current dominant methods for this task are designed in an end-to-end manner [26, 32, 40). They perceive the environment implicitly from raw RGB-D images and predict sequence of actions using a recurrent model. These methods attempt to learn mapping, instruction-visitn correspondence, and path planning implicitly, which increases the learning difficulty, To reduce the learning difficulty,
"Equal contribution. CCoresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Instruction: Exit the bedroom and turn right. Walk straight and stop between gray sofa and wooden table.
RGB
Semantics
Semantic Map
Fine- Gra ined Detail
Gualtt-Granularityy Map
Instruction Object
gray sofa, wooden table
RGB
Map
Localizer
Semantics
Encoder
Bed
Map Constructor
Coarse Localiztioon
Sofa class is not involved in map
Ground-truhh
Hard to distinguish two table instances
Map Constructor (Ours)
Object Localization Task
(a) Existing semantic map
(b) Our MGMap and yeakly-supervise object localization task
Existing semantic map (a) can only represent part of environment object classes without attribute details. Our multi- granuliaatll map (b) contains extra fine-grained environment details (e.g., texture, color) and learns to represent diverse objects with detailed attributes through a weakly- supervised object localization task.
Figure I:
modular map-based approaches [6, 19, 27] equip the embodied agent with a semantic map to perceive the environment. This map is built by projecting RGB image segmentation results to an egocentric top-down map (shown in Figure (a)), and reveals the location and semantic class of part of the environment objects.
However, this map is hard to represent all object classes (e.g., sofa is not represented in Figure I (a)). More critically, it can not represent the attributes of objects (e.g., shape, color, texture and material), which are critical for localizing object. As result, it provides insufficient information to discriminate the "long wooden dining table" from two table instances shown in Figure 1 (a). We find that such ambiguity cases appear frequently in indoor environment. Detailed statistical data and visualization examples are shown in Appendix.
In this paper, propose to learn map that represents the semantics and attributes of diverse objects in a supely-supervised manner. We achieve this goal in two steps. First, we augment the existing semantic map to multi-granulant granularity information, namely high-level object semantics recognized by a segmentation model, and low-level object fine-grained, details. To get the low-level details, we follow existing works [4] to project high-dimensions segmentation model latent features to a top-down map. These latent features have proven to contain rich object details such as color, texture, and shape [28, 52].
map shown in Figure (b). This map contains different
Second, to make the map better represent instruction objects using the multi-granularity features described above, we propose an instruction-relevan object localization auxiliary task as shown in Figure/ (b). Specifically, we feed the map representation and instruction objects to a localizer to predict the location of these objects. Instead of manually annotating the object location a map as localization ground trutt, we automatically generate coarse localization ground from instruction- paired data, considering map regions that are close the instruction path as coarse ground-truth From this task, agents learn to localize instruction-relevant objects for finding an instruction-relevart path. More critically, to localize instruction objects, the map encoder is encouraged to reason a map representation that reveals the precise semantics of each map region.
We then feed the learned map and the instruction to a waypoint predictor to determine the next navigation goal. An existing off-the-shelf local policy [47] is used to determine low-level action (e.g., go forward, turn left or right) go for the navigation goal. We name our method as multi- goatui-granularity map (WS-MGMap) for VLN. Experimental results on VLN-CE benchmark dataset show our proposed method outperforms of methods.
To sum up. our main contributions are as follows: 1) We construct multi-granularity map to represent both fine ime-grained details and abstract semantics information of the environment. To our best knowledge, is the first time to introduce multi graluiigaanlrrityy knowledge in map format for VLN task. 2) We propose
suukly--upervised object localization auxiliary task, from which agents learn to leverage multi-granularity information infer discriminative map representation without need for manual map annotation. 3) With WS-MGMapp our agent robustly localizes objects that are incorrectly recognized by segmentation models. On VLN-CE [32], our method improves navigation success rate by 4.0% and 4.6% in seen and unseen environments, respectively.

---

2 Related Works
Vision-and-languaae navigation. VLN task [2] has drawn significant attention in embodied Al domain. Early works focus on data augmentation methods [13, 44], auxiliary tasks [35, 46, 54] and pre-training methods [21, 23, 36] on the discrete environments in which the agents can only perceive from a sparse set of points. However, these methods can not perform well when the agent navigates in continuous 3D environments [25, 15]. To address this issue, Krantz el al. [32] exploits Habitat simulator [42] to convert discrete trajectory paths to continuous trajectories and proposes a continuous VLN setting. Under these settings, Krantz et al. [32] and Raychaudhuri et al. [40] propose the end-to-end method in which the agent takes representation of the visual observation and instructions as input at each time step to predict an action. Another line of works [19, 31] tackles the VLN problems by leveraging the Ianguage-condition
waypoint. Chen et al. [8]. Georgakis
et al. [19] and Irshad et al. [27] build a structured semantic top-down map, which is more relevant to our work. However, those map-based methods are limited in representing all object classes or the attributes of objects, which prevents the agent from effectively grounding instructions in maps. In contrast these works, our method learns a map that represents the semantics and attributes of diverse objects in a weakly-supervised manner.
Map representation for navigation. Perceiving the environment through map representation is helpful for navigation tasks. Previous works propose different methods for building a map, each of them focusing on representing different information on the map. For example, occupancy map indicates whether a point is occupied and provides location information for point-goal navigation and exploration [29, 30, 39, 9, 17, 16, 14]. Topological maps encode the relationship information among different nodes in the environment and have been explored to tackle different navigation tasks [3, 7, 8, 37j. Semantic maps represent the semantic information of the environment and has been successfully applied in object-goal navigation [6, 10, 50J. Concurrent works [19, 27] attempt to leverage semantic map for VINN task. Existing works [24, 4, 22, 11, 20] try to generate deep feature map by projecting high-dimensional features encoded from neural network to top-down map, which is used for localizing robot [24], predicting semantic map [4], navigating to a few target objects [22, 20], and 3D reconstructing objects [11]. However, compared to object-goal navigation, the instructions in VLN involve more object information. It is challenging to ground rich object information in the map described above individually due to their respective limitations. Unlike these maps, we propose a multi cuulti-raauulaitty map that contains complementary granularity information for representing the environment more comprehensively.
Weakly-supervised learning for object localization. Weakly-supervias object localization is a challenging task that requires learning to localize objects given solely category information in both image[55, 34, 53, 51, 48] and video[33, 49]. CAM[53, 51, 48] has been widely used in weakly- supervised object localization by learning a heat map representing the potential of object location. In contrast to this literature, we aim to utilize the weakly-supervised object localization task to learn discriminative map representation that provides environment information for VLN by localizing instraction-rellcant objects.
3
Vision-and-Language Navigation using Multi- Map
3.1 Problem formulation
We consider the vision-and-languaa navigation task in a continuous environment [32] (VLN-CE), where an agent is required to follow a specific path described by the natural language instruction I. Compared with the original VLN task, in VLN-CE settings, agents can not access to predefined navigable graph, so the agent is required to perform navigation through a sequence of low-level actions Á {FORWARD, TURN-LEFT, TURN-RIGHT and STOP). We equip the agent with an RGB-D camera with a limited field of view, capturing a 224 x 224 RGB image R and depth image D at each time step. Although some current works predefined navigable graph [8], continuous-space actions [31] and panoramic cameras [13, 25] to improve the performance, we strictly keep the same settings with VLN-CE [32], which are closer reality and more challenging.
Existing VIN methods [31, 32, 40] attempt to perceive the environment implicitly from RGB- D observation, which often requires a large number of training data. In contrast, humans will
3

---

Multi- Multa-Grandlartty Map Generation
Oocaliiooa
Waypoint Navigattor
Fines Map
N Mub Granularity
Progress Predictor
Progress Loss
Map Encodar
60%
Semantic H Hallucination
Object Localizatio
Local P olicy
Semandic Loss
(Gx, Ay)
lcbal Semantic Map
Localization Loss
Waypoint Predictor
Waypont
stop between gray and wooden
Bi-LSTM
Instruction Fealure
Figure 2: General scheme of WS-MGMap for VLN task. We assemble both fine-grained details and semantic information about environments to build a multi guati-granullritty map. Agents learn leverage such information for representing diverse objects through a veakly-supervisee object localization task. The learned map and instruction are then fed to a waypoint navigator for deciding actions.
build a map-like environment representation explicitly to provide necessary information for VLN. Equipping an agent with the mapping ability like humans is challenging because it is hard to represent diverse environment objects and their attributes in a map. this paper, we solve this challenge from two aspects: 1) we augment a semantic map with extra fine-grained environment details (e.g., texture, color), which are extracted from latent features a pre-trained segmentation model, to build multi- guatii-gaauulriity map. These different granularity features provide necessary information for representing diverse objects and attributes. 2) To make the multi- gualtiigaanllrity map representation more discriminative, we propose weakly-supervised localization auxiliary task for learning correspondence between map representation and instruction objects. With the learned multi-granularity map, feed it and the instruction to waypoint navigator to decide the next navigation action. The general scheme is shown in Figure 2.
3.2 Perceiving environment via multi-granularity map
To better represent environment objects on map, we propose to leverage both fine-grained features and semantic features about objects for building quati-graanulaitty map. We next introduce how to capture these two types of features for map representation.
Fine-grained map. To capture fine-grained environment details, we feed RGB images captured at each time step pre-trained segmentation model (a U-Net [41] in our case). Previous network interpretability works [28, 52] show that latent features from different layers of this model contain different types of fine-graid fine-grained details of objects, i.e., low-level features represent color and texture while high-level features represent object parts. Based on this observation, we use these latent features to represent environment details and project the latent features an egocentric top-down map [4]. We name the projected map as fine-grained map M E mmmxx, where m X m is map size and c 64 the number of channels of the projected latent features. Each map pixel represents 12cm x 12cm space of environments. We choose to project latent features from the last layer of U-Net, which contains both lowelevel and high-level features because of the skip connection mechanism in U-Net. We also evaluate the effect of other layer features in experiments.
Global semantic map. The fine-grained map aggregates environment details from RGB images which are captured along the navigation path. Because camera has a limited field of view, the fine-grained map does not contain environmental information that has not been observed. More importantly, requiring agents to recognize all objects, including some common-seen objects, from fine-grained details increases learning difficulty. Thus, hallucinate environments beyond the field of view and to get semantic information about common-seen objects, we feed the fine fine-grained map to a semantic hallucination module to predict global semantic map M E Rmmxxx,, where each pixel indicates whether a particular object out of c, 27 object categories locates in the corresponding space. The e, object categories are listed in Appendix. The hallucination module consists of three convolution layers with batch norm and ReLU activations, followed by U-Net which has two encoder and two decoder convolutional blocks with skip connections. We get the ground-truth global semantic map M from the available 3D semantic information in Matterport3D following existing

---

methods [18, 191. A pixel-wise semantic loss I, is defined as follows:
= Σ CrossEntropy (M;, M,).
(I)
Multi-granularity map. We merge information from both the fine-grained map and predicted semantic map using map encoder. Specifically, these two maps first processed by convolution layer, respectively. The concatenation of two processed maps is fed to a convolution layer for producing a multi-granularity map M Rmxmxx, where is the number of map channels.
3.3
Weakly-supervised map representation learning via object localization task
To learn the correspondence between multi-granularity map representation and diverse objects, we propose weakly-supervic task, the agent is required to reason precisely about semantics information for each region of map representation, which makes the map representation
instraction-relevant object localization auxiliary task. To finish this
significant.
Formulation of reakly-supervised auxiliary task. To localize the instruction object on map, we feed the instruction and multi-granularigg map to an object localization module to predict an egocentric grid map P mm which indicates a 2D distribution of potential locations of instruction- relevant objects. Because we not know the exact location of these instruction objects, we use the instraction-relevant path as guidance to generate coarse ground-truth egocentric grid map P E Rm The principle for generating coarse ground-trume truth is that the regions closer to the path have a higher probability of containing instruction-relees euclidean distance d₁ from the k-th map region to the path. A normalized distance map is defined Ama--dy 1mxm where dmax and dmmi are the maximum and minimum distances among as P' Jmaa - dmia Jk=1
objects. Specifically, we first calculate
m x m regions, respectively. For a region traversed by the path, d₂ = dmni = 0 and P 1; for the farthest region, d4 dmaxx and P 0. The coarse ground-truth grid map P is the softmax across all grids in the distance map, i.e., P = Softmax(P") An object localization loss t, is defined as KL divergence between predicted and coarse ground-truth distributions, i.e.
P
Lo = P.
P.j
2.
By minimizing the above loss function, the agent encouraged to learn a better map representation for localizing instruction objects.
Instruction-relevant object localization module. The localization module contains two main components, namely a state encoder and a state-instruction attention Att(-). The state encoder gated recurrent unit (GRU) [12] and attention module is a scaled dot-product attention [45]. Specifically, first track visual history, which consists of RGB image R, depth image D, and multi-granularity map M. using the state encoder to encode current episode state s,. Then, the state S; attends to instruction features using the staeeinnstruction attention to generate attended instruction feature where the instruction features are encoded by a bi-directional LSTM [43] from instruction T. With the attended instruction feature E R and multi- goalliigranularity map representation M E Rmxmxe predict localization grid map P mm by calculating the cosine similarity between and each map region feature in multi-granularity map:
P Softmax((W..) Softmax((W, (W,)))).))""),
where i= (s, (ILSTM( St, h, = (D() b.-1),
Wg. We are learnable parameter matrices, h, is hidden state of GRU time step t, [+ indicates concatenation operation, fR(-) and fD(() are off-the-sheel RGB and depth encoders, respectively.
3.4 Waypoint navigator and overall learning objective
With the learned multi teitiirrrallartty map, we feed it together with the instruction to waypoint navigator for predicting navigation action. Specifically, an attended map features m. produced by weighted averaging granularryul map M using predicted localization results P in the auxiliary task described in Section 3.3. The localization results help agents highlight relevant regions that
5

---

possibly contain instruction-relevan objects. Then, following CMA [32], another state encoder (GRU) is exploited predict a state s from the concatenation of attended map features m, attended instruction features and the first state feature St, which can be formulated as follows:
s h/ = (R, b/-1),
(4)
where h' is hidden state of GRU. We feed the state s/ a waypoint predictor to predict a waypoint w (Az, Ay) indicating the next navigation goal. We also feed the state to progress predictor to predict a progress value p indicating the completeness of navigation process. Both of these predictors are fully-connected layer. To get the ground-trum -tuth waypoint, we first plan the shortest path from the current agent position to the nearest waypoint on the instruction-relevant path following LAW [40]. Then, we draw circle with 3 meters radius centered at the agent position. The intersection point between the path and circle considered a ground-truth waypoint w.
The ground-truth progress is defined as the normalized distance from the agent current position to the goal following existing work [32]. The waypoint loss tu and progress loss ly are defined as follows:
Lu = w w|l?:
l = 11 - p|1
(5)
With predicted waypoint, we feed it off aff-thhe-hhele deep reinforcement learning model DD- PPO [47] to determine low-level action from action space A. Note that the DD-PPO model does not update during training. the predicted progress is higher than a threshold Ap. the agent executes STOP action. We update the waypoint every 3 time steps.
The overall learning objective for method is as follows:
L = Lg + al, + Blp + Nww
(6)
where a, 8 and y are ppper-paramtters..
Experiments
4.1
Experimental setups
Dataset and evaluation metrics. We conduct our experiments on VLN-CE dataset, which contains 16,844 path-instruction pairs from 90 scenes in Matterport3D. It also contains about 150k augmented data generated by EnvDrop [44]. The dataset is split into the train, seen validation. unseen validation, and test set. We follow the existing works [19, 32] to evaluate the navigation performance using success rate (SR), oracle success rate (OS), success weighted by path length (SPL), trajectory length (TL), and navigation error from goal (NE). Note that an episode is considered successful if the agent calls STOP action within 3m of the goal. More details about evaluation metrics are put in Appendix.
Implementation details. We implement our method based on Pytorch framework [38] and Habitat simulator [42]. Following existing work [32, 40], the training process consists of two parts, i.e., teacher forcing training augmented data [44] and fine-tuning models using dagger training. We distribute training over 2 NVIDIA V 100 GPUs for days on average. A FORWARD action moves the agent forward by 0.25 meters and TURN action turns by 15 We use the same set of hyper- parameters as used in the VLN-CE [32] and show these values in Appendix. The U-Net used for extracting fine-grained features is pre-trained separately on RGB observations from Matterport3D scenes following existing work [19] and is frozen during training. The map size m is set to 100. o, B, and y in Equation (6) are set to 10 such that four reward terms are in the same order of magnitude at initialization. Progress threshold Xp is set to 0.8.
4.2 Comparisons with Satee-ff-thearn methods
We compare our WS-MGMap with current state-of-the-ar methods in Table I on both seen and unseen validation set of VLN-CE dataset. Our proposed WS-MGMap outperforms all methods that follow the same VLN-CE setting (i.e.no panoramic images) in terms of NE, OS, SR, and SPL, which demonstrates the effectiveness of the learned map representation. Besides, we also get comparable
hutps///iiabbbtat.org/

---

Table I: Comparison between our WS- WS-MGMap and state-of-the-ar methods on VLN-CE dataset. Methods marked with use panoramic images. We highlight the best results among the methods that do not use panoramic images in all tables.
Val-Seen
Val-Unseen
TL
OS
SR f
SPL
TL 4
NE 4
OS f
SPL
AG-CMTP* [8]
6.60
56.2
35.9
30.5
7.90
39.2
23.1
19.1
R2R-CMTP [8]
7.10
45.4
36.1
31.2
7.90
38.0
26.4
22.7
HPN-DN [3]
8.54
5.48
53.0
46.0
43,0
7.62
6.31
40.0
36.0
34.0
CWP-CMA* [25]
11.47
5.20
61.0
51.0
45.0
10.90
6.20
52.0
41.0
36.0
Seq2Scq [32]
9.37
7.02
46.0
33.0
31.0
9.32
7.77
37.0
25.0
22.0
CMA [32]
9.26
7.12
46.0
37.0
35.0
8.64
7.37
40.0
32.0
30.0
LAW [40]
9.34
6.35
49.0
40.0
37.0
8.89
6.83
44.0
35.0
31.0
SASRA [27]
8.89
7.17
36.0
34.0
7.89
8.32
24.0
22.0
CM2 [19]
12.05
6.10
50.7
42.9
34.8
11.54
7.02
41.5
34.3
27.6
WS-MGMap (Ours)
10.12
5.65
51.7
46.9
43.4
10.00
6.28
47.6
38,9
34.3
Table 2: Results VLN-CE challenge leaderboard. Methods marked with * use panoramic images.
Test-Unscen
Team
TL L
NE L
OS
SR f
SPL
WWP-VLLBERT* [25]
13.31
5.89
51
42
36
CWP-CMA* [25]
11.85
6.30
49
38
33
WaypointTeam" [31]
8.02
6.65
37
32
30
VIRL_ Team [32]
8.85
7.91
36
28
25
CM2 [19]
13.85
7.74
39
31
24
WS-MGMap (Ours)
12.30
7.11
45
35
28
results compared with the methods that use panoramic images. This further verifies the importance of the WS- WS-MGMap for helping agents perceive environments comprehensively even just using a camera sensor with a limited field of view. The detailed analyses are as follows.
We categorize existing methods into two types, i.e., one for the methods that without building map explicitly and one for that with map representation. For the first type of methods (e.g., Seq2Seq, CMA, LAW), our method outperforms them by a large margin. Specifically, compared with LAW, which the strongest baseline among them, we increase the success rate from 40.0% to 46.9% and from 35.0% to 38.9% on val-seen and val-unseen, respectively. We attribute the improvement to the usage of our multi guatti-raaullarity map, which represents environments explicitly for reducing the learning difficulty. As for the concurrent methods that are also with map representation (e.g., SASRA and CM2), our MS-MGMap also brings a significant improvement against CM2, increasing success rate by 4.0% and 4.6% on val-seen and val-unseen, respectively. We suspect this is because the semantic map used by these methods provides insufficient information for VLN while our WS- MGMap solves this problem. All these results show the effectiveness of the proposed WS-MGMap, which learns a comprehensive map representation for VLN.
We note that our WS-MGMap even outperforms some basclines (i.e., AG-CMTP, R2R- R2R-CMTP and HPN+DN) that use panoramic images on val-unseen, increasing success rate from 23.1%, 26.4% and 36.0% to 38.9%, respectively. This show the potential for replacing expensive panoramic camera with a common camera for VLN. We also note that our method has a relatively longer trajectory length. We argue that this is not good metric for evaluating VLN performance because an unsatisfactory agent who always stops before it reaches the goal will also get a short trajectory length. A better alternative metric is SPL, which considers both episode length and success rate. In terms of SPL, our method significantly outperforms the existing methods who use the same settings with us (37.0% v.s 43.4% on val-seen and 31.0% v.s 34.3% on val-unseen).
VLN-CE leaderboard. We compare our WS-MGMap with prior work on the held-out test-unseen set used for VLN-CE leaderboard In Table 2, our method is lcading among those that use the standard observation (no panoramas) following VIN-CE settings [32]. For fair comparison, we only report the results whose manuscripts are publicly available to ensure that no strong prior knowledge (e.g., exploring the environment prior) is used.
7

---

Table 3: Ablation study on different granularity information for map representation.
Val-Unseen
Map Type
T
NE 4
OS f
SR f
SPL f
No Map
11.66
7.20
36.8
26.5
21.4
Fine-Grained Map
10.11
7.11
41.1
31.6
28.2
Semantic Map
10.89
6.80
42.2
33.3
28.2
Multi- Mulii-Granularity Map (Ours)
10.00
6.28
47.6
38,9
34.3
Table 4: Ablation study on weakly-supervised auxiliary task and localization ground-truth types.
Val-Unseen
Auxiliary Task
GT Type
TL
NE
OS 1
SR
SPL
10.24
6.68
41.7
33.1
29.0
Hard
9.36
6.36
43.6
35.2
32.3
Soft
10.00
6.28
47.6
38.9
34.3
4.3 Ablation studies
Effectiveness of multi-granularity map. Our WS-MGMap contains both fine-grained details and semantic information about environments. We are interested to evaluate whether this different granularity information helps represent environments for VLN. To this end, construct two variants, ie., one only uses the fine-graing greeggraiedd map to represent environments, and another one only uses semantic maps (combination of semantic map projected from image segmentation results and our predicted global semantic map). All other settings are kept the same, including training losses and the auxiliary task. In Table 3, WS-MGMap significantly outperforms these two variants, increasing success rate from 31.6% and 33.3% to 38.9%, respectively. These results show that both granularity information is important for VLN. Without the fine-grained map, agents hard to recognize object attributes. Without the semantic map, agents are required to recognize all objects, including some common objects that can be represented in the semantic map, from fine-gratn fine-grained object details, which increases learning difficulty. In comparison, our WS-MGMap leverages complementary multi-granin granularity information for representing diverse objects. We also try a variant that directly takes raw RGB-D as input to perform VLN in an end-to-end manner. However, this variant drops success rate from 38.9% 26.5% indicating the importance of building a map explicitly for VLN.
Effectiveness of weakly-supervised auxiliary task. For teaching agents to learn representative map that represents diverse objects accurately, we propose weakly-supervised instruction-relevant object localization auxiliary task. To evaluate the effectiveness of this task, we remove the object localization loss in Equation (2), so that there is no auxiliary supervision signal for learning correspondence between maps and instructions. In Table 4, this variant performs worse than our WS-MGMap. success rate decreasing significantly from 38.9% to 33.1% This shows the importance of the proposed weakly-supervisee auxiliary task, which helps learn map representation from the natural correspondence between map and instruction objects. The coarse ground truth used for this task is soft, i.e., each map region is with probability from 0 to 1. In this way, agents are required to figure out the degree of correlation between instructions and all map regions. An alternative is hard binary ground-truth. i.e., map regions whose region- -att distance smaller than a threshold are considered positive and other regions are considered negative. In Table 4, this alternative performs better than the baseline that does not use our proposed auxiliary task but worse than our soft localization ground-truth.
Effect of different layer features for fine-grained map. In this paper, we use a U-Net pre-trained for segmentation task as an RGB encoder to extract image fine-grained details. Existing network interpretability work [52, 28] points out that latent features from lower layers are dominated by color and texture concepts while features from higher layers present more object parts. To evaluate which types of features are more representative for describing environment fine ine-grained details, we project different layer latent features (i.e., first layer, middle layer, last layer, and classification layer) to build fine-grained maps. Table 5, using last layer features outperforms other variants terms of almost all metrics. This is reasonable because the last layer features contain information from both the lower layer and higher layer because of the skip connection mechanism U-Net, which make more representative for representing environments. The features produced from the classification layer

---

Table 5: Ablation study projecting latent features from different layers to build fine- fine-grained map.
Val-Unseen
Projected Feature
4
NE 4
SR f
SPL
First Layer
9.14
6.29
39.6
32.1
28.6
Classification Layer
9.05
6.49
40.2
33.3
29.6
Last Layer
10.00
6.28
38.9
34.3
WS-MGMiap fine grained map
WS-MGMap
WS MGMM
localization auxiliary task
Walk on the red carpet the the next set of doors and weight there.
[al
[b]
counter
Figure 3: Visualization of instruction-relevanl object localization results.
logits for semantic classification. These features are dominated by semantics information, providing less fine- imneggrined environment details. We also find that an agent using middle layer features cannot be trained to convergence. We suspect is because the spatial resolution of the middle features (9 x 9) is too low such that the projected fine-grained map is sparse.
4.4 Visualization results
To evaluate whether the proposed multi-granularity map helps to represent diverse objects, visualize the localization results from Equation (3) Figure 3. Compared with the variant that does not contain a fine-grained map (b) and the variant without the reakly-supervised auxiliary task (c), our WS-MGMap (a) localizes instruction objects more precisely. Specifically, red carpet incorrectly recognized as floor by segmentation model. With fine-grained map, our WS-MGMap localizes it robustly. The variant without fine-grained map confused to localize it because the semantic map can not represent red attribute and carpel object. The variant without the localization auxiliary task also performs worse because is hard to establish the correspondence between map representation and diverse instruction-relevee objects. More results shown in Appendix.
5 Discussion
Limitations and future work. Although our learned multi-granullrriyy map helps for representing environments comprehensively and provides useful information improve VLN performance, the semantic loss used in Equation (1) needs ground-truth semantic annotation, and our 2D top-down map is hard handle the situation when agents go another floor. Future work may explore to adapt 3D mapping technique [5] to our multi-granularij map. Besides, to improve the language- to-object grounding, exploiting commonsense find relevant areas of instruction-mentie could be an interesting future research direction. In addition, although our method shows promising performance in photo-realistic simulated data, it has not been thoroughly evaluated in real world. Directing adapting it to real environments may cause accidents such as breaking somethings and hitting pedestrian.
objects
Conclusion. order to solve the problem that current maps provide insufficient information for VINN task, propose gather different granularity information (i.e., fine-grained details and semantic information) for map representation. Moreover, to make the map better represent diverse objects described in instructions, we propose weakly-suprrvised auxiliary task for learning to localize instruction-rellvant objects on the map with no need of extra localization annotation. Experimental results show that the proposed method significantly outperforms state-of-the-art methods VLN-CE benchmark dataset. Qualitative results also show that the proposed multi multi-granulane granularity map helps to localize instruction objects that incorrectly classified by current maps.

---

Acknowledgments
This work was partially supported by National Key R&D Program of China (No. 2020AAA01069000, National Natural Science Foundation China (No. 62072190), Program for Guangdong Introducing Innovative and Enterpreneurial Teams (No. 20172T07X183).
References
[1] P. A.Aderoon,, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sünderhauf, D. Reid, S. Gould, and A. van den Hengel. aision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In CVPR, pages 3674-3683, 2018.
[2] P. Anderson. Q. Wu, D. Teney, J. Bruce, M. Johnson. N. Sünderhauf, I. D. Reid, S. Gould, and A. van den Hengel. iiriio-and-languaaee navigation: Interpreting environments. CVPR, pages 3674-3683, 2018. 3
groundd navigation instructions in real
[3] E. Beeching, J. Dibangoye, O. Simonin, and C. Wolf. Learning plan with uncertain topological maps. In ECCV, pages 473 493.-990220 2020.
V. Cartillier, Z. Ren, N. Jain, S. Lee, Essa, and D. Batra, Semantic mapnet: Building allocentric semantic maps and representations from egocentric views. In AAAI, volume 35, pages 964-972, 2021. 2, 3, 4
[5] D. Chaplot, M. Dalal, Gupta, Malik, and Salakhutdinov. SEAL: self-supervised embodied active learning using exploration and 3d consistency. In NewrIPS, pages 13086-130983 2021. 9
[6] D. Chaplot, D. Gandhi, A. Gupta, and R. Salakhutdinov. Object goal navigation using goal-oriented semantic exploration. In NeurIPS. pages 4247-4258, 2020. 2,3
[7] D. Chaplot, R. Salakhutdinov, Gupta, and S. Gupta. Neural topological SLAM for visual navigation. In CVPR, pages 12872-12881, 2020.
[8] K. Chen, J. K. Chen, J. Chuang, M. Vázquez, and Savarese. Topological planning with transformers for vision-and-languagge navigation. In CVPR, pages 11276-11288 2021. 3,7
[9] P. Chen, D. Ji, K. Lin, W. Hu, W. Huang, T. H. Li, M. Tan, and C. Gan. Learning active camera for multi-object navigatio. In NeurlPS, 2022. 3
[10] Y. Chen, Zhang, and Y. Lou. Topological and semantic map generation for mobile robot indoor navigation.
In ICIRA, pages 337-347 2021.
[11] R. Cheng, 7- Wang, and K. Fragkiadaki. Geometry-aware recurrent neural networks for active visual recognition. In Advances in Neural Information Processing Systems, pages 5086-5096, 2018. 3
[12] K. Cho, B. van Merrienbner, Ç. Gülçehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In EMNLP, pages 1724-1734, 2014.
[13] D. Fried, R. Hu, V. Cirik, A. Rohrbach, J. Andreas, L. Moreney, T. Berg- Kirkpatrick, K. Saenko, D. Klein, and Darrell. Speaker-follower models for lision-and-lanuue navigation. In NewrIPS, pages 3318-3322 2018. 3
[14] C. Gan, Y. Gu, S. Zhou, J. Schwartz, S. Alter, Traer, D. Gutfreund, Tenenbaum. J. McDermott, and A. Torralba. Finding fallen objects via asynchronous audio-visual integration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10523-10533 2022. 3
[15] C. Gan, J. Schwartz, S. Alter, M. Schrimpf, J. Traer, J. De Freitas, J. Kubilius, A. Bhandwaldar, N. Haber, M. Sano, et al. Threedworld: A platform for interactive multi-modal physical simulation. NewrIPS, 2021.
[16] C. Gan, Y. Zhang, J. Wu, B. Gong, and J. B. Tenenbaum. Look, listen, and act: Towards audio-visual embodied navigation. In ICRA, pages 9701-9707, 2020. 3
[17] C. Gan, S. Zhou, J. Sehwartz, S. Alter, A. Bhandwaldar, D. Gutfreund, D. L. Yamins, J. J. DiCarlo, McDermott, Torralba, et al. The threedworld transport challenge: A visually guided task-and motioo planning benchmark towards physically realistic embodied ai. In 2022 International Conference on Robotics and Automation (ICRA), pages 8847-8854, 2022. 3
[18] G. Georgakis, B. Bucher, K. Schmeckpeper, S. Singh, and K. Daniilidis. Learning to map for active semantic goal navigation. In ICLR, 2022. 5
[19] G. Georgakis, K. Schmeckpeper, K. Wanchoo, S. Dan, E. Miltsakaki, D. Roth, and K. Daniilidis. Cross- modal map learning for vision and language navigation. In CVPR, pages 15439-15449, 2022. 2, 3, 5, 6,
10

---

[20] G. Georgakis, K. Schmeckpeper. K. Wanchoo. S. Dan, E. Miltsakaki, D. Roth, and K. Daniilidis. Cross- modal map learning for vision and language navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 15460-15470, 2022.
[21] P.-L. Guhur, M. Tapaswi, S. Chen, I. Laptev, and C. Schmid. Airbert: In-domain pretraining for vision- and-language navigation. In ICCV. pages 1634- 1643, 2021. 3
[22] S. Cupta, J. Davidson. S. Levine, R. Sukthankar, and J. Malik. Cognitive mapping and planning for visual navigation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2616-2625, 2017.
[23] W. Hao, C. Li, X. Li, L. Carin, and J. Gao. Towards learning a generic agent for vision-and-language navigation via pre-traininn training. In CVPR, pages 13134- 13143, 2020. 3
[24] Henriques and A. Vedaldi. Mapnet: An allocentric spatial memory for mapping environments. In proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages
2018.
[25] Y. Hong, Z. Wang. Q. Wu, and S. Gould, Bridging the gap between learning discrete and continuous environments for vision-and- -an--agugg navigation. In CVPR, pages 15418-154285 2022. 3,
[26] Y. Hong, Q. Wu, Y. Qi, C. R. Opazo, and Gould. VLN BERT: A recurrent vision-and-language BERT
for navigation. In CVPR, pages 1643 1653, 2021. 1
[27] M. Z. Irshad, N. C. Mithun, Z. Seymour, H. Chiu, S. Samarasekera, and R. Kumar. SASRA: semantically- aware spatio-temporal reasoning agent for lision-and-lanuuae navigation in continuous environments. CoRR, abs/2108. 1945. 2021. 2,7,
[28] M. A. Islam, M. Kowal, P. Esser, S. Jia, B. Ommer, K. G. Derpanis, and N. Bruce. Shape or texture: Understanding discriminative features in cnns. In ICLR, 2022. 2,8.8
[29] M. G. Jadidi, J. V. Miró, and G. Dissanayake. Mutual information-base exploration on continuous occupancy maps. In IROS, pages 6086 6092, 2015. 3
[30] K. D. Katyal, A. Polevoy, L. Moore, Knuth, and K. M. Popek. High-speed robot navigation using predicted occupancy maps. In ICRA, pages 5476-5482, 2021. 3
[31] J. Krantz, A. Gokaslan, D. Batra, Lee, and O. Maksymets. Waypoint models for instruction-guided
navigation in continuous environments. In /CCV, pages 15142-15151, 2021. 3,7
[32] J. Krantz, F. Wijmans, A. Majumdar, D. Batra, and S. Lee. Beyond the nav-graph: Vision-andd-anguage navigation in continuous environments. In ECCV, pages 104-120, 2020. 3, 6,
[33] K. Kumar Singh and Y. Jae Lee. Hide-and-seek: Forcing a network to be meticulous for weakly-supervic object and action localization. In Proceedings of the IEEE International Conference Computer Vision, pages 4524-3533, 3
[34] D. Li, J.B. Huang, Li, S. Wang, and M.-H. Yang. Progressive representation adaptation for weakly supervised object localization. IEEE transactions on pattern analysis and machine intelligence, 42:1424- 1438, 2019. 3
[35] C. Ma, J. Lu. Z. Wu, G. AlRegib, Z. Kira, R. Socher, and C. Xiong. Self-monitoring navigation agent via auxiliary progress estimation. In /CLR, 2019. 3
[36] A. Majumdar, A. Shrivastava, S. Lee, Anderson, D. Parikh, and D. Batra. Improving lisio-andd-anguage navigation with image-text sange-textpppe pairs from the web. In ECCV, pages 259 274, 2020. 3
[37] X. Meng, N. D. Ratliff, Y. Xiang, and D. Fox. Sealing local control to large-scale topological navigation. In ICRA, pages 672-678. 2020. 3
[38] A. Paszke, S. Gross, F. Massa, A. Lerer. J. Bradbury, G. Chanan. T. Killeen, Z. Lin, N. Gimelshein. L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani. S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-pperformncce deep learning library. In NewrlPS, pages 8024-8035 8035, 2019.
[39] S. K. Ramakrishnan, Z. Al- Al-Halah, and K. Grauman. Occupancy anticipation for efficient exploration and navigation. In ECCV, pages 400 418, 2020. 3
[40] S. Raychaudburi. S. Wani, S. Patel, U. Jain, and A. X. Chang. Language-aligned waypoint (LAW) supervision for lision-and-language navigation in continuous environments. In EMNLP, pages 4018 2021. 3, 6,
[41] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation.
In MICCAI, pages 134-241, 2015. 4
[42] M. Savva, J. Malik, D. Parikh, D. Batra, A. Kadian, O. Maksymets, Y. Zhao, E. Wijmans, B. Jain, J. Straub,
J. Liu, and V. Koltun. Habitat: A platform for embodied Al research. In ICCV, pages 9338 9346, 2019. 3,
6
11

---

[43] M. Schuster and K. K. Paliwal. Bidirectional recurrent neural networks. IEEE trunsactions Signal Processing, 45:2673-2681, 1997. 5
[44] H. Tan, L. Yu, and M. Bansal. Learning to navigate unseen environments: Back translation with environ-
mental dropout. In NAACL-HLT, pages 2610 2621, 2019. 3, 6
[45] A. Vaswani, N. Shazeer, N. Parmar. J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In NeurlPS, pages 5998-6008, 2017. 5
[46] x. Wang. Q. Huang, A. Celikyilmaz, J. Gao, D. Shen, Y. Wang, W. Y. Wang, and I.. Zhang. Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation. In CVPR. pages 6629-6638. 20 3
[47] E. Wijmans, A. Kadian, A. Morcos, S. Lee. I. Essa, D. Parikh, M. Savva, and D. Batra. DD-PPO: learning near-perfect pointgoal navigators from 2.5 billion frames. In ICLR, 2020. 6,
[48] H. Xue, C. Liu, F. Wan, J. Jiao, X. Ji, and Q. Ye. Danet: Divergent activation for weakly supervised object localization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
6589-6598, 2019. 3
[49] R. 7eng, C. Gan, P. Chen, W. Huang, Q. Wu, and M. Tan. Breaking winner-takes-all: Iterativewiinnes--uut networks for weakly supervised temporal action localization. IEEE Transactions Image Processing. 28:5797-5808, 2019.
[50] L. Zhang, L. Wei, P. Shen, W. Wei, G. Zhu, and J. Song. Semantic SLAM based on object detection and improved octomap. IEEE Access, 6:75545-75559 2018.
[51] X. Zhang, Y. Wei, J. Feng, Y. Yang, and T. S. Huang. Adversarial complementary learning for weakly supervised object localization. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1325-1334, 2018.
[52] B. Zhou, D. Bau, A. Oliva, and A. Torralba. Interpreting deep visual representations via network dissection.
IEEE transactions on pattern analysis and machine intelligence, 41::21111122 -145, 2018. 2, 4,
[53] B. Zhou, A. Khosla, A. Lapedriza, Oliva, and A. Torralba. Learning deep features for discriminative
localization. In CVPR, pages 2921-2222222 19229,,,, 2016. 3
[54] F. Zhu, Y. Zhu, X. Chang, and X. Liang. Vision-language navigation with self-supervised auxiliary reasoning tasks. In CVPR, pages 10009-10019, 2020. 3
[55] Y. Zhu, Y. Zhou, Q. Ye, Q. Qiu, and J. Jiao. Soft proposal networks for weakly supervised object localization. In ICCV, pages 1841-1850, 2017.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 5
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 5
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [No] We will publish our code upon acceptance.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4.1.
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [No] Error bars are not reported because it would be too computationally expensive.
12

---

(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 4.1.
4. If you using existing assets (e.g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "mOZQ5CJSazh", "reviewer": "Reviewer_eYGK", "summary": "## Post Rebuttal\n\nI thank the authors for their response. My main concern was addressed and the zero-shot evaluation on RxR is helpful. I encourage the authors to include a full evaluation on RxR. I have increased my rating.\n\n## Pre-rebuttal\nThis paper presents a method for building an environment map towards the end of performing natural language instruction following in continuous environments (VLN-CE). In the proposed method, the map has multiple granularities of information. A coarse granularity with features projected from UNet, and a global semantic map created by hallucinating semantic labels for the entire environment based on the coarse granularity map.\n\nThe encourage the map contain features relevant to instruction following, the authors also propose an additional objective that predicts the location of the ground-truth path on the environment map. This predicted path location is then used to attend to the map.\n\nThe method is evaluated on the VLN-CE dataset and outperforms all existing single camera methods. It is also competitive with panoramic methods.", "questions": "See weaknesses", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\nThe proposed method is well motivated. It makes sense to have multiple levels of information in a semantic map and to augment what is given to the agent based on the instruction.\n\nThe method works well. VLN-CE is a challenging task and a +4.5% SR increase is meaningful.\n\nThe ablation studies are well-done and provide useful insight.\n\n### Weaknesses\n\nNo evaluation on RxR-Habitat (https://ai.google.com/research/rxr/habitat). This uses the more recent RxR dataset that fixed some of the issues with the original VLN dataset and thus evaluation on this dataset would be very helpful in evaluated the method.\n\nThe intro to the task should cite + mention VLN-CE since that's the variant of the task being studied (and the less common one). I recommend referring to the task as VLN-CE to avoid confusion.\n\nL28-29: We don't know why humans (or any animal for that matter) build cognitive maps. Neither Tolman nor O'Keefe claim to know why cognitive maps were learned to be built (other than that they are useful for survival). They are an explanation of observed behavior, seen, but we don't know why that mechanism exists other than it's useful for survival (and thus is something that was learned implicitly). I recommend weakening this statement.\n\nSelf-supervised map representation learning via object location task is not a correct name. The objective is not self-supervised. Self-supervised implies that the learning objective is valid regardless of the dataset source (i.e. training data or data we'd see at evaluation). Ground-truth paths are certainly not part of the data we'd see at evaluation so this method doesn't fit. The task is possibly object localization with pseudo-labels (so weakly supervised), but those pseudo-labels are soft labels for predicting the ground-truth path. Since the motivation comes from object localization, that part of the name is probably fine, but the self-supervised part isn't.\n\nOverall, I think this paper presents a meaningful contribution and am happy to increase my rating if these concerns are addressed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "k9MT2sk0mSE", "reviewer": "Reviewer_kB3d", "summary": "This paper tackles the Visual & Language Navigation task. The paper proposes a new \"multi-granular\" map representation for encoding coarse-grained (semantic) and fine-grained (textures, colors, etc) details about the 3D space. It also proposes a self-supervised loss to localize \"instruction-relevant\" objects on the map. Experiments are performed on the VLN-CE benchmark on MP3D dataset. The work demonstrate strong improvement over prior methods for VLN-CE while using limited field-of-view sensing (as opposed to panoramic sensing).", "questions": "# Novelty of multi-granular maps\n\n- Is the proposed mapping method achieving something different compared to prior work [1, 2, 3, 4]? \n- If so, can the authors quantitatively compare against map representations from [2] (and maybe even [1, 3, 4]) and verify it is indeed better? That is, replace the proposed multi-granular maps while keeping the auxiliary task and waypoint navigation as it is.\n\n# Novelty of auxiliary task\n\nThe method from [5] picks waypoints along the demonstration trajectory, and places gaussians at each waypoint to obtain the target heatmap for prediction. \n- Is the heatmap P in L177-182 different from the heatmap in [5]? \n- If they are indeed different (i.e., P focuses more on instruction-relevant objects rather than waypoints), could the authors replace the heatmap P with the heatmap from [5] and verify that the proposed method is indeed better? (note: the multi-granular maps and waypoint navigation pipeline remain the same)\n- Is the auxiliary task really self-supervised? That is, is it learning to localize instruction-relevant objects? Is the heatmap from [5] not resulting in localizing instruction-relevant objects in that case?\n\n# Other clarifications in approach and experiments\n- Could the authors clarify the questions raised in the weaknesses?\n\n[1] **MapNet:** Henriques, Joao F., and Andrea Vedaldi. \"Mapnet: An allocentric spatial memory for mapping environments.\" proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.  \n[2] **Semantic MapNet:** Cartillier, Vincent, et al. \"Semantic mapnet: Building allocentric semantic maps and representations from egocentric views.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 2. 2021.  \n[3] **Cognitive Mapping and Planning**: Gupta, Saurabh, et al. \"Cognitive mapping and planning for visual navigation.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.  \n[4] **Geometry-Aware RNNs**: Cheng, Ricson, Ziyan Wang, and Katerina Fragkiadaki. \"Geometry-aware recurrent neural networks for active visual recognition.\" Advances in Neural Information Processing Systems 31 (2018).  \n[5] **Cross-Modal Map Learning**: Georgakis, Georgios, et al. \"Cross-modal Map Learning for Vision and Language Navigation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.  ", "limitations": "Yes. They have been sufficiently addressed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# ----------------------- Post-rebuttal update --------------------\nThe author's responses during the rebuttal were very convincing and addressed my concerns. The main weaknesses pre-rebuttal were the novelty of the multi-grain map representation and the auxiliary loss. In response, the authors compared and contrasted with the cited works, and even showed quantitative benefits in relation to them. I've raised my rating to 7 (accept).\n\n# ----------------------- Pre-rebuttal review --------------------\n# Strengths\n\n- The paper is well motivated in building multi-granular maps for wholistic object representations and grounding instructions to relevant objects on the map. \n- The paper writing is good and it was easy to read and understand the ideas and contributions.\n- Experiments are well designed. A lot of relevant and appropriate baselines have been compared with. The experimental results show good improvements over comparable methods. Ablation studies confirm the value of the contributions. Following traditions from prior work on this task, the authors also present results on an online leaderboard, which verifies the claims made under standardized settings. \n\n# Weaknesses\n## Novelty of multi-granular maps for embodied AI\n- The proposed multi-granular maps does the following: (1) use image segmentation features to capture fine-grained details, (2) project the features to get a top-down 2.5D fine-grained map, (3) obtain coarse-grained segmentation maps with potential expansion beyond observed cells, and (4) jointly merging fine-grained and coarse-grained to obtain final representations. This particular pipeline itself appears novel (particularly in VLN).\n- However, prior work [1, 2, 3, 4] has extensively studied mapping in high-dimensional feature spaces (as opposed to occupancy / semantic maps). These have not been discussed in Sec. 2.2.\n- More importantly, they might be doing exactly what the proposed method intends to do. The general idea is to get visual feature maps of the egocentric inputs, and project them in a geometric / learned fashion to top-down maps / voxel grids. That is, the multi-grained map may be obtained implicitly here as opposed to explicitly as done in the Fig. 2. \n- For example, Semantic MapNet [2] builds such maps from visual inputs in the encoder, and decodes them to obtain semantic maps. In L145-146, the authors suggest that the last layers of an image segmentation model (Unet) contain a good mix of textural and semantic features. It is reasonable to expect the same to hold true for the last layers of Semantic MapNet to encode something similar, but in the top-down map space. \n\n\n## Novelty of auxiliary task\n- L174-176: \"The principle for generating coarse ground-truth is that the regions closer to the path have a higher probability of containing instruction-relevant objects.\" --- It is unclear to me whether this supervision is resulting in localizing \"instruction-relevant\" objects. \n- This objective could also plausibly be viewed as a form of imitation learning for waypoint prediction (like [5]). For example, in Figure 3, the model could be learning to predict the locations of future waypoints to follow instead of referring to the carpet (i.e., the carpet being along the waypoints could just be a coincidence). \n- If the model were to just predict future waypoints, then the method will no longer be self-supervised. The self-supervised component only refers to how \"instruction-relevant\" objects are localized without explicit supervision.\n\n\n## Need clarifications in approach and experiments\n- L152-156 - is hallucination of maps beyond the field of view helpful quantitatively?\n- Eqn 1 - is it a pixelwise loss?\n- L173 \"Because we do not know the exact locations of these instruction objects\" --- how often does this type of instruction-object ambiguity occur? For example, the instruction object being a long-bench and there are multiple benches nearby. I can imagine that in quite a few cases, the objects might be unique and can be localized accurately. \n- L177-184 - does P remain the same for all time-steps of the trajectory? If so, is it possible for the model to just ignore visual inputs and map the fixed language instruction to the fixed P (regardless of the agent states)?\n- In Tab. 3 ablation study - do all rows benefit from the auxiliary task? For example, the result without auxiliary task in Tab. 4 is close to semantic map (row 3 in Tab. 3). If semantic map does not use auxiliary task, then the gain from multi-granular representations itself is limited.\n- **[Minor point, not weakness]** L231 - how is the DAgger training performed without a human in the loop? Might be useful to add a section in the supplementary to explain this.\n- **[Minor point, not weakness]** In Tab. 5 ablation study - classification layer logits are better than semantic map from Tab. 3. Could the authors comment on this?\n\n\n# Typos and grammatical errors\nL11 - \"Moreover, we propose\" -> \"We further propose\"   \nL64 - followings -> follows  \nL87 - \"disables the agent to ground … well\"  -> \"prevents the agent from effectively grounding …\"  \nL114 - \"access to predefined\" -> \"access the predefined\"  \nL122 - \"larger number\" -> \"large amount\"  \nL164 - procuding -> producing  \nL168 - \"reason precise semantics\" -> \"reason precisely about semantic\"  \nL231 - \"teaching force\" -> \"teacher forcing\"  \nL322 - \"carper\" -> \"carpet\"  \n\n\n\t\n[1] **MapNet:** Henriques, Joao F., and Andrea Vedaldi. \"Mapnet: An allocentric spatial memory for mapping environments.\" proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.  \n[2] **Semantic MapNet:** Cartillier, Vincent, et al. \"Semantic mapnet: Building allocentric semantic maps and representations from egocentric views.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 2. 2021.  \n[3] **Cognitive Mapping and Planning**: Gupta, Saurabh, et al. \"Cognitive mapping and planning for visual navigation.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.  \n[4] **Geometry-Aware RNNs**: Cheng, Ricson, Ziyan Wang, and Katerina Fragkiadaki. \"Geometry-aware recurrent neural networks for active visual recognition.\" Advances in Neural Information Processing Systems 31 (2018).  \n[5] **Cross-Modal Map Learning**: Georgakis, Georgios, et al. \"Cross-modal Map Learning for Vision and Language Navigation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "O93fOlHIqhc", "reviewer": "Reviewer_RkNa", "summary": "This paper presents an approach for vision-and-language navigation in continuous environments, centered around a semantic map which both aggregates and is used to predict object-centric information from the agent's observations. The approach uses fine-grained features extracted from a U-Net segmentation model applied to the agent's visual observations, as well as features for unseen parts of the environment predicted by a network supervised with ground-truth 3D info from the Matterport environment. This \"multi-granularity\" map is supervised using the ground-truth path of the agent through the environment, and used together with a recurrent, instruction-conditioned attention mechanism to predict waypoints for an off-the-shelf point-nav model to navigate to. The approach obtains the highest performance, by a substantial margin, among published VLN-CE leaderboard approaches that do not use panoramic image representations. Ablations show that each component of the approach helps pretty substantially.", "questions": "Q1) Could the authors clarify what sources of ground-truth semantic information from the Matterport3D simulator (beyond RGB and depth imagery) are used, and when they are used -- in training, inference, or both?\n\nQ2) What parts of the model are updated during each of the two stages of training: teacher-forcing, and DAgger training? (e.g. is DD-PPO model held fixed during both of these, and the map prediction and object localization modules fine-tuned)?\n\nQ3) What are the off-the-shelf FGB and depth encoders (line 197)?\n", "limitations": "I found the discussion of limitations in the paper somewhat lacking, although it should be easy to address given more space. In addition to discussing limitations due to object and environmental supervision (see above), it would also improve the paper to discuss more about possible directions that could improve the language-to-object grounding.\n\n*Suggestions*\n- Figure 1: the \"bed\" and \"door\" layers were unclear.\n- The motivation in the intro confused me a bit about what type of object supervision is used.\n- The object supervision in coarse localization was confusing to me. The localizer doesn't really seem to get any explicit object supervision, just closeness to the path -- and there doesn't seem to be flexibility for the model to choose how to focus the region attention using e.g. some sort of weakly-supervised or MIL loss. I think it might help to present this in a different way, e.g. as learning general info about the path (which may or may not be object-relevant). \n\n*Minor suggestions*\n- 112: \"continued\" -> \"continuous\"\n- 149: \"camera is with a\" -> \"the camera has a\"\n- 168: \"reason over precise semantic information\"\n- 322: \"carper\" -> \"carpet\"\n", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths*\n\nS1) The object representation and semantic map approach is well-motivated and well-executed, and the performance on the benchmark seems very strong.\n\nS2) The paper performed a thorough set of experiments that helped me convince me all the decisions made in the approach (both levels of granularity in the semantic map; the localization task) were useful. \n\n*Weaknesses*\n\nW1) One weakness of the approach is that the full version depends on \nground-truth 3D semantic information from the Matterport3D simulator in training (although not in inference, I believe). If I understand Table 3 correctly, without this info (just using the fine-grained map with segmentation features) the approach performs underperforms most of the other non-panoramic approaches (CMA, LAW, CM2). Since this info is only used in training, I don't think it's a crucial weakness, but I feel the paper could be clearer about this limitation, in particular perhaps rephrasing or removing the motivating claim in the intro that contrasts its approach with \"a naive solution that annotated a dataset ... with object classes and attributes for a segmentation model\". (lines 40-41). \n\n_Edit after response_: The response presented some compelling experiments showing that the proposed approach also improves when the semantic information is predicted by a pre-trained Faster R-CNN model, so I now feel like this is even less of a weakness. \n\nW2) The approach, as I understand it, is quite similar to the approach of Blukis et al, \"Mapping Navigation Instructions to Continuous Actions with Position-Visitation Prediction\", CoRL 2018, which also uses object-centric semantic maps and waypoint prediction, to do language-conditioned navigation in a continuous environment. However, I don't think this is a crucial weakness as that approach was applied in a different environment with what seem to be much simpler visual observations (involving a discrete set of objects). It would strengthen this paper to contrast with that approach in the related work.\n\n_Edit after response_: As the authors pointed out in their response, Blukis et al. does not use the multi-granular information that the proposed approach does. \n\nW3) The writing could be clearer, see suggestions below.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "06N1mDn3HYJ", "reviewer": "Reviewer_2vTe", "summary": "This paper presents a map-based method for Vision-and-Language Navigation (VLN). The approach builds a semantic map which includes object categories and properties such as color, texture and shape. The semantic map is used to localize the objects and their properties mentioned in the instruction. The localization, map and instruction is used to predict waypoints for a local policy. Experiments onVision-and-Language Navigation in Continuous Environments (VLN-CE) benchmark show competitive results.", "questions": "- I do not understand why the authors use the term \"Multi-granularity\" for describing maps representing objects and their properties. It seems a bit misleading to me because the granularity of the map for both object and properties is the same (denoted by m L143 and L154). What parts of the map have multiple granularities?\n- A potential benefit of a modular map-based approach which is trained offline vs end-to-end RL approaches is the sample efficiency as shown in https://arxiv.org/abs/2201.10029. Did you observe such benefits for the proposed approach as compared to the baselines?", "limitations": "Yes.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper presents of novel application of map-based navigation techniques to VLN-CE. The use of maps encoding both object categories and properties for navigation is also novel to the best of my knowledge.\n- The use of ground truth trajectories for language-to-map grounding is innovative. It allows the authors to have a localization objective without requiring map annotations.\n- The ablation experiments show the importance of both kinds of map.\n\nWeaknesses:\n- There is a published method (CWP-CMA) which performs better than the proposed approach on the public VLN-CE benchmark. The benefit of the proposed approach is that it doesn't require the agent to turn-around to get a panorama between waypoints. However, it is unclear which approach would be better given access to a panoramic camera.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept