VTNet: Visual Transformer Network for Object Goal Navigation
Authors: Heming Du
Xin Yu
Liang Zheng
Venue: ICLR 2021 Poster (2021)

Abstract: Object goal navigation aims to steer an agent towards a target object based on observations of the agent. It is of pivotal importance to design effective visual representations of the observed scene in determining navigation actions.  In this paper, we introduce a Visual Transformer Network (VTNet) for learning informative visual representation in navigation.  VTNet is a highly effective structure that embodies two key properties for visual representations: First, the relationships among all the object instances in a scene are exploited; Second, the spatial locations of objects and image regions are emphasized so that directional navigation signals can be learned. Furthermore, we also develop a pre-training scheme to associate the visual representations with navigation signals, and thus facilitate navigation policy learning. In a nutshell, VTNet embeds object and region features with their location cues as spatial-aware descriptors and then incorporates all the encoded descriptors through attention operations to achieve informative representation for navigation. Given such visual representations, agents are able to explore the correlations between visual observations and navigation actions. For example, an agent would prioritize ``turning right'' over ``turning left'' when the visual representation emphasizes on the right side of activation map. Experiments in the artificial environment AI2-Thor demonstrate that VTNet significantly outperforms state-of-the-art methods in unseen testing environments.

Published as a conference paper at ICLR 2021
VTNET: VISUAL TRANSFORMER NETWORK FOR OBJECT GOAL NAVIGATION
Heming Du Xin Yu Liang Zheng!
1 Australian National University
University of Technology Sydney
$CSIRO-DATA61
fheming. du, liang. zheng eauu. www.anu..uuuaaaaaeaamaauaaa...aa edu. au, yin.yueuts.ed edu.aasseeduaa
ABSTRACT
Object goal navigation aims to steer an agent towards a target object based on observations of the agent. It is of pivotal importance to design effective visual representations of the observed scene in determining navigation actions. In this paper, we introduce a Visual Transformer Network (VTNet) for learning infor- mative visual representation in navigation. VTNet is highly effective structure that embodies two key properties for visual representations: First, the relation- ships among all the object instances in a scene are exploited; Second, the spatial locations of objects and image regions are emphasized so that directional naviga- tion signals can be learned. Furthermore, we also develop pre-training scheme to associate the visual representations with navigation signals, and thus facilitate navigation policy learning. In nutshell, VTNet embeds object and region fea- tures with their location cues as spatial-aware descriptors and then incorporates all the encoded descriptors through attention operations achieve informative representation for navigation. Given such visual representations, agents are able explore the correlations between visual observations and navigation actions. For example, an agent would prioritize "turning right" over "turning left" when the visual representation emphasizes on the right side of activation map. Experi- ments in the artificial environment A12-Thor demonstrate that VTNet significantly outperforms tfte-offthheaae methods unseen testing environments.
INTRODUCTION
The goal of target-driven visual navigation is to guide an agent to reach instances of given target category based on its monocular observations of an environment. Thus, highly desirable to achieve an informative visual representation of the observation, which is correlated to directional navigation signals. In this paper, we propose a Visual Transformer Network (VTNet) to achieve an expressive visual representation. In our VTNet, we develop Visual Transformer (VT) to extract image descriptors from visual observations and then decode visual representations of the observed scenes. Then, present a pre-training scheme associate visual representations with directional navigation signals, thus making the representations informative for navigation. After pre-training, visual representations an end-to-end manner. In particular, our VT exploits two newly designed spatial-aware descriptors as the key and query, (i.e., a spatial-enhanced local descriptor and positional global descriptor) and then encodes them to achieve an expressive visual representation.
fed navigation policy network and train entire network in
Our spatial-enhanced local descriptor is developed fully take advantage of all detected objects for the exploration of spatial and category relationships among instances. Unlike the prior work (Du) 2020) that only leverages one instance per class mine the category relationship, our VT able to exploit the relationship of all the detected instances. To this end, we employ an object detector DETR (Carion al.) (020)) since features extracted from DETR not only encode object appearance information, such as class labels and bounding boxes, but also contain the relations be- tween instances and global contexts. Moreover, DETR features are ccale-invarian (output from the
"Corresoonding author

---

Published as a conference paper at ICLR 2021
Observation
Instances
Activation Map
Action
ResteRight
Desklamp
CellPhone
RotatoLoft
Moxohrood
Book
LoOkDWW
Dara
Laptop
Figure I: Motivation of the Visual Transformer Network (VTNett) A target class (cellphone) is highlighted by green bounding boxes. An agent first detects objects of interest from its observation. Then, the agent attends detected objects to the global observation by the visual transformer (VT). High attention scores are achieved on the left side of the observation, which correspond the target (cellphone). Then, the agent will choose RotateLef ft to reach targets.
same layer) in comparison to features used in ORG (Du et al. 2020). Considering that object posi- tions cannot be explicitly decoded without the foed-forward layer of DETR, we therefore enhance all the detected instance features with their locations to obtain spatial-enhanced local descriptors. Then, we take all the spatial-enhanced local descriptors as the key our VT encoder to model the relationships among detected instances, such category concurrence and spatial correlations
Furthermore, we introduce a positional global descriptor as the query for our VT decoder. In partic- ular, we associate the region features with image region positions (such as bottom and top) and thus facilitate exploring the correspondences between navigation actions and image regions. To do so, we divide a global observation into multiple regions based on spatial layouts and assign a positional em- bedding to each region feature as our spatial- query descriptor, we attend the spatial-enhanced local descriptor to the positional global descriptor query to learn the relationship between instances and observation regions via our VT decoder.
global descriptor. After obtaining the global
However, we found directly training our VTNet with a navigation policy network fails converge due to the training difficulty of the transformers (Vaswani et al.) 2017). Therefore, we present pre- training scheme associate visual representations and directional navigation signals. We endow our VT with the capability of encoding directional navigation signals by imitating expert experi- ence. After warming-up through human instructions, VT can learn instructional representations for navigation, as illustrated in Figure]
After pre-training our VT, we employ a standard Long Short Term Memory (LSTM) network to map the eurrent visual representation and previous states agent action. We adopt A3C archi- tecture (Mnih et al. 2016) to learn the navigation policy. Once our VTNet has been fully trained, agent exploit the correlations between observations and navigation actions improve visual navigation efficiency. In the popular widely-used navigation environment A12-Thor (Kolve et al. 2017), , method significantly outperforms the state-of-the--ar. Our contributions are summarized as follows:
We propose novel Visual Transformer Network (VTNet) extract informative feature representations for visual navigation. Our visual representations not only encode relation- ships among objects but also establish strong correlations with navigation signals.
We introduce a positional global descriptor and spatial-enhanced local descriptor as the query and key for our visual transformer (VT), and then the visual representations decoded by our VT are attended to navigation actions via our presented pre-training scheme, thus providing a good initialization for our VT.
Experimental results demonstrate that our learned visual representation significantly im- proves the efficiency of the state-of-the-an visual navigation systems in unseen environ- ments by 14.0% relatively on Success Weighted by Path Length (SPL).
As observed in DETR, the number of objects of interest in scene is usually less than 100. Thus, set the key number 100 in the VT encoder.
2

---

Published as a conference paper at ICLR 2021
2
RELATED WORKS
Visual navigation, as a fundamental task in robotic and artificial intelligence, has attracted increasing attention recently. Traditional methods (Oriolo al. 1995) often leverage environment maps for navigation and divide a navigation task into three steps: mapping, localization and path planning. Some approaches employ given map to obviate obstructions (Borenstein & Koren 1989 1991). Dissanayake et al. (2001) infer robot positions by simultaneous localization and mapping (SLAM). However, maps are usually unavailable in unseen environments.
Recently, reinforcement learning (RL.) has been applied in visual navigation. In general, it takes vi- sual observations as inputs and predicts navigation actions directly. MMrowwki et al. (2016) develop navigation approach in 3D maze environments and introduce depth prediction and loop closure classification tasks to improve navigation performance. Parisotto & Salakhutdinov (2017) investi- gate memory system to navigate in mazes. Some methods (Sepulveda et al. [0112018]]] Chen et al.
2019 Savinov et al., 2018) use both visual features and the topological guidance of scenes for nav-
igation, while natural language instructions are employed to guide agent to route among rooms
[Anderson et al. 2018b Wang et al.) 2019 Deng et al. 2020 Hu et al. 2019 Majumdar et al. 2020
Hao et al. 2020). We notice that transformer architectures are also employed by Hao et al.) (2020).
named Prevalenet. However, Prevalenet is used to model languages and predict camera angles rather than encoding local and global visual features. Hence, Prevalenet is essentially different from our
VT. Furthermore, Kahn et al.; (2018) design a self-supervised approach to model environments by reinforcement learning. TTag et al. (202T) customize specialized network for visual navigation via an Auto-Navigator. A Bayesian relational memory is introduced by Wu et al. (2019) to explore the spatial layout among rooms rather than steering an agent to desired objects with least steps. Mean- while, [Shen et al. (2019) employ multiple visual representations to generate multiple actions and then fuse those actions to produce an effective one. However, requesting such a large number of visual representations may restrict the transferring ability of navigation system and increases the difficulty of data labeling. Note that Fang et al. (2019) propose a transformer to select the embedded scene memory slot, while our VT is designed to learn expressive visual representations correlated with directional signals.
Target-oriented visual navigation methods aim at steering an agent to object instances of specified category in an unseen environment using least steps. Zhuu et al. (2177) search a target object given in image by employing RI. produce navigation actions based on visual observations. Mousa- vian et al. (2019) take semantic segmentation and detection masks as visual representations and also employ RL to learn navigation policies. Yang et al. (2018) exploit relationships among object cate- gories for navigation, but they need an external knowledge database to construct such relationships. Wortsman al./ (2019) exploit word embedding (i.e., GloVe embedding) to represent the target cat- egory and introduce a meta network mimicking reward function for navigation. Furthermore, Du (et al.) (2020) introduce an object relation graph, dubbed ORG, to encode visual observations and design a tentative policy for deadlock avoidance during navigation. In ORG, object features are extracted from the second layer of the backbone in Faster R-CNN (Ren et al., 2015) and thus not the most prominent ones across the feature pyramid. Additionally, ORG chooses one instance with the highest confidence per category from detection results, and it may be affected by the false positive.
3 VISUAL NAVIGATION REVISIT
In this section, we mainly revisit the definition of object goal navigation and its general pipeline.
3.1 TASK DEFINITION AND SETUP
In this object goal visual navigation task, prior knowledge about the environment, i.e. topologi- cal map and 3D meshes, and additional sensors, i.e. depth cameras, are not available to an agent. RGB images in an egocentric view are the only available source to an agent, and the agent predicts its actions based on the current view and previous states. Following the works (Wortsman et al.) 2019 Du et al. 2020), an environment is divided into grids and agents move between grid points via 6 different actions, consist of MoveAhead, Rotateleft, RotateRight, LookUp, LookDown, Done. To be specific. the forward step size is 0.25 meters, and the angles of turning- left/right and looking-up/down are 45e and 30, respectively. An episode is defined as a success
3

---

A target class T . and start state {2, y. 0r, 0r set randomly at the beginning of each episode, where x and y represent the coordinates, 6, and fy, indicate the view of monocular camera. At each timestamp i,, the agent records the observed RGB image O from its monocular camera. Given the observation O, and the previous state h,, the agent employs Published as a conference paper at ICLR 2021
Visual Transformer (VT)
Spetial
Sponial Erhancod
Observation
Encoder
Locatin
100 2516
x 254
7xx
arboromeent
Decoder
Giobal Feasure
Positional Glonal Deser gxor
Embasiing
Visual Reprttintaioon
Navigation Policy Network
1x
1x6
Previous
Action
1x
2 512
Pre pra-tran Scherne
LSTM
Predicted
Environment
Value
State Represionatt
Frevious Stote
Suparitiio
Action
Visual Transformer Network (VTNet)
Figure 2: Overview our visual transformer navigation system, Our visual transformer navi- gation network (VTNet) involves visual transformer (VT) and navigation policy network. The agent first fuses instance features and spatial features into spatial-enhanced local descriptor. Mean- while, the positional global descriptor is obtained by adding a positional embedding to the global feature. Then the visual representation is decoded from these two spatial-aware descriptors by our VT. Our VTNee is pre-trained with the supervision of optimal navigation actions. The navigation policy network adopts A3C architecture and is trained with navigation rewards after pre-training
when the following three requirements met simultancously: (i) the agent chooses the ending action Done within allowed steps; (ii) a target is in the view of the agent; (iii) the distance between the agent and the target is less than the threshold (L.e. 1.5 meters). Otherwise, the episode will be regarded as a failure.
visual navigation network generate policy (a Or, h2). where a₂ represents the distribution of actions at time t. The agent selects the action with the highest probability for navigation.
3.2 PIPELINE
typical pipeline of visual navigation consists of two parts, visual representation learning and nav- (i) Visual representation learning: To encode the current observation in a compact way, existing works extract visual features from an image and then transform them into vector-based representation, where direct concatenation (Wortsman et al. 2019) graph embed- ding (Du et al. 2020) are used. (ii) Navigation driven by visual features: Once visual features extracted, navigation policy network that generates action in each step for agent will be learned. There are several ways to learn policy networks. such as Q-learning (Watkins & Dayan 1992). , PPO (Schulman al. 2017) and A3C (Mnih et al. 2016). As navigation policy learning not our focus, we adopt the standard Asynchronous Advantage Actor-Critic (A3C) architecture (Mnih) al. 2016). The navigation policy network takes the combination the current visual represen- tation, the previous action and state embedding as input, and outputs the action distribution and value. The agent selects actions with the highest probability from the predicted policy and uses the predicted value to train the navigation policy network.
igation policy learning.
4 PROPOSED VISUAL TRANSFORMER NETWORK
As illustrated in Figure visual navigation system includes two parts: (i) learning visual rep- resentations from RGB observations: (ii) learning navigation policy from the visual representations and previous states. In VTNet, introduce visual transformer (VT) in the first part to
4

---

Published as a conference paper at ICLR 2021
plore the relationship among all objects and their spatial correlations. In VT, we further design two spatial-aware descriptors, ie., a spatial-enhanced local descriptor and a positional global de- scriptor, to allow us extract visual information effectively. Then, VT fuses these two types of descriptors with a multi-head attention operation to produce final visual representations. Moreover, VT enforces visual representations to be highly correlated to navigation signals via our devel- oped pre-training scheme, thus easing the training difficulty of VT and facilitating navigation policy learning.
4.1 EEHANCL LOCAL. DESCRIPTOR
To learn the relationship among all the instances, we first perform object detection and locate all the object instances of interest by detector DETR (Carion et al. 2020). DETR transforms N encoded d-dimension features -.A. xd from the same layer to N detection results, including the bounding boxes, confidence and semantic labels by a feed forward network. Note that ORG (Du et al. 2020) extracts object features from the second layer of the backbone in Faster R-CNN based the pre- dicted bounding-bowd boxes rather than the penultimate layer of the classifier in Faster R-CNN as in the work (Anderson et al. 2018a). Hence, ORG features not the most prominent ones across the feature pyramid and scale-sensitive. In contrast, features extracted by DETR not only contain bounding- boxes and class labels but also scale-robust as features are aligned by DETR decoder, i.e. output from the penultimate layer.
Remark. Benefiting from our VT, we leverage all the information of the detected objects while Du et al. (2020) only select the proposal with the highest confidence in each category. Therefore, the agents in ORG will miss important information from other objects of the same class or might be severely affected if selected proposals are false positive. In contrast, our VT preserves all the information, and thus our agents are able exploit the relationship among instances. This makes our visual representation more comprehensive and essentially different from prior works.
Our local spatial feature is obtained by concatenating the normalized bounding box, confidence and top-rated semantic label for each object. To indicate the target class to an agent, we also concatenate a one-hot encoded target vector RA x1 with our spatial feature xx After obtaining the instance feature and spatial feature, employ multi-layer perceptron (MLP) (i.e., two fully-connected layers with RcLU) and fuse them to a spatial-enhanced local descriptor E N xx so as to act as the key of our VT encoder.
4.2 POSITIONAL GLOBAL DESCRIPTOR
In addition to the spatial-enhanced local descriptor, agents require a global feature to describe the surrounding environment. Similar SAVN (Wortsman et al., 2019). we adopt ResNet18 (He al. 2016) pretrained on ImageNet (Deng et al. 2009) to extract global features of the observations. Given global feature xx we first employ Tx convolution reduce the channel dimension of a high-level activation map from D to a smaller dimension d, where h and w represent the height and width of activation maps, respectively. This ensures that global descriptors have the same dimension the key of our VT.
Unlike previous works that directly concatenate global feature as a part the visual representation, we introduce a positional global descriptor as the query in our VT decoder. A region feature only represents visual contents in each region. To emphasize the region position information, we incorpo- rate positional embedding to each region feature. Then we add positional encoding hxx to the global feature. Let u and v represent the row and column indexes of an image region respectively, and is the index along the dimension d. Our positional embedding is expressed as:
PEz(u,, v) sin H sin
10000 ) V
<<< S i < d
PF =) cos( 10000 )
D<i<> Ae cost 10000 Kig d
(1)
10000 )
Therefore, each global feature represents particular region of the observation. Finally, we shape positional embedded global features into a matrix-basee representation, namely positional global descriptor G Rhuxa
5

---

Published as a conference paper at ICLR 2021
4.3 VISUAL TRANSFORMER
After obtaining our extracted spatial-enhanced and positional global descriptors, we introduce our visual transformer.
Encoder. In order to exploit the spatial relationship between detected instances and observed gions, we attend spatial-enhanced local descriptors to positional global descriptors via a transformer. We first feed the spatial-enhanced local descriptors into the encoder as keys and values by employing multi-head self-attention. Following the transformer architecture (Vaswani et al.) 2017 Fan et al., 2021). each encoder layer consists of multi-head self-attention module and feed-forwer layer.
Decoder. Inspired by human navigation behaviors, we aim to explore the correspondences between observation regions and navigation actions. For example,
an agent notices target lying on the right side of the field of view, it should prioritize to select Rotat t eRi ght. instead of Rotate eeeeeeeeeeee ft. Since each positional global descriptor corresponds to certain region of the observation, we refer to the positional global descriptor as the location query and feed the query into the decoder. Given po- sitional global descriptor G and encoded spatial- enhancee local descriptor L', our attention function of visual transformer decoder is expressed as:
GL vd
Attention I.') = somma
I.).
(2)
4.4 PRE-TRAINING VISUAL TRANSFORMER
We observed that directly feeding the decoded representation from our VT navigation network, we fail to learn successful navigation policy. This is mainly because training a deep VT is very difficult especially when the supervision signals are provided by weak reward from reinforcement learning. Therefore, the decoded features might be uninformative and confuse an agent. The agent would prefer to choose the termination action (often around steps in our experiments) in order to reduce penalties from reinforcement learning.
To address the aforementioned issue, we propose a pre-training scheme for our VT. To be specific, enforce the decoded features to be expressive by introducing an imitation learning task, seen in 2 Concretely, human navigation behaviors can be predicted from the decoded representations in step-wist wise fashion. We use Dijkstra's 's Shortest Path First algorithm generate optimal action instructions as human expert experience. Under the supervision of optimal action instructions, our VT learns imitate the optimal navigation action selection.
In the pre-training stage, we do not employ our navigation network (i.e., LSTM), and previous actions as well states not available. Note that, in navigation network, previous actions, previous states and current visual representations are exploited, as seen in Figure Thus, we replace LSTM with an MLP and predict action distributions based the current visual representation. A cross-entrope loss Lu, = CE(a,, â) is employed to train our VT and the MLP, where is the predicted action, represents the optimal action instruction and CE indicates the cross-entropy function. After pre-training, features from our VT also exhibit strong association with directional navigation signals as only an MIP employed on top of the features. Therefore, the decoded features will facilitate the navigation network training.
5 EXPERIMENTS
5.1 PROTOCOLS AND EXPERIMENTAL DETAILS
Dataset. We perform experiments A12-Thor Thor (Kolve et al., 2017), an artificial 3D environment with realistic photos. It contains 4 types of scenes, i.e., kitchen, living room, bedroom and bathroom. In each type of scenes, there are 30 different rooms with various furniture placements and items. Following Du et al. (2020), we choose 22 categories as the target classes and ensure that there are at least 4 potential targets in each room.
We use the same training and evaluation protocols as the works (Wortsman et al. 2019 Du et al.] 2020). 80 rooms out of 120 are selected as the training set while each scene contains 20 rooms.

---

Published as a conference paper at ICLR 2021
Table 1: Comparison with the otate-of-the-art. We report the average success (%) and SPL. as well as their variances in parentheses by repeating experiments five times. L > represents the episodes which require at least steps.
ALL
I 2 5
Method
Success
SPI.
Success
SPI.
Random
8.0 (1.3)
0.036 (0.00))
0.3
0.001 (0.00))
WE
33.0 (3.5)
0.147 (0018)
21.4 (3.0)
0.117 (0.019)
SP (Yang et al. 2018)
35. (1.3)
0.155 (0011)
22.2 (2.7)
0.114 (0.016)
SAVN Wortsman et al. 2019
40.8 (1.2)
0.161 (0005)
28.7 (1.5)
0.139 (0.05)
ORG (Du et al. 2020
65.3 (0.7)
0.375 (0008)
54.8 (1.0)
0.361 (0009)
ORG+TPN (Du
2020
69.3 (1.2)
0.394 (0010)
60.7 (1.3)
0.386 (oo1))
Baseline
62.6 (0.9)
0,364 (0006)
51.5 (1.3)
0.345 (0007)
VTNet
72.2 (1.9)
0.449 (0,077)
63.4 (1.1)
0.440 (0.099)
VTNet TPN (Du al. 2020
73.5 (1.3)
0.440 (0.009)
63.9 (1.5)
0.440 (0.011)
We equally divide the remaining 40 rooms into validation and test sets. We report the results of the lesting data by using the model with the highest success rate on the validation set.
Evaluation metrics. We evaluate model performance by success rate and Success Weighted by Path Length (SPL). The success rate measures navigation effectiveness and is computed by -. Sn. where N is the number of episodes and S, is success indicator of the n-th episode. We adopt SPL to measure the navigation efficiency, Given the length of the 7-th episode Len, and Lenn
N
its optimal path Lenopt, SPL is formulated as N En=o Sa maz[Len,, Lentope)
Training details. We use two-stage training strategy. In Stage 1, we train our visual transformer for 20 epochs with the supervision of optimal action instructions.
this fashion, we explicitly construct the association between visual representations and navigation actions. In Stage 2, we train the navigation policy for 6M episodes in total with 16 asynchronous agents. We set a penalization -0.001 on each action step and a large reward 5 when agent completes an episode successfully. We adopt DETR as the object detector and fine-tune DETR on the A12-Thor training dataset. In training DETR, applied data augmentation, such resize and random crop. We the Adam optimizer (Kingma & Ba 2014) to update the policy network with a learning rate 10-4 and the pre-trained VT with learning rate 10-3, Our codes and pre-trained model will be publicly released for reproducibility.
5.2 COMPETING METHODS
We compare our method with the following ones: Random policy. An agent chooses actions based on a uniform action probability. Thus, the agent will walk or stop a scene randomly. Scene Prior (SP) Yang et al., 2018) learns a graph neural network from the FastText database (Joulin et al.) 2016 and leverages the scene prior knowledge and category relationships for navigation. Word Embedding (WE) uses GloVe embedding (Pennington et al. 2014) to indicate the target category rather than detection. The association between object appearances and Glo embeddings is learned through trail and error. Self-adaptive Visual Navigation (SAVN) (Wortsman al. 2019) introduces a meta reinforcement learning method that allows an agent to adapt to unseen environments. Ob- ject Relationship Graph (ORG) (Du et al. (020)) is visual representation learning method to encode correlation among categories and employs a tentative policy network (TPN) to escape from deadlocks. Baseline is vanilla version of V'Neet We feed the concatenation of the local instance features from DETR and the global feature to A3C for navigation. Note that, our baseline does not employ spatial-enhanced local and positional global descriptors as well as our visual transformer.
5.3 EVALUATION RESULTS
Improvement over Baseline. Table indicates that VTNeet surpasses the baseline by a large margin on both success rate (+9.6%) and SPL (+0.085). Baseline only resorts to the detection features and global feature for navigation. The relations among local instances and the association between the visual observations and actions are not exploited. This comparison suggests that our VT leads to informative visual representations for navigation, and thus significantly improves the effectiveness and efficiency of our navigation system.
Comparison with competing methods. As indicated in Table I we observe that VTNet signifi- cantly outperforms SP (Yang et al. 2018) and SAVN (Wortsman et al. 2019). Since SP and SAVN
7

---

Published as a conference paper at ICLR 2021
stops
steps
21 steps
steps
Rermole Combrol
SAVIN
Baseline
ORG
VTNat
Figure 3: Visual results of four different models in testing environments. The target objects (i.e., RemoteComtrol) are highlighted by the blue boxes. Green and red curves represent success and failure cases, respectively. The episode produced by our VTNet is successful in reaching the target and use shortest steps. In comparison, ORG takes more steps reach the target. SAVN and Baseline miss both targets.
employ word embedding as a target indicator while VTNet replaces word embedding with our VT, our method achieves expressive object and image region representations for navigation. In addition, SP and SAVN concatenate features from various modalities directly to generate visual represen- tations. The gap between different modalities may not facilitate navigation policy learning. In contrast, benefiting from our pre- pre-training, features from VT are more correlated to navigation actions, thus expediting navigation policy learning.
Our method outperforms the state-of-the ahr method ORG (Du al. 2020) by +2.9% in success rate and +0.055 in SPL. Moreover, when ORG does not employ TPN, the advantage of our method becomes more obvious (6.9% improvement), and this mainly comes from superior visual presen- tations. Since ORG only chooses an object with the highest confidence in each class, the relationship among objects is not comprehensive. On the contrary, our method can exploit all the detected in- stances to deduce the relationships among objects due to our VT architecture. Moreover, since DETR infers the relations between object instances and the global image context, the local features output by the DETR are more informative compared to the object features used in ORG. This can be proved by the result when we use Faster R-CNN as our backbone, as indicated by Table 2 We also (more visual results are provided in the appendix). Furthermore, we
show a case study in Figure 3 also try to employ TPN to improve our navigation policy. As seen in [ VTNet+TPN improves the success rates but the improvement is not much ORG+TPN. This also implies that our visual representations significantly facilitate navigation action selections.
Case Study. As illustrated in Figure 3 SAVN and Baseline both issue the termination command after navigating a few steps (7 and 19 steps, respectively). but fail to reach the target. This indicates that the relationships among categories and VTNee find the target. Since our visual transformer provides clear directional signals, VTNee uses the least steps to find the object.
not clear in SAVN and Baseline. In contrast, both ORG
5.4 VARIANT AND ABLATION STUDY
In this section, we analyze the impact of each component in VTNet, including the spatial-enhanced local descriptor, positional global descriptor, visual transformer that fuses these two spatial-aware descriptors and pre-training scheme.
To illustrate the necessity of the spatial enhancement, we directly use the object features without spatial enhancement. In this case, our network fails to converge because the feed-forward layers that predict bounding-boxes and class labels in DETR are not used in VTNet and spatial information cannot be decoded by the navigation network. Thus, spatial enhancement allows an agent to exploit instance location information explicitly.
As indicated in Table 2 we achieve better navigation performance using instance features from DETR compared to employing Faster R-CNN features following the feature extraction of Du et al. (2000) ("Faster R R-CNN"). Unlike Faster R-CNN, DETR infers the relations between object in- stances and the global image context via its transformer to output the final predictions (i.e., class labels and bounding boxes). Although DETR and Faster R-CNN achieve similar detection perfor- mance (Carion et al. 2020), features extracted by DETR are more informative and robust than those of Faster R-CNN used in ORG. Specifically. ORG extracts features from the second layer of the backbone in Faster R-CNN based on the predicted bounding- boxes to ensure the features are com- parable, but the features are not the most prominent ones across the feature pyramid. Therefore, the
8

---

Published as a conference paper at ICLR 2021
Table 2: Impacts of different components on navigation performances. Faster R-CNN and Faster
R-CNNI represent instance features extracted by Faster R-CNN following Du et al. (020) and
Anderson et al. (2018a), respectively.
w/o global decoder
w/o
w/o
Bascline
VTNet
Method
VTNeI,
pe
Faster
Faster
DETR
Faster
Faster
DETR
R-CNN R-CNN
R-CNN R-CNN
67.0
67.0
71.0
70.1
56.4
57.2
62.6
70.1
70.3
72.2
Success
ALL
(2.8)
(1.4)
(0.7)
(1.3)
(0.9)
(1.1)
(0.8)
(1.0)
(1.2)
(1.0)
0.390
0.373
0.432
0,411
0.319
0.308
0.365
0.396
0.387
0.449
SPL.
(0.021)
(0.013)
(0.009)
(0.009)
(0.007)
(0.008)
(0.010)
(0.010)
(0.012)
(0.007)
54.5
53.2
61.2
60.6
42.5
46.7
51.5
61.7
62.1
63.4
Success
L 2
5
(3.1)
(1.6)
(0.9)
(1.5)
(1.2)
(1.3)
(1.0)
(1.2)
(1.4)
(1.1)
0.357
0.343
0.416
0.395
0.270
0.276
0.345
0.399
0.376
0,440
SPL
(0.021)
(0.017)
(0.010) (0.011)
(0.009)
(0.010)
(0.012)
(0.016)
(0.012)
(0.009)
object features "Faster R-CNN" extracted by ORG are inferior to DETR features, and our navigator employing DETR features outperforms ORG.
Moreover, adopt the instance features from Faster R-CNN following the feature extraction fash- ion of Anderson et al. (2018a) ("Faster R-CNNI"). Thus, we obtain the instance features from the penultimate layer of the classifier in Faster R-CNN. We observe that instance features from DETR also improve the navigation performance compared to the Faster R-CNNI features. We speculate the improvements mainly come from the fact that the features output by DETR decoder have embedded global context information, and those features are more suitable for the feature fusion operations.
As seen in Table 2 first remove the global feature from system ("VTNet w/o global"), and the navigation performance degrades significantly. This validates the importance of global features, which provide contextual guidance to an agent. Moreover, when we remove the positional embed- ding from the global feature ("VTNeet w/o pe"), we observe that both effectiveness and efficiency of the navigation decrease. This indicates that the position embeddings facilitate our VT to exploit the spatial information of observation regions. Furthermore, when we remove the VT decoder ("Vt w/o decoder") and concatenate the global and local descriptors directly, our method suffers per- formance degradation. This demonstrates that our VT plays a critical role in attending the global descriptors to local ones. Additionally, we feed the positional global features into the transformer encoder ("VTNet,, "). The navigation performance of VTNet, is superior to that of ORG but slightly inferior to the performance of our VTTNet. This demonstrates the transformer architecture is effective to extract informative visual representations, and assigning different functions different modules would further facilitate the establishment of mappings in our VT.
When the pre-training scheme is not applied our VT, our agent fails to learn any effective navi- gation policy and thus we do not report the performance. This manifests that our VT pre-training procedure provides a good initialization our transformer and prior knowledge on associating vi- sual observations with navigation actions to agents.
CONCLUSION
In this paper, we proposed a powerful visual representation learning method for visual navigation, named Visual Transformer Network (VTNet). In our VTNet, a visual transformer (VT) has been developed to encode visual observations. Our VT leverages two newly designed spatial-aware de- scriptors, i.e., spatial-enhanced local object descriptor and positional global descriptor, and then fuses those two types of descriptors via multi-head attention to achieve our final visual representa- tion. Thanks our VT architecture, all the detected instances will be exploited for understanding the current observation. Therefore, our visual representation is more informative compared to that used in state-of the--art navigation methods. Benefiting from our pre-training strategy, our VT is able to associate visual representations with navigation actions, thus significantly expediting navigation policy learning. Extensive results demonstrate that our VTNet outperforms the state-of-the-art in terms of effectiveness and efficiency.
ACKNOWLEDGMENTS
This work was supported by the ARC Discovery Early Career Researcher Award (DE200101283), the ARC Discovery Project (DP210102801) and the Data61 Collaborative Research Project.
9

---

Published as a conference paper at ICLR 2021
REFERENCES
Peter Anderson, Xiaodong He, Chris Buchler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answer- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 6077-6086, 2018a.
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, lan Reid, Stephen Gould, and Anton van den Hengel. lision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pp. 3674 3683, 2018b.
Johann Borenstein and Yoram Koren. Real-time obstacle avoidance for fast mobile robots. IEEE Transactions systems, Man, and Cybernetics, 19(5):1179-118) 1989.
Johann Borenstein and Yoram Koren. The vector field histogram- obstacle avoidance for mobile robots. IEEE transactions robotics and automation, 7(3):278-288, 1991.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. arXiv preprint crXiv:2005.12877 2020.
Kevin Chen, Juan Pablo de Vicente, Gabriel Sepulveda, Fei Xia, Alvaro Soto, Marynel Vázquez, and Silvio Savarese. A behavioral approach to visual navigation with graph localization networks. arXiv preprint arXiv:1903.00444 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255, Icee, 2009.
Zhiwei Deng, Karthik Narasimhan, and Olga Russakovsky, Evolving graphical planner: Contextual global planning for vision-and-language navigation. Advances in Neural Information Processing Systems, 33, 2020.
MWM Gamini Dissanayake, Paul Newman, Steve Clark, Hugh F Durrant-Whyte, and Michael Csorba. A solution to the simultaneous localization and map building (slam) problem. IEEE Transactions on robotics and automation, 17(3):229-241, 2001.
Heming Du Xin Yu, and Liang Zheng. Learning object relation graph and tentative policy for visual navigation. arXiv preprint w:Xiv:2007.11018, 2020.
Hehe Fan, Yi Yang, and Mohan Kankanhalli. Point 4d transformer networks for modeling in point cloud videos. In 2021 IEEE/CV Conference on Computer Vision and Pattern Recognition, CVPR 2021, 2021.
Kuan Fang, Alexander Toshev, Li Fei-Fei, and Silvio Savarese. Scene memory transformer for embodied agents in long-horizon tasks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 538-547. 2019.
Weituo Hao, Chunyuan Li, Xiujun Li, Lawrence Carin, and Jianfeng Gao. Towards learning generic agent for vision-and rision-and-annguaee navigation via pre-training. In Proceedings of the JEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 13137-13146, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Ronghang Hu, Daniel Fried, Anna Rohrbach, Dan Klein, Trevor Darrell, and Kate Saenko. Are you looking? grounding to multiple modalities in vision-and ssioo-and-langeee ann-lang navigation. arXiv preprint 1xXiv:1906.00347, 2019.
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classification. arXiv preprint 1607.0/00..//11 2016.
10

---

Published as a conference paper at ICLR 2021
Gregory Kahn, Adam Villaftor, Bosen Ding, Pieter Abbeel, and Sergey Levine. Self-supervised deep reinforcement learning with generalized computation graphs for robot navigation. In 2018 IEEE International Conference Robotics and Automation (ICRA), pp. 1-8. IEEE, 2018.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 11X.6914 2014.
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. A12-THOR: An Interactive 3D Environment for Visual AI. arXiv, 2017.
Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han. Understanding the diffi- culty of training transformers. arXiv preprint arXiv:2004.08249, 2020.
Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter Anderson, Devi Parikh, and Dhruv Batra. Improving sision-and-lanuuaa navigation with image-text pairs from the web. arXiv preprint arXiv:2004. 14973, 2020.
Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, et Learning to navigate in complex environments. arXiv preprint arXiv:1611.0366 2016.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. International conference on machine learning, pp. 1928-1937, 2016.
Arsalan Mousavian, Alexander Toshev, Marek Fišer, Jana Košecká, Ayzaan Wahid, and James Davidson. Visual representations for semantic target driven navigation. In 2019 International Conference on Robotics and Automation (ICRA), pp. 8846-8852 IEEE, 2019.
Giuseppe Oriolo, Marilena Vendittelli, and Giovanni Ulivi. On-line map building and navigation for autonomous mobile robots. In Proceedings of 1995 IEEE International Conference on Robotics and Automation, volume pp. 2900-2900090 2906. IEEE, 1995.
Emilio Parisotto and Ruslan Salakhutdinov. Neural map: Structured memory for deep reinforcement learning. arXiv preprint arXiv: 2017.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-enn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pp. 91-99, 2015.
Nikolay Savinov, Alexey Dosovitskiy, and Vladlen Koltun. Semi-parametric topological memory for navigation. arXiv preprint arXiv: 00653, 2018.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 170v:01707.06447 2017.
Gabriel Sepulveda, Juan Carlos Niebles, and Alvaro Soto. A deep learning based behavioral ap- proach to indoor autonomous navigation. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 4646 4653. IEEE, 2018.
William B Shen, Danfei Xu, Yuke Zhu, Leonidas Guibas, Fei-Fei, and Silvio Savarese. Sit- uational fusion of visual representation for visual navigation. arXiv preprint arXiv: 1900.09073,, 2019.
Tianqi Tang, Xin Yu, Xuanyi Dong, and Yi Yang. Auto-navigator Decoupled neural architecture search for visual navigation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3743-3752, 2021.

---

Published as a conference paper at ICLR 2021
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008. 2017.
Xin Wang. Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, and Lei Zhang, Reinforced cross-modal matching and self-supervised imita- tion learning for rision-language navigation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6629 0229-6638, 2019.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3-4):279-292. 1992.
Mitchell Wortsman, Kiana Ehsani, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Learning to learn how to learn: Self-adaptive visual navigation using meta-learning. In Pro- ceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 6750-6759, 2019.
Yi Wu, Yuxin Wu, Aviv Tamar, Stuart Russell, Georgia Gkioxari, and Yuandong Tian. Bayesian relational memory for semantic visual navigation. In Proceedings of the IEEE International Con- ference on Computer Vision, pp. 2769-276 2779, 2019.
Wei Yang, Xiaolong Wang, Ali Farhadi, Abhinav Gupta, and Roozbeh Mottaghi. Visual semantic navigation using scene priors. arXiv preprint arXiv: 1810.06543, 2018.
Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In 2017 IEEE international conference on robotics and automation (ICRA), 3357-3364. IEEE, 2017.
12

---

Published as a conference paper at ICLR 2021
A
APPENDIX
A.1
FEATURE DETAILS IN VTNET
Local Branch
Instance Feature
Spotial Feature 100 7
Encoder Output 100 256
Local Descriptor
100 256
100 249
Location
100 x 256
VTi enooder
Visual Representation
Global Branch
Global Feature
Positional Embedding
Positiona Global Descriptor
x 512
7 x 256
X 7 x 256
7 x 7 x 256
49 256
VT decoder
Figure 4: Illustration feature flowchart in VTNet.
We extract instance features R 100x and location features IR 100x and concatenate them into a spatial- enhanced local descriptor I 100x256 The local branch integrates semantic labels R 100x bounding boxes 1100x4 confidences 10=00x1 and the target labels R R100x1 for the current observation. In the global branch, the positional embedding R Rxx7x256 is added to the global feature Tx leading to a positional global descriptor IR 256 The spatial- spatial-enhanced local and positional global descriptors are fused by VT encoder and then the visual representation RR 25x is output by VT decoder.
For reproducibility, we illustrate the detailed feature flowchart of our VTNet in Figure 4
A.2 FAILURE CASE STUDY
1.530m 1.5m
1.57 2m > 1.5m
Target: Bowl
Target: Laptop
Figure 5: Visual results of failure cases in testing environments. The target objects (i.e., Bowl and Laptop) are highlighted by the blue boxes. Red lines indicate the distance between the agent and the target object. Gray curves represent trajectories of agents. Both episodes fail because distances (i.e., 1.530m and 1.572m) between the agent and the target in the field of view larger than the threshold (i.e., 1.5m).
As demonstrated in Figure 5) our VT'eet fails to reach targets because the distances between agents and targets larger than the threshold distance (i.e., 1.5m). In these two failure cases, agents find targets but implement the termination action before reaching a position closer than the threshold. Due to the lack of depth information and variances of target object sizes, agent may predict that is within a 1.5 meter radius of the target by mistake and thus terminates current episode.
13

---

Published as a conference paper at ICLR 2021
A.3
CASE STUDY
SAVN
Baseline
ORG
VTNet
Microwave
12 staps
8 steps
14 steps
steps
Bowl
25 steps
100 stepss
19
steps
Chone
4B steps
7D steps
12
sceps
Lamp
steps
steps
steps
stepps
AlarmClock Clock
1D steps
20 steps
100 steps
steps
AlarmClock
28 steps
17 steps
21 steps
27 steps
Sink
13 steps
steps
steps
steps
Figure 6: Visual results of four different models in testing environments, We compare VTNee with SAVN (Wortsman et al. 2019), Baseline and ORG (Du al.] 2020). The target objects highlighted by the blue boxes. Green and red curves indicate success and failure cases, respectively. Our VTNett successfully reaches targets and uses the shortest steps.
14

---

Published as a conference paper at ICLR 2021
A.4 DIFFERENT VISUAL TRANSFORMER ARCHITECTURES.
Table 3: Comparison of visual transformer architectures. We report the pre-training accuracy on the
validation dataset, navigation success rate and SPL the test dataset.
Multi-head numbers
Encoder layers
Decoder layers
Accuracy
Success
SPL
0.722
71.2
0.433
4
2
2
0.723
72.2
0.449
4
4
0.715
70.0
0.419
0.707
70.4
0.422
2
2 0.718
70.9
0.436
8
4 4
0.710
68.9
0.423
6
0.701
68.2
0.41I
We construct different transformer architectures by varying the number of encoder and decoder layers. Table summarizes the performance of these architectures. We observe that as a visual transformer becomes too deep, transformer may fail to converge an optimal policy. On the other hand, a transformer with single encoder and decoder layer does not have sufficient network capa- bility to produce representative features. The highest success rate is achieved when VT contains four multi-head self-attention mechanism modules and two layers in the encoder and decoder.
A.5 NECESSITY OF PRE-TRAINING SCHEME
20.0
VTNet VTNet w/o Pre-training
17.5
Episode length
12.5
10.0
5.0
2.5
1000
2000
3000
4000
5000
6000
Training episodes
Figure 7: Average episode lengths of VTNet and VTNet without pre-training during training. We compare VTNet with VTNet without pre-traie trainininng scheme. Blue and orange curves represent VTNet and V'TNet w/o pre-training, respectively.
As demonstrated in Figure 7 our VTNee spends nearly 10 steps per episode in training, while the navigator w/o pre-training scheme often fails reach targets stops around 5 steps after being trained tens of thousands of episodes. Due to the large parameters and complex architectures of transformers, it is often difficult to train our transformers from scratch (Liu et al.) 2020). Without good initialization for our VT, is very difficult to learn VT and policy network in end-to-end fashion with RL rewards. This is because the visual representations from VT are not informative even meaningless and the inferior visual representations would harm policy network learning. As a result, the navigation policy network may be trapped into a local minimum (i.e., terminating navigation early to avoid more penalties) and our VT cannot receive positive rewards from preceding trajectories.
15

---

Published as a conference paper at ICLR 2021
A.6
ADDITIONAL VISUALIZATION RESULTS
Observation
Instances
Activation Map
Action
Roceeeight
StoveBurmer
RotateLeft
Monsllasal
Pan
Plate
Bols
Look
MoveAhead
GarbageCan
Dore
RocoeeRigte
Kettle
Lookilp
MoveAhead
Souuttat
StoveBurner
LOORDONT
Toaster
Figure 8: Visualizations of attention scores. The target classes (i.e., StoveBurner, GarbageCan, Kettle) are highlighted by green bounding boxes. Our agent detects the instances of interest and then attends the detected instances the global image regions by our VT. We observe that high attention scores are obtained on the areas corresponding to the targets. Guided by the visual representations, the agent selects actions approach the targets.
16


Reviews:
[{"review_id": "Ug5Atue5oZX", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper demonstrates a model that uses the Transformer to encode the visual features that appeared in the visual input image during navigation. The model is firstly pre-trained under imitation learning objective with self-generated shortest-path trajectories. The empirical results show that the model used in the paper outperforms previous methods on AI2-THOR environment. The authors also show some studies on the contributions of each component in the model.\n\nPaper strengths:\n\n+ The proposed method further show that the Transformer is a powerful model for feature extraction\n\n+ The authors demonstrate one method to make the training of Transformer work, i.e. pre-training transformers using shortest-path trajectories\n\n+ Empirical result support the authors' claims.\n\n+ A thorough ablation study and discussions are provided.\n\nCons:\n\n- The paper adopts the Transformer and adapted it into the navigation problem. No new architecture/model is proposed.\n\n- It seems that a similar usage of Transformer already appeared in the vision-and-language navigation task [1]. The paper also shows that pre-training of navigation tasks using Transformers can help to boost the performance. \n\nMinor: Two missing citations [2,3] that are potentially relevant. \n\n[1] Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-training\n\n[2] Evolving Graphical Planner: Contextual Global Planning for Vision-and-Language Navigation\n\n[3] Are You Looking? Grounding to Multiple Modalities in Vision-and-Language Navigation\n\n--\n\nI've read the authors' response and would like to maintain my original score", "extra_scores": {}, "extra_text": {}}, {"review_id": "OOSgzqW2IDO", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Paper summary**\n\nThe paper addresses the problem of navigation towards objects (ObjectNav) in a virtual environment. The idea of the paper is to incorporate spatial information of objects using a transformer-based framework called Visual Transformer Network. The paper compares the results with a number of state-of-the-art ObjectNav models and provides an ablation study. The results have been reported on the AI2-THOR framework.\n\n**Paper strengths**\n\n- The idea of incorporating object information using transformers for a navigation agent is new.\n\n- The proposed method outperforms a number of strong baselines.\n\n- The ablation studies show that the introduced components are effective.\n\n**Paper weaknesses**\n\n- It is hard to understand some parts of the paper. For example, the introduction discusses details such the difference between DETR and Faster RCNN or difficulty of training the transformers. It is difficult to understand these details without knowing the proposed method. The introduction should provide a high-level overview of the paper instead of these types of details. Also, the paper requires proof reading. There are several sentences with grammar issues.\n\n- It is a bit strange that nothing is learned without the imitation pre-training. It would be good to dig deeper and provide a better explanation for why this happens.\n\n- Equation 1 is not clear. A brief explanation would help.\n\n- I recommend running the method on some other frameworks which include slightly larger scenes to see if the method generalizes to those as well. RoboTHOR (https://github.com/allenai/robothor-challenge) is very close to the framework used in this paper so it might be a good choice for these experiments.\n\n**Justification of rating**\n\nOverall, I am leaning towards accepting this paper since it introduces a new way for incorporating object information and it outperforms strong object navigation baselines. Writing is the main issue of this paper. \n\n**Post-rebuttal**\n\nI read the rebuttal and the other reviews. The rebuttal addresses my concerns to some extent (writing has improved in the revised version, but it still has some issues). So I am going to keep my rating. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Diu0X8TZTdJ", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### summary\nThis paper introduces transformer network to visual navigation, specifically, object-goal navigation. It also develops several new feature descriptors as the input of the transformer encoder and decoder. To properly train the whole model, a supervised pre-training stage is used to warm-up the transformer model. Great performance has been achieved on AI-THOR benchmark.\n\n### pros\n1. Lots of people must have thought to use transformer to replace the RNN/LSTM in lots of visual navigation framework. This paper provides a good example. Most importantly, this paper focus on the representation learning part of the whole pipeline, which isn't that straightforward of how to use a transformer.\n2. The writing is mostly clear with clear motivation and background discussion. \n3. The performance boost, especially SPL, is relatively significant compared to previous SOTA, and the ablation studies have verified most of the design choices.\n\n### cons\nThere are a couple of things which are not clear to me, or confused me when I was reading the paper:\n1.  The writing in the approach section isn't very clear. First, it would be much better to define clear notations for all the features/descriptors, and use such notations in the figure. The current writing uses \"instance feature\", \"global feature\", \"positional-global\", \"spatial feature\", \"spatial-enhanced\"..., which are a little bit confusing to me. Second, I think most details are properly ignored in Fig.2. It becomes not as informative as the detailed version (Fig.4) in Appendix. Note that these two figures are not consistent that the \"add\" symbol for positional enhancement is missing is Fig.4. I also suggest that the positional embedding blob not crossing the arrow of global feature, they are just added together.Third, Sec.4.2 writes \"we first reduce the channel dimension of a high-level activation map from D to a smaller dimension d\", how the reduction is done exactly? From appendix it seems like a 256-dim vector is transformed into 249-dim. Fourth, $h$ and \"w\" are abused. In figure, they are annotated  on the long side of the tensor, in eq (1) they seem to be the output of positional embedding, and in Sec.4.2 description they are the resolution of 7. Similarly, $L$ is abused as it means input of encoder in Sec.4.1 but output of encoder in Sec.4.3. Let me stop here, but these things make the approach not super clear to me. \n2. In Sec.4.1, I'm not fully convinced of the statement of faster rcnn even thought the experiments empirically verified it. Faster RCNN w/o FPN only output features after \"conv4+ROI-pooling\" (ResNet-101-C4 variant). Why is it blamed for scale-sensitive? Actually, what does scale-sensitive mean here? Why DETR doesn't suffer from it? Honestly I don't think that's the reason why Faster RCNN performs worse.\n3. Also, I'm not fully convinced of the statement of the \"early stopping\" in Sec.4.4. The penalties are the same for different model in RL,  why this transformer based representation learner suffers from \"early stopping\"? Is there a plausible explanation? It's fine that you cannot conclude something for sure because transformers are always hard to train, but the statement in paper reads not super convincing to me.\n4. Sec.5.1 SPL formulation seems to be wrong? The success indicator seems missing? The current equation is simply a ratio between any episode length over the optimal length regardless whether it's an success episode or not.\n5. Why not also adding global features into the transformer encoder? For example, reshape and concat with the input. Is the encoder supposed to be local?\n\n### misc\n1. The best results of VTNet in Tab.1 used TPN. It might be better to introduce TPN in Appendix for completeness. \n2. Variance is not reported in Tab.1, which is uncommon for RL/control paper. \n3. Because transformer has attention module and the relationship can be easily visualized. I was expecting more interpretation/visualization like Fig.1 right to show the proposed methods actually attend to proper areas. The numbers are hard to tell what do each modules do exactly.\n\n### questions\n1. Just to make sure I understand correctly, the instance feature (100x249) and spatial feature (100x7) are fed into a MLP for fusion? Can you describe the archi?\n2. Local spatial feature contains the normalized bounding box, confidence and top-rated semantic label. Is the semantic label the class index (1,2,...,C)? why not use a one-hot embedding or something?\n3. Is AI2-THOR the most popular benchmark for object-goal nav? I have seen lots of prior paper running on habitat. What's the specific reasons of using AI2-Thor over habitat? \n\nPlease address my questions. I'm looking forward to discussing with the authors and the peer reviewers. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "7IqYKzVLUBT", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary \nThe paper proposes Visual Transformer Network which encodes the relationship between all detected object instances in a frame and uses it for navigation. The paper uses DETR for object detection and learn an association between local descriptors (from the object detector) with global descriptors (ResNet18) using the proposed VT model. They show that using VT improves performance on the object navigation task in AI2-THOR simulator compared to existing methods. \n\nStrengths \n- The paper proposed a novel transformer architecture that learns an association between local object descriptors with global image region features so that actions can be grounded to visual regions in the image. \n\n- Different from prior work, the paper uses all the objects detected for a label instead of just the most confident detection. \n\n\nWeaknesses \n\n- The paper doesn't fully address why DETR performs better than FasterRCNN features. Appearance features from FasterRCNN have been widely used for several downstream tasks in Vision and Language Navigation[1], Vision and Language tasks[2]. From the experiments, it's not clear why DETR is doing better than Faster-RCNN especially when the detection accuracy of DETR is also better than Faster RCNN.\n\n- Additionally, I didn't fully follow how authors obtain the appearance features from Faster RCNN based method. The authors mention that object appearance features are extracted from different layers of a backbone network. How is it different from the approach taken by Bottom-Up, Top-Down[3] paper in which 2048-dim appearance features are extracted for each visual region?   \n\n- The experimental setup isn't fully reflective of the object goal navigation task. The experiments are conducted in AI2 thor scenes which only contain one room. It's not clear, how this method will perform when evaluated on significantly more complicated environments like Matterport / Gibson [4]. Specifically, I am interested in how will the proposed architecture perform when the goal object is not in the same room as the agent. \n\n- The navigation task is also made simpler by discretizing into a grid. Single room environments and discrete grids simplify a lot of navigation-related challenges and the authors don't discuss how the proposed architecture will generalize to more complex object navigation tasks. \n\n- The use of spatial embeddings as well as appearance embedding isn't all that surprising. Existing work including Du et al. uses bounding box coordinates to help learn spatial associations between objects. \n\nOther questions: \n- Instead of pre-training without employing the navigation policy, did the authors try using shortest-path based demonstrations to help learn the navigation policy as well? In the first stage, the navigation policy learns using imitation learning and then finetuned with A3C? \n\n- What is the step size of the agent for the forward step? What are the turn angles for Turn-left, Turn-right actions? What are the tilt angles for look-up and look-down actions? \n \n- What's the reason for improvement over ORG (in absence of TPN). Is it superior visual representations (Faster RCNN vs DETR) or the fact ORG only chooses objects with the highest confidence while VT uses all the detected objects? \n\n- How does the agent learn long-term associations between objects across multiple frames. In my opinion, the proposed architecture puts all the burden of learning these long-term object relationships across multiple frames on the LSTM policy since the VT only learns association within a single frame. \n\n[1] Improving Vision-and-Language Navigation with Image-Text Pairs from the Web; Majumdar et al. \n\n[2] Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks; Li et al. \n\n[3] Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering; Anderson et al. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses the problem of visual object navigation by defining a novel visual transformer architecture, where an encoder consisting of a pretrained object detector extracts objects (i.e. their visual features, position, semantic label, confidence) that will serve as keys in an attention-based retrieval mechanism, and a decoder computes global visual features and positional descriptors as a coarse feature map. The visual transformer is first pretrained (using imitation learning) on simple tasks consisting in moving the state-less agent / camera towards the target object. Then an RL agent is defined by adding an LSTM to the VTNet and training it end-to-end on the single-room subset of the AI2-Thor environment where it achieves state-of-the-art performance.

After rebuttal, all four reviewers converged on a score of 6. The reviewers praised the novelty of the method, extensive evaluation with ablation studies, and the SOTA results. Main points of criticism were about clarity of writing and some explanations (which the authors improved), using DETR vs. Faster R-CNN, and the relative simplicity of the task (single room and discrete action space). There were also minor questions, a request for more recent transformer-based VLN bibliography, and a request for a new evaluation on RoboThor. One area of discussion -- where I empathise with the authors -- was regarding the difficulty of pure RL training of transformer-based agents and the necessity to pre-train the representations.

Taking all this into account, I suggest this paper gets accepted.

Decision: Accept (Poster)