A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Execution
Authors: Valts Blukis
Chris Paxton
Dieter Fox
Animesh Garg
Yoav Artzi
Venue: CoRL2021 Poster (2021)
Keywords: vision and language
spatial representations
semantic mapping
representation learning
instruction following

Abstract: Natural language provides an accessible and expressive interface to specify long-term tasks for robotic agents. However, non-experts are likely to specify such tasks with high-level instructions, which abstract over specific robot actions through several layers of abstraction. We propose that key to bridging this gap between language and robot actions over long execution horizons are persistent representations. We propose a persistent spatial semantic representation method, and show how it enables building an agent that performs hierarchical reasoning to effectively execute long-term tasks. We evaluate our approach on the ALFRED benchmark and achieve state-of-the-art results, despite completely avoiding the commonly used step-by-step instructions. https://hlsm-alfred.github.io/

A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Execution
Valts Blukis Chris Paxton', Dieter Fox!.3, Animesh Garg! Yoav Artzi
INVIDIA
CComell University University of Washington
University of Toronto, Vector Institute
Natural language provides an accessible and expressive interface to specify long-term tasks for robotic agents. However, non-experts are likely to specify such tasks with high-level instructions, which abstract over specific robot actions through several layers of abstraction. We propose that key to bridging this gap between language and robot actions over long execution horizons are persistent representations. We propose a persistent spatial semantic representation method, and show how it enables building agent that performs hierarchical reasoning to effectively execute long-term tasks. We evaluate our approach on the ALFRED benchmark and achieve state-of-the-art results, despite completely avoiding the commonly used step-by-step instructions. https: //hl sm- al fred. gi github.io.
Abstract:
Keywords: vision and language, spatial representations
Introduction
Mobile manipulation a home environment requires addressing multiple challenges, including exploration and making long-term inference about actions to perform. In addition to reasoning, robots require an accessible, yet sufficiently expressive interface to specify their tasks. Natural Language provides intuitive mechanism for task specification, and coupled with advances in automated language understanding, is increasingly applied to embodied agents [e.g.. 1-11].
In this paper, we study the problem of learning to map high-level natural language instructions to low-level mobile manipulation actions in an interactive 3D environment [12]. Existing work largely studies language tightly aligned to the robot actions, either using single-sentence instruc- tions [e.g., 1, 2, 5, 9] or sequences of instructions [13-18]. In contrast, we focus on high-level instructions, which provide more efficient human-robot communication, but require long-horizon reasoning across layers of abstraction to generate actions not explicitly specified in the instruction.
Robust reasoning about manipulation goals from unrestricted high-level natural language instruc- tions has variety of open challenges. Consider the instruction secure two discs a bedroom safe (Figure 1). The robot must first locate the safe in the bedroom. It then needs to distribute the actions entailed by secure to two objects (two discs), each requiring a distinct sequence of actions, but tar- geting the same safe. It is also required to map the verb secure to its action space. In parallel, the robot must address mobile manipulation challenges, and often can only identify required actions as it observes and manipulates the world (e.g., if the safe needs to be opened).
We propose to construct and continually update spatial semantic representation of the world from robot observations (Figure 2). Similar to widely used map representations [19-22], we retain the spa- tial properties of the environment, allowing the robot to navigate and reason about relations between objects, as required to accomplish its task. We propose the Hierarchical Language-cone -onnuiig-oondiioonedd Spatial Model (HI.M),, a hierarchical approach that uses our spatial representation as a long-term memory to solve long-horizon tasks. HLSM consists of a high-level controller that generates sub- goals, and a low-level controller that generates sequences of actions to accomplish them. example (Figure 1), the sequence of subgoals is (pick up a CD, open the safe, put the CD in the safe, ...). each requiring a sequence of actions. The spatial representation allows selecting subgoals that use previously observed objects outside of the agent's 's view, to decide about needed exploration.
our
We evaluate our approach on the ALFRED [12] benchmark and achieve ofate-of-the-art results without using the low-level instructions used by previous work [16-18, 23), neither during training 5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

Egocentric
RGB Observations
Subgoals
RrxurOueeenCC)
MexeFonwvo
ROTATERIGHT
OrenDeusst
Task: Secure two discs in bedroom safe
Figure 1:
Illustration of the task and hierarchical formulation. The agent receives high-level task in natural language. needs to map RGB images navigation and manipulation actions to complete the task.
nor at test-time. This paper makes three key contributions: (a) a modular representation learning approach for the problem of mapping high-level natural language task descriptions to actions in 3D environment; (b) method for utilizing spatial semantic representation within hierarchical model for solving mobile manipulation tasks; and (c) state-of-the-arr performance on the ALFRED benchmark, even outperforming all approaches that use detailed sequential instructions.
Related Work
Natural language has been extensively studied in robotics research, including with focus on instruc-
tion [1, 24], reference resolution [25], question generation [26-28], and dialogue [4, 29, 30]. Most work in this area has considered either synthetic instructions of relatively simple goals [7, 31-33,, 3, or natural language instructions where all intermediate steps explained in detail [5, 12-14, 34-38). In contrast, we focus on high-level instructions, which are more likely in home environments [39].
Representation of world state, action history, and language semantics plays central role in robot systems and their algorithm design. Symbolic representations have been extensively studied for in- struction following agents [1-4, 19, 20, 39-45]. While they simplify the symbol grounding problem and enable robustness, the ontologies on which they rely on are laborious to scale to new, unstruc- tured environments and language. Representation learning presents alternative by learning to map observations and language directly to actions [5, 8, 9, 11, 13, 34). World state and language semantics are represented with vectors [13] by memorizing past observations [8. 17). Modelling improvements have enabled these approaches to achieve good performance on complex navigation tasks [7, 9, 11, 13, 14. 37], success that has not yet translated to mobile manipulation [12, 46, 47].
We propose integrating a semantic voxel map state representation within a hierarchical repre- sentation learning system. Similar semantic 2D maps have been successfully used in naviga- tion [7, 8, 48, 49] and more recently even in mobile manipulation instruction-follont followin tasks [23]. We extend these maps to 3D and show ttate-of-the-we results challenging mobile manipulation benchmark. Our map design is related to sparse metric, topological and semantic maps [10, 19- 21, 50] that have enabled grounding symbolic instruction representations. Our map does not impose a topological structure or require reasoning about object instances, instead modelling a distribution over semantic classes for every voxel.
3 Problem Definition
Let A be the set of agent actions, and S the set of world states. Given natural language instruction L and an initial state 8p S, the agent's goal to generate an execution (s0, (0, 81, .......... ST, aT), where E A is an action taken by the agent at time 8t E S is the state before taking a,, and $1+1 = T(s;, under environment dynamics T S S S. The state 8; is defined by the environment layout and the poses and states of all objects and the agent. The agent does not have access to the state 8t, but only to an observation o,, An observation (4, P., of. L) includes first-person RGB camera image 4t the agent's pose Pt. one-hot encoding of the object class the agent is holding niS, and the instruction Thehe task is considered successful if all goal-conditions corresponding to the task L true at the final state sy. Partial success is measured as the fraction goal-conditions that have been achieved.
The ALFRED dataset includes sets of seen and unseen environments. of actions A Anav U Aint. includes parameter-free navigation actions Anav (MOVEAHEAD, ROTATELEFT, ROTATERIGHT) and interaction actions Aut
The set

---

E Environment
Observation
Action ag
4
Mimor
Floor
MONTAHEA
Laundry Hamper
πI"
Observation Model F
Subgoal 9k OPEnOBueC((Sare)
Pilow
Semantic Voxel Map V
Agent Pt
Segmentation
Sanondss Ie
Depth
Fostion
πI
Instruction
Figure 2: Model architecture consisting of observation model, high-level controller (n"), and low-level controller () The observation model updates semantic voxel map state representation from RGB obser- vations. x predicts the next subgoal given the instruction and the map. T outputs a sequence of actions to achieve the subgoal. The semantic voxcl map is visualized in the middle with agent position illustrated as a black pillar. ans the current sugoal argument mask in yellow. Other colors are different segmentation classes. Saturated voxels are observed in the current timestep.
(PICKUP, PCT, TOGGLEON, TOGGLEOFF, OPEN, CLOSE, SLICE} parameterized by a binary mask that identifies the object of the interaction in the agent's current first-person view. We compute Pt and ry using dead-reckoning from RGB observations and actions.
Hierarchical Model with a Persistent Spatial Semantic Representation
We model the agent behavior with a policy that maps instruction L and the observation at time t to an action at. The policy made of an observation model and two controllers: high-level controller n and a low-level controller nh. The observation model builds spatial state representation & that captures the cumulative agent knowledge of the world time t. 8, used by both n for high-level long-horizon task planning, and x for near-term reasoning, such as object search, navigation, collision avoidance, and manipulation. Figure 2 illustrates the policy.
The high-level controller " computes a probability over subgoals. A subgoal g is tuple (type, arg arg"), where type é Ant an interaction type (e.g., OPEN. PICKUP), arg is the semantic class of the interaction argument (e.g.. SAFE, CD), and arg" is a 3D mask identifying the location of the argument instance. In ALFRED, each interaction action in the set Amt corre- sponds to subgoal type. When predicting the k-th subgoal at time f, π considers the instruction L. the current state representation & and the sequence of past subgoals (gr, )ick- During inference, sample from л #! Unlike arg max, sampling allows the agent to re-try the same or different subgoal incase potentially random failure (e.g., if MUG was not found, pick up a CUP).
The low-level controller n is given the subgoal as its goal specification at time I. At every timestep j > t, -h maps the state representation &y and subgoal gl to an action aj. until it outputs one of the stop actions: GPASS or aFAIL to indicate successful or failed subgoal completion.
The execution flow is follows. At time 0 the initial observation On is received. At each timestep. we update the state representation & using the observation model. there is no currently active subgoal, sample apAss. we increment subgoal counter k. If is @FAIL. we discard the current subgoal k. We repeat sampling subgoals and actions until executable action o, is sampled. We execute a,, increment the timestep I, and receive the next observation o,. The episode ends when the subgoal gSTOP is sampled the horizon Tiaa is exceeded. Algorithm ?? in Appendix ?? describes this process.
subgoal gA: from ? and then sample action a, from rh. If a, is
4.1 State Representation
The state representation & time I captures the agent's current understanding the state of world, including the locations of objects observed and the agent's relation to them. The state repre- sentation is tuple (V.S, 10, og Pt). The semantic map V E [0. x is 3D voxel map that for every position indicates which of the E [1, CJ object classes are present in the voxel. The

---

observability map Vi E position has been observed. The inventory vector uy E {0. 1] indicates which of the C object classes the agent is currently holding. The agent pose Pt (x,, y, (p: is specified by the 2D position (z. y), pitch angle wpp and yaw angle wy.
is 3D voxel map that indicates whether the corresponding
We also compute 2D state affordance features AFFORD($;) E [0, 1]7xXXÝ in top-down view that represent each position with one or more of seven affordance classes (pickable, receptacle, togglable, openable, ground, obstacle, observed). Each [AFFORD( (T.2,) least one of the voxels at position (I, y) has affordance class otherwise it is zero. AFFORD($;) is suited for object class agnostic reasoning, for example predicting a pose to pick up an object.
1.0 if
4.2 Observation Model
The observation model F(,,-1, o.. 9s) updates the state representation with new observations. considers the current subgoal en to actively acquire information relevant to Jk- The computation of F consists of three steps: perception, projection, accumulation.
Perception Step We predict semantic segmentation I and depth map I," from the RGB observation I. We use neural networks pre-trained in the ALFRED environment. The semantic segmentation [45)(n,)) a distribution over C object classes at pixel (u, v). The depth map [4, (.,)) a binned distribution over B bins.? We also heuristically compute a binary mask M," that indicates which pixels have confident depth readings. We allow more confidence slack in pixels that correspond to the current subgoal argument arg; according to I,S. Appendix ?? provides further details. We perception models based the U-Net [51] architecture, but framework supports other, potentially more powerful models as well (e.g. 53J).
Projection Step We use a pinhole camera model to convert depth 1P and segmentation 18 to a point cloud that represents each image pixel (u, v) with a 3D position (z, v) E RXxXX2 x and a semantic distribution [4 [u.)(..). We use arg maxg (I,)) to compute the 3D positions, and discard points at pixels (u, v) when the binary mask value is [MP) ((,,)) = 0. We construct a discrete semantic voxel map V,S E [0, 1,]1XXxxXXX where X, Y, and Z the width, height, and length. The value at each voxel [V/ (a....) the element-wise maximum of the segmentation distributions [151(a across all points (u, v) within the voxel. We additionally compute a binary observability map V0 (0, xxxXxx that indicates the voxels observed at time t. A voxel is observed if it contains points, or if a ray cast from the camera through the voxel centroid has expected depth greater than the distance from the camera to the centroid.
Accumulation Step We integrate ViS and Vi0 into a persistent state representation:
(1) This operation updates each voxel with the most recent semantic distribution, while retaining the values of all voxels not visible at time t. The output of the observation model is the spatial state representation & (V28, vo ouf Pt). The inventory of and pose P. are taken directly from or.
v x V. + VE x (1 (.0)
ve = max V70)
4.3 High-level Controller "T
At timestep t, when invoked for the k-th time, the input to x Hf is the instruction L, the sequence of past subgoals (g.) and the current state representation št. The output is the next subgoal M). ). Figure illustrates the high-level controller architecture. gk = (type,, arg, argg)
Input Encoding We encode the text L using a pre-trained BERT [54] model that we fine-tune during training. We use the CLS token embedding as the task embedding &²_ We encode the state representation & to account for classes of all observed objects, and the object that the agent is spatial
holding: o ($x) = (of; (ax,). (V,5)) where max.a.y;)) is max-pooling operation dimensions and [ denotes concatenation. We compute the representations of previous subgoals as (R) k-1 where REPR(g;) is the sum of sinusoidal positional encoding [55] of index and
'W assume a known mapping between object semantic classes and affordance classes. We use B uniformly spaced depth bins (0, AD, 2Ap,.., (B 1)An) J. where Ap is depth resolution. We suggest Ap should be less than 50% of the voxel size. We used voxels with edge length 0.25m.

---

Semantic Map
V
Spatial MaxPool
H
Subgoal hartory tarer
ippresntation
Birdk-eye
Inventory v/?
AFFORD
State ftahurs
Asvowajt))
EGoUENTICC TRANS
Statt Encoder
Sliee
Argument
Whard
Subgoal
Language L.
BERT CL.
REFINER
argg
Language
argi
Subgooa History (rypef. arg )
typb
Trazsforuset Fineoder
Dense MLP
Sample
H--1
Ployped 1)
Subgool History Encoder
P(ang) (sypel))
Subgoai Predictor
Figure Illustration the high-level controller !f (Section 4.3).
learned embeddings for type, and arg, We process this sequence with two-layer Transformer autoregressive encoder [55] compute (01)==)- We take db as the subgoal history embedding vector. We additionally encode the argument mask information arg," from the subgoal history in an integer-valued subgoal history tensor H N XxXXxX where
is the number of times interaction action type was performed 2D position (xr, y) in the birds-eye view:
k-1
Ha-1)(sa))
max((arge l(n.,)))
axd
Subgoal Prediction We concatenate the three representations h(a 10 of; o/-1l. We use densely connected multi-layer perceptron [56] to predict two distributions P(type; h(.,)) and P(arg typp,, ((,,)),, from which we sample subgoal type type; and argument class arg
The remaining component of the subgoal is the action argument mask arg, Let [V;") (arge) be voxel map that only retains the object information for objects of class arg in the semantic map V,S. We refine it to identify a single object instance. We compute a birds-eye view representation:
(AFFORD(&)): H (
Leype,
(3)
where AFFORD($ is a birds-eye view state affordance feature map (Section 4.1) and leype, is a one-hot encoding of type,. 3 Finally, we compute the 3D argument mask argl M [0, x|xxXxXXXX xz.
argk REeNee((GooRRASFFO)
P)).
(4)
where EGOTRANSFORM(X Pt) transforms the map to the agent egocentric pose P.. REFINER is neural network based on the LingUNet architecture [36], and o- is the language embedding. The refined arg, M is [0, 1]-valued 3D mask that identifies the instance of the interaction argument object. If the object is believed be unobserved, then arg/. contains all zeroes. The controller output the subgoal gk (typeş; arg )r
4.4 Low-level Controller (TT²
The low-level controller < is conditioned on the most recent subgoal 9b (type argā arg."" ). M)
At time 1, maps the state representation & an action a₁. combines engineered and learned components. Appendix ?? provides the implementation details. The controller v invokes a set of procedures: NavigateTo, SsupleeEpploraaiiooootttt. and InteractMask. Their invokation follows pre-specified execution flow across multiple timesteps. First, we perform a 360 rotation to observe the nearby environment. If no ob- jects of type argī are observed, we explore the environment by sampling a position (æ, v) www.lexxpooraaiioooooott..oo
SampleinteractionPoss
navigating there using the procedure NavigateTo and performing a 360 rotation. We repeat exploration until voxel in V contains the class arg with >50% probability. To interact with an object, we sample an interaction pose (a, y. wy Wp) = SampleInteract onPose (5t, 9k), invoke NavigateTo(r,, y, 8.) reach the position (x.y), and then rotate according to yaw and pitch angles (wyy Wp). Finally, we generate the egocentric interac- tion mask mask, = InteractMaski & arg.
$1),
M) ), and output the interaction action (type,, mask;).
2 denotes multiplication of a X x Y tensor with a K --dimensional vector to obtain K x X x tensor. [ denotes channel- wise concatenation.
5

---

All procedures use the spatial representation St. NavigateTo navigates to a goal position us- ing value iteration network (VIN) [57] that reasons over obstacle and observability maps from samples positions on the boundary of observed space in 8t- SampleInteractionn onne uses a learned neural network NAVMODEL to predict a distributon of poses from which the interaction Ik will likely succeed. InteractMask uses the segmentation image 1 and the 3D argument mask arg, to compute the first-person mask of the target object.
8,.
5
Learning
The policy contains four learned models: the segmentation and depth networks, n and the navi- gation model NAVMODEL. used by x². We train all four networks independently using supervised learning. We assume access to training dataset D {(L6) " of high-level natural lan- guage instructions LG) paired with demonstration execution E() in a set of seen environments. Each execution E(i is sequence of states and actions {sg (5) ..... (i) (f)). We denote Np the total number of states in dataset D, and NG the total number of subgoals.
We process D into three datasets. The perception dataset DP (((T)() (Top() [1 in- cludes RGB images [T](0) with ground truth depth [T0](1) and segmentation [ The subgoal dataset D {(L(), (()) (0) (oo"))==)) contains natural language instructions ( state repre- sentations a(") at the start of k-th subgoal execution, and sequences of the first &s subgoals (0) )) extracted from E() The navigation dataset D.N {($(() gg(h) ))
consists of state rep- resentations g(0), subgoals g()) and agent poses p(t) at the time of taking the interaction action corresponding to subgoal g (*)) The state representations g() in datasets D and D'N are constructed using the observation model (Section 4.2), but using ground-true depth and segmentation images.
We train the perception models on DP and the H on D to predict the k-th subgoal by optimizing cross-entropy losses. We use D to train the navigation model NAV MODEL by optimizing a cross- entropy loss for positions and yaw angles, and an L2 loss for the pitch angle.
Experimental Setup
Environment, Data, and Evaluation We evaluate our approach on the ALFRED [12] benchmark. It contains 108 training scenes, 88/4 validation seen/unseen scenes, and 107/8 test seen/unseen scenes. There are 21,023 training tasks, 820/821 validation seen/unseen tasks, and 1533/1529 test seen/unseen tasks. Each task specified with high-level natural language instruction. The goal of the agent is to map raw RGB observations to actions to complete the task. ALFRED also pro- vides detailed low-level step-by-step instructions, which simplify the reasoning process. We do not use these instructions for training or evaluation. We collect training dataset of language- demonstration pairs for learning (Section 5). To extract subgoal sequences, we label each inter- action action
(type,, mask;) and any preceding navigation actions with a single subgoal of type type,- We compute the subgoal argument class arg" and 3D mask arg labels from the first-person mask mask,, and ground truth segmentation and depth. Completing a task requires sat- isfying several goal conditions. Following the common evaluation [58, 59], we report two metrics. Success rate (SR) is the fraction of tasks for which all goal conditions were satisfied. Goal condition rate (GC) is the fraction of goal-conditions satisfied across all tasks.
Systems We compare our approach, the Hierarchical to others on the ALFRED leaderboard that only use the high-level instructions. At the time of writing, the only such published approach is HITUT [47], an approach that uses flat BERT [54] architecture to model a hierarchical task structure without using a spatial representation. See Ap- pendix ?? for detailed comparison. We also compare to approaches that use the step-by-step instructions, which puts our method at a disadvantage. Of these, LAV [60] also imposes a hierarchi- cal task structure and uses pre- trained depth and segmentation models, but without using spatial state representation.
condiggeconddtooneed Spatial Model (HLSM)
Additionally, we perform ablations and study sensory oracles. To study the observation model, we compare to using sensory oracles for ground truth depth, ground truth segmentation, and both. We report high-level controller ablations that remove the subgoal encoder, language encoder, and state

---

Middle- Task: Secure discs the bedroom safe
12
308
PiCKuPOnJECT
OrCSOBJECT
PUTORIECT
CLoSEORItCT
PIOKUFORJECT
OPENDRIECT
OrtxOOn.cT
PuTOsteCT
CLUSEOBJECT
Correctty ideniitedd the sale. baled open due agent blocking the sate doo
2nd attempt succeeded from diterent agent pose
Perception fallure
Tas Hold the clock and turn on lamp Vaku
Non-fatal percaption
Image trput
Pradited Sngrestarte
Deyth
Function 120)
Image lppt
Samartic Vasel Noy (80)
186
Alam
Agent
Agent facing
butinterpees
Grounding failure
Task: Put a washed bowl away kitchen cabinet
Subgoal prediction failure
bosi on the counter
Tasik Put in
161
162
109
PENIFDRRCY (Bom)
PurOkacT (5AK)
TauAsOs (FAAU1)
X
PICHURORLEET (CD)
PInORUTT [SDA]]
insledd of bowl
mistale
Looks
Figure 4: Qualitative results showcasing successes and failures of our approach. Top row: snapshots of every interaction action taken during successful task. Action argument masks are overlaid in red over the RGB images. The white numbers timesteps. Middle-right: illustration of non-fatal perception left: illustration of fatal perception error. The agent incorrectly interprets the reflection on the alarm clock as an obstacle, causing agent (blue star) believe that path to goal (green star) is blocked reflected in the navigation value function computed by the value iteration network (VIN) [57], where black cells obstacles with value -1 White cell is the goal with value 1. Bottom-left: grounding failure. The agent wrongly picks up the cup instead of bowl. Predicted subgoals shown green. Bottom-right: high-level controller and percepton failure. π predicts the wrong subgoal argument class (CD instead of EGG). The segmentation model then mistakes the vase for a CD.
This
representation encoder as used for predicting subgoal type type, and argument class arg while still using the state representation & to predict the subgoal argument mask arg M. We also study low-level controller ablation that removes the exploration procedure.
7 Results
Table 1 shows test and validation results. Our approach achieves state-of-the-art performance across both seen and unseen environments in the setting with only high-level instructions. We achieve 10.04% absolute (98. 1% relative) improvement in SR on the test unseen split, and 11.53% absolute (62.6% relative) improvement in SR the test seen split compared HiTUT G-only.
Our approach performs competitively even when compared to approaches that also use the low-level step-by-step instructions. We achieve 4.84% absolute (31.4% relative) improvement in SR on the test unseen split compared to ABP [61]. On the test seen split, our approach performs reasonably well, however ABP [61] and LWIT [18] perform better, reflecting potentially stronger scene overfitting.
Tables and show development results. We performed five runs of the full HISM model on the validation unseen data and found the sample standard deviation of the success rate is 1.1% (absolute). All other results are from a single-evaluation runs. Ground truth depth alone (+ gt depth) does not significantly affect performance. Ground truth segmentation (+ gt seg) provides 6.6% absolute improvement in seen/unseen scenes. Using both (+ gt depth, gt seg) provides 11.1% absolute improvement and narrows the scen/unseen gap from 11.3% to 0.5%. This points to perception being the main bottleneck in generalization to unseen scenes.
We report high-level controller x input encoder ablations. The poor performance without the language encoder reflects task difficulty. Zeroing the input to the subgoal history encoder (but keeping position encodings) does not significantly affect performance, showing that knowing the index of the current subgoal addition to the state representation often sufficient. Not using the state representation for predicting subgoal type and argument class gives mixed results in

---

Test
Validation
Method
Seen
Unseen
Seen
Unseen
SR
GC
SR
GC
SR
GC
SR
GC
Low-level Sequential Instructions High level Goal Instruction
SEQ2SEQ [12]
3.98
9,42
0.39
7.03
3.70
10.00
0.00
6.90
MOCA [46]
22.05
28.29
5.30
14.28
19.15
28.5
3.78
13.4
E.T. [17]
28.77
36.47
5.04
15.01
33.78
42.48
3.17
13.12
E.T. synth. data [17]
38.42
45.44
8.57
18.6
46.59
52.82
7.32
20.87
LWIT [62]
30.92
45.44
9,42
20.91
33.70
43.10
9.70
23.10
HrTUT[47]
21.27
29.97
13.87
20.31
25.24
34.85
12.44
23.71
ABP [6P[61]
44.55
51.13
15.43
24.76
42.93
50.45
12.55
25.19
High-level Goal Instruction Only
HrTUT G-only[47]
18.41
25.27
10.23
20.27
13.63
21.11
11.12
17.89
LAV [60]
13.35
23.21
6,38
17.27
12.7
23.4
HLSM (Ours)
29.94
41.21
20.27
30.31
29.63
38.74
18.28
31.24
Table 1: Test results. Test seen/unseen and validation seen/unscen splits. Top section approaches use sequential step-by-step instructions. The bottom section uses only high-level instructions. Best results using only high- level instructions and using both types of instructions are highlighted.
Validation
Validation
Method
Seen
Unseen
Task Type
Seen
Unseen
SR
GC
SR
GC
SR
GC
SR
GC
HLSM
29.6
38.8
18.3
31.2
Overall
29.6
38.7
18.3
31.2
gt depth
29.6
40.5
20.1
33.7
gt depth, gt seg.
40.7
50.4
40.2
52.2
Examine
46.8
59.0
36.6
59.9
Pick & Place
57.0
57.0
34.8
34.8
gt seg.
36.2
47.0
34.7
47.8
Stack & Place
13.0
27.0
4.4
14.3
w/o language enc.
0.9
8.6
0.2
7.5
Clean Place
25.0
39.5
11.3
25.8
w/o subg. hist. enc.
29.4
38.5
16.6
29.2
Cool Place
39.6
w/o state repr enc.
30.0
40.6
30.8
Heat & Place
9.3
29.1
0.0
17.0
Pick & Place
34.7
51.9
18.0
34.7
w/o exploration
32.2
42.4
18.1
31.3
Table 2: Development results on validation split. Per- formance of our full approach, with perception oracles, perception ablation, " ablations, and n² ablations
Table 3: Performance breakdown per
task type on validation split.
and unseen scenes, but without a significant difference performance. Therefore, predicting the sequence of subgoal types and argument classes (i.e., what do) times possible without spatial reasoning, while grounding the subgoal (i.e.. where to do it) requires spatial information. Removing random exploration from 7 does not significantly affect unseen performance.
Figure 4 illustrates the model behavior, showing both successes and common failures. The main failures in valid unseen scenes are due to (1) perception errors that result in missing extraneous obstacles or picking up wrong objects; (2) insufficiency of random exploration (e.g., not searching inside cabinets); (3) navigation model enrors (e.g.. blocking objects from opening); (4) subgoal prediction errors (e.g., picking up wrong objects); and (5) lack of state-aware multi-step planning and backtracking. More qualitative results available in Appendix ??.
8 Discussion and Limitations
We showed that persistent spatial semantic representation enables a hierarchical model to achieve state-of-the-art performance on a challenging catructtionfollowinn, mobile manipulation task. The main performance bottlenecks include long-horizon exploration, perception generalization to un- seen environments, and low-level motion planning for continuous collision avoidance. In terms of learning, incorporating reinforcement learning train π nh, and observation model F jointly could improve robustness. We defined the interface to n". to be faithful to skills available on phys- ical robots, but the exact implementation of π is not the focus of our work. Physical deployment would require changes to x", and study on robustness to errors in continuous environments, such as localization motion uncertainty.

---

Acknowledgements
This research was supported by ARO W911NF-21-1-0106, Google Focused Award, and NSF under grant No. 1750499, Animesh Garg is supported in part by CIFAR Al Chair and NSERC Discovery Grant. A significant part of the work was done during the first author's s internship at Nvidia. We thank the authors of ALFRED for maintaining the benchmark. We thank Mohit Shridhar and Jesse Thomason for their help answering our questions, and the anonymous reviewers for their helpful comments.
References
[1] S. Tellex, T. Kollar, S. Dickerson, M. R. Walter, A. Gopal Banerjee, S. Teller, and N. Roy. "Ap- proaching the Symbol Grounding Problem with Probabilistic Graphical Models. Al Magazine, 2011.
[2] C. Matuszek, E. Herbst, L. Zettlemoyer, and D. Fox. Learning to parse natural language commands to a robot control system. In ISER, 2012.
[3] D. K. Misra, J. Sung, K. Lee, and A. Saxena. Tell me dave: Context-sensitive grounding of natural language to mobile manipulation instructions. In RSS, 2014.
[4] J. Thomason, S. Zhang, R. J. Mooney, and P. Stone. Learning interpret natural language commands through human-robot dialog. IJCAI, 2015.
[5] D. Misra, J. Langford, and Y. Artzi. Mapping instructions and visual observations to actions with reinforcement learning. In EMNLP, 2017.
[6] D. Nyga, S. Roy, R. Paul, D. Park, M. Pomarlan, M. Beetz, and N. Roy. Grounding robot plans from natural language instructions with incomplete world knowledge. In CoRL, 2018.
[7] V. Blukis, N. Brukhim, A. Bennet, R. Knepper, and Y. Artzi. Following high-level navigation instructions on a simulated quadcopter with imitation learning. In RS5, 2018.
[8] V. Blukis, D. Misra, R. A. Knepper, and Y. Artzi Mapping navigation instructions to continu- control actions with position-visiie vesition-visitationn prediction. In CoRL., 2018
[9] V. Blukis, Y. Terme, E. Niklasson, R. A. Knepper, and Y. Artzi Learning to map natural language instructions to physical quadcopter control using simulated flight. In CoRL, 2019.
[10] Patki, F. Fahnestock, T. M. Howard, and M. R. Walter. Language-guided semantic mapping and mobile manipulation in partially observable environments. In CoRI., 2019.
[11] V. Blukis, R. A. Knepper, and Y. Artzi. Few-shot object grounding and mapping for natural language robot instruction following. In CoRL, 2020.
[12] M. Shridhar, J. Thomason, D. Gordon, Y. Bisk, W. Han, R. Mottaghi, I.. Zettlemoyer, and
D. Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In CVPR, 2020.
[13] P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sünderhauf, I. Reid, S. Gould, and A. van den Hengel. Vision-and-language navigation: Interpreting visually-grounde naviga- tion instructions in real environments. In CVPR, 2018.
[14] H. Tan, L. Yu, and M. Bansal. Learning to navigate unseen environments: Back translation with environmental dropout. In NAACI-HLT, 2019.
[15] P. Anderson, A. Shrivastava, J. Truong, A. Majumdar, D. Parikh, D. Batra, and S. Lee. Sim- to-real transfer for sadion-and-langueee navigation. In CoRL, 2020.
[16] B. Kim, S. Bhambri, K. P. Singh, R. Mottaghi, and J. Choi. Abp, alfred leader- board, may 10th 2021. https://eeaderboare allenai. org/olllea.o hrg/alfred/ htt./suumissooo 22770337g445c1445888, Accessed: June 16th, 2021.

---

[17] A. Pashevich, C. Schmid, and C. Sun. Episodic Transformer for Vision-and Uision-add-Language Language Navi- gation, 2021.
[18] Anonymous. Lwit, alfred leaderboard, may 10th 2021. https://leaderboard. allenai. Accessed: May 25th, 2021.
[19] M. R. Walter, S. Hemachandra, B. Homberg, S. Tellex, and S. Teller. Learning Semantic Maps from Natural Language Descriptions. In RSS, 2013.
[20] S. Hemachandra, F. Duvallet, T. M. Howard, N. Roy, A. Stentz, and M. R. Walter. Learning models for following natural language directions in unknown environments. In ICRA, 2015.
[21] S. Patki, A. F. Daniele, M. R. Walter, and T. M. Howard. Inferring compact representations for efficient natural language understanding of robot instructions. In 2019 International Confer- ence on Robotics and Automation (ICRA). pages 6926-6933. IEEE, 2019.
[22] Kostavelis and A. Gasteratos. Semantic mapping for mobile robotics tasks: A survey. Robotics and Aulonomous Systems, 2015.
[23] H. Saha, F. Fotouhi, Q. Liu, and S. Sarkar. A modular vision language navigation and manipu- lation framework for long horizon compositional tasks in indoor environment. arXiv preprint arXiv:2101.07891, 2021.
[24] T. Kollar, S. Tellex, D. Roy, and N. Roy. Toward understanding natural language directions. In HRI, 2010.
[25] C. Matuszek, N. FitzGerald, I.. Zettlemoyer, L.. Bo, and D. Fox. Joint Model of Language and Perception for Grounded Attribute Learning. In JCML, 2012.
[26] Tellex, R. Knepper, A. Li, D. Rus, and N. Roy. Asking for help using inverse semanties. In RSS, 2014.
[27] R. A. Knepper, S. Tellex, A. Li, N. Roy, and D. Rus. Recovering from Failure by Asking for Help. Autonomous Robots, 2015.
[28] Z. Gong and Y. Zhang. Temporal spatial inverse semantics for robots communicating with humans. In ICRA, 2018.
[29] T. Brick and M. Scheutz. Incremental natural language processing for hri In HRI, 2007.
[30] S. Tellex, Thaker, R. Deits, D. Simeonov, T. Kollar, and N. Roy. Toward information theo- retic human-robot dialog. Robotics, 2013.
[31] K. M. Hermann, F. Hill, S. Green, F. Wang, R. Faulkner, H. Soyer, D. Szepesvari, W. Czar- necki, M. Jaderberg, Teplyashin, et al. Grounded language learning in simulated 3d world. arXiv preprint arXiv: 170..07666.6,110 2017.
[32] D. S. Chaplot, K. M. Sathyendra, R. K. Pasumarthi, D. Rajagopal, and R. Salakhutdinov. Gated-attention architectures for orientrie language grounding. AAAI, 2018.
[33] C. Paxton, Bisk, J. Thomason, A. Byravan, and D. Fox. Prospection: Interpretable plans from language by predicting the future. In ICRA, 2019.
[34] A. Suhr and Y. Artzi Situated mapping of sequential instructions actions with single-step reward observation. In ACL, 2018.
[35] D. Fried, R. Hu, V. Cirik, A. Rohrbach, J. Andreas, L.-P. Morency, T. Berg-Kirkpatricks K. Saenko, D. Klein, and T. Darrell. Speaker-follower models for vision-and-language navi- gation. In NewrIPS, 2018.
[36] D. Misra, A. Bennett, V. Blukis, E. Niklasson, M. Shatkin, and Y. Artzi. Mapping instructions actions in 3D environments with visual goal prediction. EMNLP, 2018.
[37] C.-Y. Ma, J. Lu, Z. WI, G. AlRegib, Kira, Socher, and C. Xiong. Self-monitoring navigation agent via auxiliary progress estimation. In ICLR, 2019.
10

---

[38] H. Chen, A. Suhr, D. Misra, and Y. Artzi. Touchdown: Natural language navigation and spatial reasoning in visual street environments. In CVPR, 2019.
[39] D. K. Misra, K. Tao, Liang, and A. Saxena. Environment-driven lexicon induction for high- level instructions. In ACL, 2015.
[40] M. MacMahon, B. Stankiewicz, and B. Kuipers. Walk the talk: Connecting language, knowl- edge, and action in route instructions. In AAAI, 2006.
[41] S. R. K. Branavan, S. Zettlemoyer, and R. Barzilay. Reading between the lines: Learning to map high-level instructions to commands. In ACL, 2010.
[42] S. Tellex, Т. Kollar, S. Dickerson, M. Walter, A. Banerjee. S. Teller, and N. Roy. Understanding natural language commands for robotic navigation and mobile manipulation. AAAI, 2011.
[43] Duvallet, T. Kollar, and A. Stentz. Imitation learning for natural language direction following through unknown environments. In ICRA, 2013.
[44] Y. Artzi and L. Zettlemoyer. Weakly supervised learning of semantic parsers for mapping instructions actions. TACL, 2013.
[45] E. C. Williams, N. Gopalan, M. Rhee, and S. Tellex. Learning to parse natural language to grounded reward functions with weak supervision. In ICRA, 2018.
[46] K. P. Singh, S. Bhambri, B. Kim, R. Mottaghi, and J. Choi. Moca: A modular object-centric approach for interactive instruction following, arXiv preprint arXlw:2012.03208, 2020.
[47] Y. Zhang and J. Chai. Hierarchical task learning from language instructions with unified trans- formers and self-monitoring. ACL Findings, 2021.
[48] D. Gordon, A. Kembhavi, M. Rastegari, J. Redmon, D. Fox, and A. Farhadi. Iqa: Visual question answering in interactive environments. In CVPR, 2018.
[49] P. Anderson, A. Shrivastava, D. Parikh, D. Batra, and S. Lee. Chasing ghosts: Instruction following as bayesian state tracking. In NeurlPS, 2019.
[50] S. Hemachandra, M. R. Walter, S. Tellex, and S. Teller. Learning spatial- representa- tions from natural language descriptions and scene classifications. In ICRA, 2014.
[51] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.
[52] J. McCormac, A. Handa, S. Leutenegger, and A. J. Davison. Scenenet rgb-d: Can 5m synthetic images beat generic imagenet pre-training on indoor segmentation? In ICCV, 2017.
[53] Y. Wu, A. Kirillov, F. Massa, W.-Y. Y. Lo, and R. Girshick. Detectron2. https: htt///gttthh/..//// com/ facbbolrreearccc/eeeetttnn 2019.
[54] Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
[55] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, Jones, A. Gomez, L. Kaiser, and I. Polo- sukhin. Attention is all you need. In NeurIPS, 2017.
[56] G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In CVPR, 2017.
[57] A. Tamar, Y. Wu, G. Thomas, S. Levine, and P. Abbeel. Value iteration networks. In NeurIPS, 2016.
[58] M. Shridhar and Hsu. Interactive visual grounding of referring expressions for human-robot interaction. In RS5, 2018.
[59] M. Shridhar, J. Thomason, D. Gordon, Y. Bisk, W. Han, R. Mottaghi, L. Zettlemoyer, and D. Fox. Alfred leaderboard. https://leaderboars www.arrdddllllaaaa.eeeeae allenai wwl.maa..rg/alfreee....... org/alfred/. Accessed: June 16th, 2021.
11

---

[60] K. Nottingham, L. Liang, D. Shin, C. C. Fowlkes, R. Fox, and S. Singh. Modular framework for visuomotor language grounding. arXiv preprint arXiv:2109.02161/ 2021.
[61] B. Kim, S. Bhambri, K. Singh, R. Mottaghi, and J. Choi. Agent with the big picture: Per- ceiving surroundings for interactive instruction following. In Embodied Al Workshop CVPR, 2021.
[62] V.Q. Nguyen, M. Suganuma, and T. Okatani. Look wide and interpret twice: Improving performance on interactive instruction-follwwiii tasks. In IJCAI, 2021.
[63] H.-T. L. Chiang, A. Faust, M. Fiser, and A. Francis. Learning navigation behaviors end-to-end with autorl. RA-L, 2019.
[64] M. Sundermeyer, A. Mousavian, R. Triebel, and D. Fox. -ratact-graspne:: Efficient 6-dof grasp generation in cluttered scenes. arXiv preprint arXiv:2103. 14127, 2021.
[65] D. Meagher. Geometric modeling using octree encoding. Computer graphics and image pro- cessing, 1982.
[66] T. Takikawa, J. Litalien, K. Yin, K. Kreis, C. Loop, D. Nowrouzezahrai, A. Jacobson, M. McGuire, and S. Fidler. Neural geometric level of detail: Real-time rendering with im- plicit 3d shapes. In CVPR, 2021.
[67] H. Grice. Logic and conversation. In Speech acts 1975.
[68] Roh, Paxton, A. Pronobis, Farhadi. and Fox. Conditional driving from natural language instructions. In CoRI., pages 540-551, 2020.
[69] C. Matuszek, D. Fox, and K. Koscher. Following directions using statistical machine transla- tion. HRI, 2010.
[70] N. Gopalan, D. Arumugam, L. L. Wong, and S. Tellex. Sequence-to-sequencc language grounding of mon-markovian task specifications. In RSS, 2018.
[71] D. Bahdanau, F. Hill, J. Leike, E. Hughes, A. Hosseini. Kohli, and E. Grefenstette. Learning to understand goal specifications by modelling reward. In ICLR, 2018.
[72] P. Goyal, S. Nickum, and R. J. Mooney. Pix12r: Guiding reinforcement learning using natural language by mapping pixels to rewards. CoRL., 2020.
[73] Krantz, Wijmans, A. Majumdar, D. Batra, and Lee. Beyond the nav-graph: Vision- and-language navigation in continuous environments. arXiv preprint arXiv:2004.02857 2020.
[74] Jain, G. Magalhaes, A. Ku, A. Vaswani, E. Ie, and J. Baldridge. Stay the path: Instruction fidelity vision-and- lan--uuugg navigation. In ACL, 2019.
[75] A. Ku, P. Anderson, R. Patel, E. Ie, and J. Baldridge. Room-across-roomi Multilingual vision- and-language navigation with dense spatiotemporal grounding. In EMNLP, 2020.
[76] M. Persson, T. Duckett, C. Valgren, and A. Lilienthal. Probabilistic semantic mapping with a virtual sensor for building/nature detection. In Computational Intelligence in Robotics and Automation, 2007.
[77] H. Zender, M. Mozos, P. Jensfelt, G.-J. Kruijff, and W. Burgard. Conceptual spatial repre- sentations for indoor mobile robots. Robotics and Autonomous Systems, 2008
[78] A. Pronobis, O. Martinez Mozos, B. Caputo, and P. Jensfelt. Multi-modal semantic place classification. IJRR, 2010.
[79] A. Pronobis. Semantic mapping with mobile robots. PhD thesis, KTH Royal Institute of Technology, 2011.
12


Reviews:
[{"review_id": "URo0VNCAlAh", "reviewer": "Reviewer_h53W", "summary": "This paper presents a spatial semantic representation method that enables an agent to reason about and plan behaviors to complete long-term tasks. The developed method called hierarchical language-conditioned spatial model (HLSM) connects vision, language, and control that are three important topics toward developing general AI. The input include high-level language instructions and raw images, and the output are low-level mobile manipulation actions. The developed approach was evaluated using the recent ALFRED benchmark. HLSM was compared to a recently published approach called HiTUT, and methods that process detailed language-based instructions. Results show improvements in success rate. ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The reviewer didn't observe any major issues from the paper. Technical details were well presented, and it's quite easy to follow the paper. The baseline methods were carefully selected, and the results are convincing. There are the following suggestions that can potentially further improve the work. \n\nThe evaluation and implementation highly depend on ALFRED. Although it's a well developed benchmark, all benchmarks have limitations and ALFRED is no exception, which was not discussed. It's necessary to discuss how the developed methods can be applicable to real-world scenarios. There are assumptions such as reliable action executions that won't be valid for real robots. \n\nIt's unclear what makes the learned spatial representation \"persistent\". The term is highlighted in the title, and throughout the paper, but was not explained. \n\nAll components were learned using supervised learning methods. The datasets were separately collected, and the neural networks were learned separately. This type of \"piece by piece\" learning requires a lot of fine-tuning work. Also, it's unclear which part is the bottleneck of the current system. Some more discussions will be appreciated. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The reviewer didn't observe any major issues from the paper. It's a nicely written paper of solid research. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "See above. ", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}, {"review_id": "IFrNHPRoXV2", "reviewer": "Reviewer_s6Fc", "summary": "This work presents an approach to instruction following in which an agent is provided with a high-level natural language instruction of a task and expected to accomplish that task in a previously-unseen environment, discovering and interacting with task-relevant objects as it goes. The addition of a novel \"persistent spatial model\" (in the form of a semantic voxel grid) helps the agent maintain knowledge of the environment over longer timescales without the need for difficult-to-train recurrent architectures. The voxel grid is built using pretrained segmentation and depth perception models (each of which takes an RGB input to estimate its target output) and the map is fused with language to predict a subgoal---a high-level prediction for the next component of the task---that is then used to guide low-level action selection. Using this approach, the authors achieve state-of-the-art performance, achieving the highest performance to date on the ALFRED dataset for agents not provided with step-by-step instructions at test time. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper presents a useful and novel approach to improving performance in this domain. Tables 2 and 3 are particularly helpful for both understanding the results and to illustrate where improvements might be made into the future. Though the results are (for the most part) thorough, perhaps the biggest change that I recommend is that it would be particularly helpful if the authors described the HiTUT G-only approach in more detail, since it is the primary approach against which the proposed method is compared. The inclusion of such a description would help to better contextualize the proposed approach. The authors could revisit the motivation for a spatial model, which the HiTUT approach does not have and further reinforce the importance of having a persistent spatial representation.\n\nIn the abstract (and perhaps elsewhere), it should be made clearer that though the agent is \"avoiding the commonly used step-by-step instructions\" at test time, plan subgoals are indeed provided at training time. As there exists other work in which embodied agents are expected to learn their own subgoals, this point should be more clearly communicated. This should not weaken the significance of the work.\n\n[Sec. 4] It would be helpful if the authors could comment on the significance of using \"sampling\" during inference (perhaps, as opposed to iteratively selecting the next-most-likely subgoal). Is the performance also stochastic, since sampling is randomized, and by how much does performance change for the various tasks if the random seed is changed. I do not think more experiments are necessary here, but some clarification regarding why this process was selected (as opposed to a sequential \"maximum likelihood\" procedure) and the impact of this decision would help clarity.\n\nAdditional Comments and Suggestions:\n- Suggestion: the notation \"circle with a dot in it\" [or \\odot in LaTeX] (see footnote 5) is typically reserved for elementwise multiplication. Perhaps using \"circle with a \\times in it\" [or \\otimes in LaTeX] is a more appropriate choice, since it often represents the tensor product (similar to the operation defined in footnote 5).\n- I believe there is a typo on line 79 (section 3), since T: S x A -> S", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, the paper is quite thorough. In particular, the detailed studies conducted to understand the importance of each part of the pipeline were a welcome addition, and contributed greatly to the understanding of where this work and other related works in this space may need to improve if the community is to make non-trivial progress. This work represents progress in a challenging application domain that is likely to be used in future work in this area. There are, however, a few places in which the authors should be more clear and help to provide additional context.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "Generally, the paper seems mostly complete, I have made a few suggestions in my comments in \"Strengths and Weaknesses\" above where I think the authors can provide some more context. In summary:\n\n1. The authors should describe in more detail the HiTUT approach that represents their primary point of comparison.\n2. The paper should be clearer that, even though step-by-step instructions are not used at test time, they are required to train the system.\n3. The rationale behind the \"sampling\" procedure used during inference should be expanded upon."}}, {"review_id": "0xJfID_k0R", "reviewer": "Reviewer_Xmxc", "summary": "The paper proposes augmenting hierarchical mobile manipulation policies with a persistent semantically-labeled voxel map (spatial semantic representations) learned from natural language instructions. It seeks to show that conditioning a hierarchical policy on these persistent representations allows agents to succeed at high-level mobile manipulation tasks.\n\nProposed contributions:\n* \"First\" demonstration of a modular representation learning approach for the problem of mapping high-level NL task descriptions to actions in a 3D environment\n* Method f or utilizing spatial semantic representations within a hierarchical model for solving MM tasks\n* SOTA performance on ALFRED with only high-level instructions (rather than step-by-step instructions)", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n---\n* Clear, unambiguous, and generally high-quality presentation\n* Results and figures included are helpful for understanding the method and thorough\n* Method improves on SOTA by eliminating the need for step-by-step instructions to achieve current success rates\n\nWeaknesses\n---\n* The proposed architecture is exceedingly complex, and seems highly-customized to a specific benchmark (ALFRED). I would like to see an application to at least one other benchmark to be assured that this system is generally-applicable. The complexity also make it unlikely future authors will be able to successfully build on the method.\n* Results seem to support the assertion that HLSM allows this system to meet the performance of other methods which are provided with step-by-step instructions rather than high-level instructions (a triumph in and of itself!) but not necessarily improving SOTA on ALFRED\n* The paper spends so much time on very detailed descriptions of the method that it leaves little space for building intuition behind the method, analysis, or discussion --- i.e. it is a description of a method, plus results, plus a declaration of beating SOTA. The reader will not learn much about the problem or future possibilities by reading this paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "My recommendation for a weak reject is motivated by a couple of my comments above, specifically:\n\n(a) I'm concerned this work as-presented may be overfitted to the ALFRED benchmark, leaving me with few reasons to believe it might work if applied to real robots and\n\n(b) while a thorough and convincing description of a method and results, I don't think the reader will learn much about the core research problem, future possibilities for new research, or build an understanding as to why this method succeeds or fails.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "* Spending more time on building intuition, teaching the reader about the problem, and telling us what insights about the problem we can take from the method\n* Better-explaining why the method works, and why it fails\n* Adding (even a small) evaluation on some challenge other than ALFRED\n\nDetailed Comments\n---\nFigure 2: This figure is very dense and takes a lot of work to parse. I suggest splitting it into two or more figures for clarity, each representing a single idea.\n\nLine 239: \"...even though this evaluation is not fair to our method\" -- Fairness is very much in the eye of the beholder and really depends on what the reader is looking for in a method. I suggest instead noting that evaluating on step-by-step instructions is *different* and won't necessarily demonstrate all of your method's potential. Similarly for Line 251.\n\nLine 252: Whether or not the authors of ABP have published the details of their method is irrelevant to your performance comparison. Similarly with LAV. Notes like this (and greying them in the table) come off as unprofessional and a bit petty.\n\nTable 1: What is the purpose of bolding the line for HLSM? Bolding is sometimes used to call out the highest numbers in a column in ML/CV/NLP papers, but that doesn't seem to be the case here and there is no explanation in the caption. Please consider removing this. Similarly with Tables 2 and 3. \n\n"}}, {"review_id": "-PUe5qRCOq6", "reviewer": "Reviewer_xkZt", "summary": "The paper considers the problem of executing high-level commands provided via natural language, which typically requires reasoning over extended time horizons. Modeled in similar spirit to a partially observable MDP, there are two aspects to the proposed framework that are fundamental to addressing the problem. One is a 3D spatial-semantic model of the environment that is maintained over time based upon visual (monocular) observations, which the paper argues as providing a persistent representation of the environment. The second is a hierarchical representation of the action space, whereby language is mapped to (a distribution over) abstract subgoals based on the state representation along with the subgoal history. A low-level policy then maps a particular subgoal to an action based on the current state. The method is evaluated on the ALFRED benchmark and shown to outperform a recent method (HiTUT) that also reasons over high-level instructions. An ablation study demonstrates the contributions of the different architecture components on seen and unseen environments.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "STRENGTHS\n\n+ The problem of executing natural language instructions that provide only a high-level description of the task is challenging and, as the authors note, has received less attention recently when compared to methods that reason over the more detailed, lower-level (step-by-step) instructions.\n+ The method is evaluated on a standard, albeit simulated, benchmark and. shown to outperform a recent high-level baseline.\n\nWEAKNESSES\n\n- While the body of work that is cited is reasonable, the discussion of how this work is situated with regards to the large body of work in language understanding for robotics is inadequate. At only two paragraphs, the related work section provides only a cursory description of a small fraction of work in language understanding, in many cases, without a discussion of how the proposed approach differs. This makes it difficult to assess the significance of the contributions. This is not the first paper to consider spatial-semantic environment models in the context of language understanding nor is it the first to consider mapping language to high-level abstractions (symbols) of the environment and the robot's actions. That said, I think that the work is interesting and potentially novel---the paper just needs to do a better job of making this evident.\n- The paper initially presents a formulation of the problem as a stochastic process, but it is not always clear how the proposed framework is consistent with formulation (e.g., it isn't clear whether or how the prediction and update step are principled, or what the prediction step is, for that matter).\n- Related, the technical details of the algorithm can be difficult to parse at times, in part because various details are omitted. As presented, some of aspects of the algorithm come across as ad hoc.\n- It isn't clear how well the method will scale to larger environments\n- It is unclear whether the approach can be expected to generalize to real-world settings (e.g., including the ability to handle uncertainty in perception and motion).\n\n\nMINOR\n\n* Line 66: \"... language directly actions ...\" --> \"... language directly to actions ...\"\n* Figure 2: Where is s_t? Also, it would help to use the same notation as in the text.\n* Line 156: Erroneous comma before footnote.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper considers an important problem, namely the ability to interpret and carry out high-level instructions. The results indicate that the method outperforms a recent baseline, however the qualitative comparison to related work in language understanding for robotics is lacking, which makes it difficult to judge the contributions relative to existing work. With an evaluation that is limited to simulation, it is also difficult to predict how the approach would generalize to the real world, where the uncertainty is presumably far greater than in the simulator.\n\n\nUPDATE BASED ON AUTHOR RESPONSE\n\nI appreciate the authors' detailed response, which clarified several of the questions that I had. I remain concerned that the related work discussion doesn't adequately identify the paper's contributions relative to existing work. Several citations have been added to the related work section, but they are only given a cursory description in the primary paper. While the appendix has been updated to include more details, the paper's contributions need to be clearly presented as part of the main text.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {"issues": "* Lines 64–65: The statement that \"laborious to scale to new environments and complex language\" is questionable, particularly given the success of symbol grounding-based approaches to language understanding in robotics. It is the symbolic representation of the state and actions that enable generalization to new environments and the exploitation of the hierarchy of language (e.g., [1 in the paper]) that supports the interpretation of complex language.\n* How does the spatial-semantic representation of the environment proposed here compare to representations used by others in the context of language understanding [37(paper), 2]\n* How are the state dynamics modeled? Are they assumed to be known or are they learned?\n* Lines 133–134: How exactly is the perception step modeled to support training from a limited amount of data?\n* Lines 141–142: What are the segmentation distributions, exactly?\n* The approach relies upon a dead-reckoned estimate of the robot's pose, which will quickly drift in practice. How does this affect the projection of the environment representation? Regardless of projection, how does this influence the environment model, which, as far as I can tell, does not account for the pose uncertainty.\n* Can the authors elaborate on the reduced performance on unseen environments when including the state encoding? Was the model with or without state encodings used for the test set (i.e., did this function as a validation set in terms of choosing the architecture)?\n\n\nREFERENCES\n\n[1] Siddharth Patki, Ethan Fahnestock, Thomas M. Howard, and Matthew R. Walter. Language-guided Semantic Mapping and Mobile Manipulation in Partially Observable Environments, In Proceedings of the Conference on Robot Learning (CoRL), 2019."}}]

Author rebuttal:

Meta-review: This approach to an important problem appears to have achieved a substantial advance over SOTA, though there is some concern about overfitting to the ALFRED benchmark, which it would be helpful for the authors to discuss.

 — post-rebuttal, I believe many of the questions surrounding the contribution and method have been resolved, and the reviewers are largely positive about the paper. One reviewer is still a weak reject, but did not respond to the authors’ rebuttal.
Decision: Accept (Poster)