Embodied Concept Learner: Self-supervised Learning of Concepts and Mapping through Instruction Following
Authors: Mingyu Ding
Yan Xu
Zhenfang Chen
David Daniel Cox
Ping Luo
Joshua B. Tenenbaum
Chuang Gan
Venue: CoRL 2022 Poster (2022)
Keywords: Embodied AI
Embodied Concept Learner
Instruction Following

Abstract: Humans, even at a very early age, can learn visual concepts and understand geometry and layout through active interaction with the environment, and generalize their compositions to complete tasks described by natural languages in novel scenes. To mimic such capability, we propose Embodied Concept Learner (ECL) in an interactive 3D environment. Specifically, a robot agent can ground visual concepts, build semantic maps and plan actions to complete tasks by learning purely from human demonstrations and language instructions, without access to ground-truth semantic and depth supervision from simulations. ECL consists of: (i) an instruction parser that translates the natural languages into executable programs; (ii) an embodied concept learner that grounds visual concepts based on language descriptions; (iii) a map constructor that estimates depth and constructs semantic maps by leveraging the learned concepts; and (iv) a program executor with deterministic policies to execute each program. ECL has several appealing benefits thanks to its modularized design. Firstly, it enables the robotic agent to learn semantics and depth unsupervisedly acting like babies, e.g., ground concepts through active interaction and perceive depth by disparities when moving forward. Secondly, ECL is fully transparent and step-by-step interpretable in long-term planning. Thirdly, ECL could be beneficial for the embodied instruction following (EIF), outperforming previous works on the ALFRED benchmark when the semantic label is not provided. Also, the learned concept can be reused for other downstream tasks, such as reasoning of object states.

Embodied Concept Learner:
Self-supervised Learning of Concepts and Mapping through Instruction Following
Mingyu Ding HKU
Yan Xu CUHK
Zhenfang Chen MIT-IBM Watson AI Lab
David Cox MIT-IBM Watson AI Lab
Ping Luo HKU
Joshua B. Tenenbaum MIT BCS, CBMM, CSAIL
Chuang Gan
UMass Amherst MIT-IBM Watson Al Lab
Humans, even at a very early age, can learn visual concepts and under- stand geometry and layout through active interaction with the environment, and generalize their compositions to complete tasks described by natural languages in novel scenes. To mimic such capability, we propose Embodied Concept Learner (ECL) in an interactive 3D environment. Specifically, a robot agent can ground visual concepts, build semantic maps and plan actions to complete tasks by learn- ing from human demonstrations and language instructions. ECL consists of: (i) an instruction parser that translates the natural languages into executable programs; (ii) an embodied concept learner that grounds visual concepts based on language descriptions/embedddins and pretrained object proposal network; (iii) map constructor that estimates depth and constructs semantic maps by leveraging the Iearned concepts; and (iv) program executor with deterministic policies to exe- cute each program. ECL has several appealing benefits thanks to its modularized design. Firstly, it enables the robotic agent to learn semantics and depth unsu- pervisedly acting like babies, e-g., ground concepts through active interaction and perceive depth by disparities when moving forward. Secondly, ECL is fully trans- parent and steeby interpretable in long-term planning. Thirdly, ECL could be beneficial for the embodied instruction following (EIF), outperforming previ- ous works on the ALFRED benchmark when the semantic label is not provided. Also, the learned concept can be reused for other downstream tasks, such as rea- soning of object states.
Abstract:
Keywords: Embodied AI, Embodied Concept Learner, Instruction Following
Introduction
Embodied instruction following (EIF) [1] is a popular task in robot learning. Given some multi- modal demonstrations (natural language and egocentric vision, as shown in Fig. 1) in 3D environ- ment, a robot is required to complete novel compositional instructions in unseen scenes. The task challenging because it requires accurate 3D scene understanding and semantic mapping, visual navigation, and object interaction.
Recent works for EIF can be typically divided into two streams and they have certain limitations. 1) End-to-end imitation learning methods [1, 2, 3, 4] directly input the visual observation of the current step and language instructions into the model, and output the action for the next step. For exam- plc, Pashevich et al. [4] has presented the episodic transformer to predict the agent's actions with
Project page: http://ecl.cor http:://ccl.ccaall..m wwa.l.mit.e.......u..o
6th Conference Robot Learning (CoRL. 2022), Auckland, New Zealand.

---

an attention mechanism and a progress monitor. Such models work by simply memorizing train- ing scenes and trajectories. While they achieve good performance in seen environments, they fail to generalize well in unseen scenes. Furthermore, these black-box models often lack transparency, interpretability. and generalizability. 2) Mapping-based methods [5, 6] leverage the map representa- tions [7. 8, 9, 10] by building a 3D voxel map from the predicted depths and instance segmentation masks. A semantic top-down map of the scene is then constructed and updated at each step. These works perform explicit exploration and interactions through semantic search policies [6] achieve the natural language goal, which is transparent and interpretable. However, they assume that the agent has learned the depth and semantics passively from large amounts of data. The semanttc la- bels and depth supervision are often labor-intensive and hard to obtain in the real world. We argue that such supervision signals are unnecessary since we can learn language concepts and visual dis- parity through interactions in the environments. For example, by achieving the goal described in Fig. I, humans can learn what the concepts "knife" and "table" are and perceive that the table in frame is physically closer to the agent than frame 1.
This paper answers a question naturally raised from the above issues: can we make the agent behave like baby? A baby able to learn domain knowl- edge from environmental inter- actions and expert demonstra- tions without additional supervi- sion to achieve the natural lan- guage goal.
Goal: "place knife the microwave table'.
York Dowr to
gysh the kmill the back the kitchen sinee"
Telt OVNT To microwave"
plaae the linile on the Iufi side table"
kücceen sink
speculate that babies do this possibly in a way similar (i) Learn skills and concepts from expert demon- strations (environment observa- tions and language instructions), e.g., the skill "place" and con- cepts "knife" and "table" can grounded from the demonstra- tion "place knife the mierowave like "put clean tomato the dining table" in Fig. 2, one may process it into many subgoals, like "pickup tomato", "clean tomato",, and "put it on the table". (iii) Explore the scene and build a semantic map, where depth information is estimated automatically based on the disparity when moving forward or backward. (iv) Complete each subgoal based on the learned semantic map and skills, and update the semantic map dynamically.
Exploration
Interaction
Visual Navigation
Interaction
Figure 1: An example of language goal and its corresponding four subgoals. The top and bottom rows show visual observations by the robotic agent and our grounded semantics, respectively. We show that we align object concepts encoded subgoals with visual proposals to learn concepts in the embodied environment.
table". (ii) Given new compositional language goal
Motivated by the above observations, propose Embodied Concept Learner (ECL) that mimics baby learning for embodied instruction following. It consists of: (i) an instruction parser that parses the natural languages into executable programs; (ii) an embodied concept learner that aligns lan- guage concepts with visual instances (a pretrained object proposal network and a word embedding model are used); (iii) a map constructor based on the grounded semantic concepts and unsupervised depth estimation; and (iv) a program executor with deterministic policies to perform each subtask.
Our contributions three-fold. 1) We introduce ECL, a modular framework that ground vi- sual concepts, build semantic maps and plan actions to complete complex tasks by learning purely from human demonstrations and language instructions. 2) ECL achieves competitive performance without semantic labels on embodied instruction following (ALFRED) [1], while maintaining high transparency and step-by-step interpretability. 3) We could also transfer the learned concepts to other tasks the embodied environment, like the reasoning of object states.

---

2 Related Work
Embodied Instruction Following, Language-guided embodied tasks have drawn much attention, including visual language navigation (VLN) [11, 12, 13, 14, 15, 16, 17], embodied instruction fol- lowing (EIF) [18, 19, 20, 21, 22, I]. object goal navigation [23, 24, 25], embodied question answer- ing [26, 27]. program sketch generation [28, 29], and embodied representation learning [30, 31, 32]. Among them, EIF is one of the most challenging tasks, requiring simultaneous accurate 3D scene understanding and memory, visual navigation, and object interaction. [1, 4] present end-to-end mod- els with an attention mechanism to process language and visual input and past trajectories, predicting the subsequent action directly. After that, works [20, 22, 19] modularly process raw language and visual inputs into structured forms by object detectors [33, 34, 35]. The above methods lack trans- parency and generalizability to unseen scenes. Recently, [5, 6] proposed mapping-
methods to convert visual semanties and estimated depth into sirdds-eye-view (BEV) semantic maps and navi- gate based the spatial memory. However, such methods require depth and semantic supervisions, hence impractical in real-world scenarios.
Visual Grounding and Concept Learning. Our work is also related to visual grounding [36, 37, 38, 39, 40, 41, 42, 43] and concept learning [44, 45, 46, 47, 48, 49], which align concepts onto objects in the visual scenes. Traditional visual grounding methods [39, 37] map text phrases and regional features of images into common space for cross-modality matching. Recently, there some works [44, 45, 50] learning visual concepts through question answering in passive images or videos. Differently, we study learning both visual concepts and physical depths through language instructions in the active embodied environment, which is more similar to how humans learn in the real world. Some works study language grounding in 3D world [51, 52, 53]. However, they do not involve robot agents and active exploration. Hermann al. [48] interprets language in a sim- ple simulated 3D environment, which does not consider diverse objects and actions in challenging photorealistic environments.
3 Method
In this work, we focus on the embodied instruction following task, i.e., robotic agent is required to achieve the goal in the language instruction by exploring. navigating, and interacting with the em- bodied environment. Embodied Concept Learner (ECL) includes an instruction parser, an embodied concept learner, a map constructor, and a program executor. The modularized design ensures its transparency and step-by-step interpretability. An overview of ECL is shown in Fig.
3.1 Instruction Parser
The instruction parser converts high-level instructions into a sequence of subtasks represented by programs. Existing works [6, 5, 20, 22, 4, 28, 29] use expert trajectories with subtasks annotations as supervision because they are easy to obtain as stated in [6]. Following this strategy, we fine-tune a pre-trained BERT model [54] learned the mapping from a high-level instruction to a sequence of subtasks (e.g., "put clean tomato on the diningtable" -> "(Pickup, Tomato), (Put, SinkBasin))... leveraging the subtasks sequences annotations in ALFRED [1]. For each subtask, the instruction parser predicts the arguments, which are the same as in [6]: (i) "obj" for the object be picked up, (ii) "recep" for the receptacle where "obj" should be ultimately placed, (iii) "sliced" for whether "obj" should be sliced, and (iv) "parent" for tasks with intermediate movable receptacles (e.g., "cup" in "Put a knife in a cup on the table"). After we get the subtask programs, we extract the language embeddings E IR 768 of the object words in all subprograms through a pretrained Bert model baseutbac--uccsed) [55] for the follow-up concept learner module.
3.2 Embodied Concept Learner
Humans, at very early age, naturally perceive and parse the scene objects for further understanding, i.e., grouping pixels to regions without knowing their semanties [56, 57]. They then

---

Put a clean Tonato DiningTable.
Program Parsing
PickUp
|-|
Put
ToggleOn
|-
ToggleOff
PickUp
Put
Tonato
Tonato, SinkBasin
Faucet
Faucet
Torato
Torato,
Table
I. Instruction Parsor
Object Proposuls
Object Features
Word Embeddings
Proposal
Obj
Tonato
Extraction
Obj
Concept
SinkBasin
Grounding
Faucet
Obj
DiningTable
Bayesian Filtering
Embodied Concept Learner
Depth (unsup.)
BEV Map
Depth
Parsing
3D Projection
Deterministic Policy
& Mapping
Action: ToggleOn Faucet
t-2
III. Map Constructor and Program Executor
Figure 2: The framework of ECL. (i) Given a natural language goal, the instruction parser first parses it into a sequence of executable programs. (ii) The embodied concept learner extracts regional proposals in current frame and align them with the learned concepts. (iii) The map constructor then builds up semantic maps based on estimated depths and grounded visual concepts. semantic maps and executable programs, the program executor predicts the agent's 's next action with a deterministic policy.
(iv) Having the
learn the object concepts from active interactions expert demonstrations. Similarly, the embodied concept learner leverages object proposal network [33] without category labels and grounds the object semantics from subgoal programs. There are two cases to be considered: 1) If subgoal completes, the object and its corresponding receptacle objects must be displayed the current visual frame, and most likely in adjacent frames. this way, the concept of these objects can be grounded. For example, "go to microwave". "put the mug on the coffeemachine" and "put a mug with a pen in on the shelf" involve and 3 objects, respectively. We sample visual data from four frames before completing the subtask and two frames after learn the visual concept based on the corresponding action descriptions. 2) If the robot agent acts "Pickup an object", the object appears in visual observation until the robot drops it. The two types of interaction data are merged and shuffled and used as input to our embodied concept learner.
Concretely, let f0r, ..., 0 denotes k objects detected in an visual input, and {f1 f2. f.....fil fk] is their corresponding feature representations from the last layer of the object proposal network R R1024 ). Let {ei, C2, e1) represents I word embeddings in subgoal (program representation, E 768 stated in Sec. 3.1). We first project the visual representation f into the semantic space f' 768 where the word embeddings reside by a two-layer perceptron (MLPs). The MLPs have dimensions of 1024 - 1024 + 768 with Layer Normalization [58] and GELU activation [59] between the two layers. We then leverage the Hungarian maximum matching algorithm [60] for h-l matching, and min(k, object visual representations can be matched with their word embeddings. Given an assignment matrix æ gkx! the task could be formulated as a minimum cost assignment problem mathematically as follows:
d(L, s.t. Σ sy 1
Σ my
(0,1), (0, E (0,
where d(-) denotes the mean square error (MSE) and we assume < & here, vice versa. In this way, we solve min-min optimization problem by Hungarian matching, where the first minimization is used to find the best match among the two sets of features (Hungarian matching): and the second minimization is optimize a smaller L2 loss on the matching for learning better projected represen- tation f'. Both the mapping function (MLPs) and the matching learned at the same time. Thus the MLP and the matching matrix is jointly learned from Hungarian matching.

---

During inference, we project each object proposal representation into the semantic space and per- form nearest neighbor search (NNS) to assign a category label for it. We also calculate a soft class probability p; for the i-th object by softmax [[0.1/d.)] 15), where dis is the retrieval distance be- tween the i-th object feature and the j-th word embedding. The semantic probability p will be used for 1) Bayesian filtering in mapping and 2) statisties of the most likely location of each type of object navigation policy.
3.3 Map Constructor
Human beings understand the semantics and layouts of space, e.g., room, mainly by first moving around, then perceiving the depth (geometry), and finally building up a semantic virtual map in our mind [61]. To mimic this process, we propose a semantic map construction module leveraging the unsupervised depth learning technique [62. 63] and probabilistic mapping inspired by Bayesian filtering. Coneretely, we first train monocular depth estimation network unsupervisedly, leveraging the photometric consistency [62] among adjacent RGB observations captured by roaming agent. We use the unsupervised depth estimation for map construction. To build up the map, we represent the scene as voxels. Each voxel maintains semantic probability vector (obtained from Sec. 3.2) and a scalar variable oy that represents the measurement uncertainty of this voxel. As the new depth observation come in, we first project it to 3D space as a 3D point cloud and then transform into the map space according to the agent ego-motion. The transformed point cloud is voxelized for the follow-up map fusion. We denote the newly observed point clouds (after voxelization) as (S]
= {(ss, ox)) a=1 and the current voxel map as M = {(Pm, IMM mm))m The newly observed
voxels are fused to update the previous map as:
Pm -2+02
Pm
+2 Ps: on (0 o 3)
(2)
Here, we assume ps and Pmn are the semantic log probability vectors (obtained from Sec. 3.2) be- longing a pair of corresponding voxels in the new frame and the current map respectively, a, and om the estimated variances of these two voxels. Initially, the variance a, of the observed voxel is predicted by CNN. This CNN is trained with the depth estimation network in an unsupervised manner by assuming Gaussian noise model following [64]. The uncertainty aware mapping makes it possible to correct previous mapping errors as the exploration goes on. Our probabilistic mapping is proven to be essential especially when the depth mcasurements are erroncous.
3.4 Program Executor
After concept learning and mapping, we take the average semantic probability map from demonstra- tions as navigation policy. It indicates the location where each type of object most likely exists. Although the previous work FILM [6] trains semantic policy model to predict the possible location of an object given part of the semantic layout, the model is likely to be over- wer-fiitinn. In contrast, our semantic policy is the averaged semantic map based on statistics without training, producing stable results. As shown in Fig. 2, given the predicted subprogram, the current semantic map, and a search goal sampled from the semantic policy (averaged semantic map). the deterministic policy outputs a navigation interaction action.
The deterministic policy is defined as follows. If the object needed in the current subtask is observed in the current semantic map, the location of the object is selected as the goal; otherwise, we sample the location based on the distribution of the corresponding object class our averaged semantic map as the goal. The robot agent then navigates towards the goal via the Fast Marching Method [65] and performs the required interaction actions.
4 Experiments
We show the effectiveness of each component of ECL the ALFRED [1] benchmark. For the EIF task, we report Success Rate (SR), goal-condition success (GC), path length weighted SR (PLWSR),
5

---

Table 1: Comparison with other methods on ALFRED benchmark. The upper part contains unsu- pervised methods while the lower part contains the supervised counterparts with semantic or depth supervisions. We also report the ECL-Oracle model as an upper bound, with supervised segmenta- tion and depth. The top scores are in bold. Red denotes the top success rate (SR) (ranking metric of the leaderboard) on the tes sest_unseen set.
Supervision
Test Seen
Test Unseen
Method
Semantic
Depth
PLWGC (%6
(56
GC
PLWSR (%)
(56)
SR
PLWGC (5)
PLWSR (%)
(%)
SR
SEO2$EO
MOCA [2]
22.05
28
15.10
22.05
9.99
14.28
2.72
5.30
LAV
13.18
23.21
6.31
13.35
10.47
17:27
3.12
6.38
E.T.
4,10
ECL (OURS)
9.47
18.74
4.97
10.37
11.50
19.51
4.13
9.03
EMBERT
32.63
24.36
31.48
8.87
2.17
5.05
LWIT [19]
23.10
40.53
43.10
30.92
16.34
20.91
5.60
9.42
HITUT [22]
17.41
29.97
11,10
21.27
11.51
20.31
5,86
13.87
ABP
VLNBERT [21]
19.48
33.35
13.88
24.79
13.18
22.60
7,66
16.29
HLSM [S]
11 53
35.79
6,69
25.11
8:45
27.24
4.34
16.29
ECL
ECL-ORACLE (OURS)
15.19
36.40
10.56
25.90
13.08
35.02
9.33
23.68
25
60
376
Laarned Word Ennledding
Maximum Fusion.
20-
Bayesian Flkering
16.6
10.9
25.4
10-
15.9
6.0
10G 11.1
2.8
Grounding Auc. PLWGC
GC
PLWSR
SR
PLWGC
GC
PLWSR
SR
Figure 3: Results with different language repre- sentations in concept learning on test.unseen.
Figure 4: Evaluation with different semantic mapping techniques on test.unseen.
and path length weighted GC (PLWGC) the evaluation metrics both seen and unseen environ- ments. SR is binary indicator of whether all subtasks conditions completed at the end of an episode. Both SR and GC can be weighted by (path length of the expert trajectory)/ length taken by the agent), which are called PLWSR and PLWGC. We also report the (grounding) accuracy for the concept learning and downstream reasoning tasks. More details of the benchmark and the training settings each component can be found in Appendix.
completed. GC denotes the ratio of goal
4.1 Embodied Instruction Following on ALFRED
The results on ALFRED are shown in Tab. 1. ECL. achieves new state-of-the-art (SR: 9.03 vs. 8.57) on the test_urt sestun set when there are no semantic and depth labels. Though counterparts [4, 2] have better performance on bast_seen, they are likely to be over-fitting by simply memorizing the ECL achieves stable results between the test_seen set and unseen show trajectory execute "place washed
visible scenes. However, set, demonstrating its generalizability. In Fig. sponge in tub" and the intermediate estimates generated by ECL.
When depth supervision is used, our ECL w. depth model has a 17.24% success rate on the test_unseen set, as well as competitive goal-condition success rate and path length weighted re- sults. Note that FILM [6] leverages additional dense semantic maps as supervision train a policy network, hence not apple-to-apple comparable
work. We report the FCL-Oracle model as an upper bound, which learns supervised segmentation and depth, and can be seen as variant of FILM [6] without the policy network. It achieves 23.68% SR on masteunseen.
Ablation Study. We conduct experiments to study the effect of the language representation in concept learning, and the mapping strategy map construction. The results are shown in Fig. and Fig. offering 1) benefiting from the natural structure of language, the word embedding is better than the learned encoding, and 2) Bayesian filtering outperforms maximum fusion the soft probabilities could correct wrong labels.

---

Instruction: Place washed sponge in tub.
Search for "Sponge"
Plck up "sponge"
Go to "SinkBasin"
Wash "Sponge"
Pick up "Sponge"
"Sponge" "Hub"
Egocentric RGB Observation
1=4
1=12
t=17
1-45
F-50
1-53
Unsupervised Depth Estimation
Unsupervised Semantic Grounding
Map
Figure 5: Visualization of intermediate estimates by ECL when an agent tries to accomplish an instruction. Based the RGB observations, our system estimates the depths and semantic masks. The BEV semantic map is gradually established with these estimates as the exploration goes on. The goals (sub goal/final goal) are represented by big blue dots in the semantic map, while the agent trajectories are plotted as small red dots.
Table 2: The percentage of failure cases belong-
Table 3: Concept reasoning accuracy. We lever-
ing to different failure modes on validation set.
age ECL to infer whether an object exists or to
Error mode
Seen %
Unseen %
count its number in scene.
Grounding erros' Tarret not found
36.38
28.53
Model
Grounding % Exist Count
Interaction failures
6.59
10.39
Random Guess
50.0
25.0
Collisions
4.34
4.43
Blocking/Ohjeet not accessible
31.29
39.75
C3D [66]
78.1
34.4
Others
21.41
16.90
(Ours)
57.6
90.6
56.3
4.2 Evaluation of Concept Learning
Quantitative Evaluation. We report the per-task evaluation results in Fig. 6. The concept learn- ing accuracies objects "HandTowel", "Laptop", "Bowl", and "Knife" are above 80%, because these objects frequently appear alone in the scene (easy learn and less likely be confused). Objects like "Basketball", "Glassbottle". "Cellphone", and "Teddybear" are rarely shown in the vironment, thus their concepts diffieult to learn. We also notice that the object "apple" appears very rarely, but our model grounds its concept well with the help of language embeddings, e.g. the relationship between "tomato" and "apple".
Error Modes. Tab. 2 shows the emro mode of ECL w. depth on ALFRED validation set. We see that "blocking and object not accessible" the common enror mode, which is mainly caused by incorrectly estimated depth undetected visual objects/concepts. Additionally, around 30% of the failures due to wrongly grounded concepts the target object not being found. If replace our unsupervised concept learning with supervised semantics (ECL-Oracle), the percentage of the emorr mode for "Grounding error/Target not found" changes to 7.38% and "blocking and object not accessible" becomes 44.00%.
Visualization. We visualize our concept learning results in Fig. 8 by showing the original image, the supervised learned semantics, and grounded semanties by the concept learner. We observe concept learning keeps more object proposals than the supervised model. While most of the
7

---

Q1: How many sofas are in the room?
Basketball +00
Overall Average
Glassbottle 41.3
CollPhone
13.2
A: 2.
TeddyBear
Clath
Mug
46.7
Apple
50.2
Plate
51.4
Baok
53.1
Newspaper
15.5
60.0
Vase
Visual Frame
Semantic Map
Knife
Pillow
68.7
Q2: Are the knife and spatula on the same table?
Towel
67.4
FloorLamp
68,
Laptop
70.0
A: Yes,
74.0
Bowl
KeyChain
HandTowel
85.4
20
40
60
80
100
Concept Leaminn Accuracy
(Small Object) (5)
Visual Frame Leamed Concepts
Figure Concept learning vi- sualization. From left to right: the original image, supervised instance segmentation map, and our concept learning results.
Figure 6: Concept learning ac- Figure 7: Examples of con- curacy. Results for challenging cept reasoning by ECL: the small objects are shown. Com- count task and the high-level plete analyses are in appendix. question-anse answering.
main objects in an image can grounded correctly, there exist few wrong labels in overlapped corner areas. We also show failure cases the third and fourth rows of Fig. 8. The first recognizes "floor" "diningtable", bug that could be fixed by our Bayesian Eltering-based semantic mapping. The other one identifies "coffeetable" "drawer", which causes the error "target not found". The instruction would succeed if we take the ground truth concept for "coffeetable".
4.3 Concept Reasoning
In addition to EIF, we show the learned concept can be transferred to embodied reasoning tasks, e.g., (i) the existence of objects in the scene, (ii) count the number of objects in the scene (Fig. 7). We build the reasoning dataset by randomly sampling 16 objects from 10 scenes, of which scenes are used for training and the other 2 for testing. A naïve baseline is random guessing with 50% accuracy for the exist task and 25% accuracy for the count task. We also train C3D model [66] that samples 16 frames as input and outputs predictions directly. Our ECL performs clear and step-by-step interpretable reasoning through semantic grounding and mapping. As Tab. 3 shows, it outperforms both baselines by large margin. By embodied concept learning, ECL can also resolve high-level 3D question-answering tasks, like "whether two objects appear on a table" in Fig. 7.
5
Discussions
This paper proposes ECL., a general framework that can ground visual concepts, build semantic maps and plan actions to accomplish tasks by learning purely from human demonstrations and lan- guage instructions. While achieving good performance on embodied instruction following, ECL has limitations. Although the ALFRED benchmark is photo-realistic, comprehensive, and challenging, there still exists a gap between the embodied environment and the real world, We leave the physical deployment of the framework as our future work.
Acknowledgements. This work supported by MIT-IBM Watson AT Lab and its member company Nexplore, Amazon Research Award, ONR MURI, DARPA Machine Common Sense program, ONR N00014418-11220 and Mitsubishi Electric. Ping Luo is supported by the General Research Fund of HK No.27208720, 1o.127222220 and No. 17200622..

---

References
[1] M. Shridhar, J. Thomason, D. Gordon, Y. Bisk, W. Han, R. Mottaghi, L. Zettlemoyer, and D. Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 10740-10749, 2020.
[2] K. P. Singh, S. Bhambri, B. Kim, R. Mottaghi, and J. Choi. Moca: A modular object-centric approach for interactive instruction following, arXiv preprint arXiv:2012.03208 2020.
[3] K. Nottingham, L. Liang, D. Shin, C. C. Fowlkes, R. Fox, and Singh. Modular framework
for visuomotor language grounding. arXiv preprint arXiv:2109.02161, 2021.
[4] A. Pashevich, C. Schmid, and C. Sun. Episodic transformer for vision-and-langaage naviga- tion. arXiv preprint wrXiv22105.06455 2021.
[5] V. Blukis, C. Paxton, D. Fox, A. Garg, and Y. Artzi A persistent spatial semantic representation for high-level natural language instruction execution. In Proceedings of the Conference on Robot Learning (CoRL), 2021.
[6] S. Y. Min, D. S. Chaplot, P. Ravikumar, Y. Bisk, and R. Salakhutdinov. Film: Following instructions in language with modular methods. arXiv preprint arXiv:2110.07322, 2021.
[7] M. R. Walter, S. M. Hemachandra, B. S. Homberg, S. Tellex, and S. Teller. Learning semantic maps from natural language descriptions. In Robotics: Science and Systems, 2013.
[8] S. Hemachandra, F. Duvallet, T. M. Howard, N. Roy, A. Stentz, and M. Walter. Learning models for following natural language directions in unknown environments. In 2015 IEEE International Conference on Robotics and Automation (ICRA). pages 5608-5615. IEEE, 2015.
[9] S. Patki, A. F. Daniele, M. R. Walter, and T. M. Howard. Inferring compact representations for efficient natural language understanding of robot instructions. In 2019 International Confer- ence on Robotics and Automation (ICRA), pages 6926-6933. IEEE, 2019.
[10] I. Kostavelis and A. Gasteratos. Semantic mapping for mobile robotics tasks: A survey. Robotics and Autonomous Systems, 66:86-103, 2015.
[11] Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sünderhauf, I. Reid, S. Gould, and A. Van Den Hengel. Vision-and iidoon-nnd-aaee lan-lan navigation: Interpreting visually- naviga- tion instructions in real environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3674-360 7774--363, 2018.
[12] D. Fried, R. Hu, V. Cirik, A. Rohrbach, J. Andreas, L.-P. Morency, T. King-Kirkpatrick, K. Saenko, D. Klein, and Darrell. Speaker-follower models for dision-and-language navi- gation. In Advances in Neural Information Processing Systems, 2018.
[13] F. Zhu, Y. Zhu, X. Chang, and X. Liang. Vision-language navigation with self-supervised auxiliary reasoning tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10012-10022, 2020.
[14] L. Ke, X. Li, Y. Bisk, Holtzman, Z. Gan, J. Liu, J. Gao, Y. Choi, and S. Srinivasa. Tactical rewind: Self-correction via backtracking in vision-and -nd-o-uu-- navigation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 6741-6749, 2019.
[15] X. Wang, Q. Huang, A. Celikyilmaz, Gao, D. Shen, Y.-F Wang, W. Y. Wang, and L. Zhang- Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6629-6638, 2019.

---

[16] C.-Y.Ma, Z. Wu, G. AlReeg,,, C. Xiong, and Z. Kira. The regretful agent: Heuristic-aided navi- gation through progress estimation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 6732-6740, 2019.
[17] A. Zadaianchuk, G. Martius, and Yang. Self-supervised reinforcement learning with inde- pendently controllable subgoals. In Conference on Robot Leurning. PMLR, 2022.
[18] A. Suglia, Q. Gao, J. Thomason, G. Thattai, and G. Sukhatme. Embodied bert: A transformer model for embodied, language-guider visual task completion. arXiv preprint arXiv:2108.04927, 2021.
[19] V.-Q. Nguyen, M. Suganuma, and Okatani. Look wide and interpret twice: Improving per- formance on interactive instruction-following tasks. arXiv preprint arXiv:2 (tt:::20060009 2021.
[20] B. Kim, S. Bhambri, K. Singh, R. Mottaghi, and J. Choi. Agent with the big picture: Per- ceiving surroundings for interactive instruction following. In Embodied Al Workshop CVPR, 2021.
[21] C. H. Song, J. Kil, T.-Y. Pan, B. M. Sadler, W.-L. Chao, and Y. Su. One step at a time: Long- horizon vision-and- lnn--andn-aageeeeeeeeee navigation with milestones. arXiv preprint rrXiv:2202.07023 2022.
[22] Y. Zhang and J. Chai. Hierarchical task learning from language instructions with unified trans- formers and self- safitoonitoinng, arXiv preprint XXXv:2106.03427,, 2021.
[23] D. S. Chaplot, D. P. Gandhi, A. Gupta, and R. R. Salakhutdinov. Object goal navigation using goal-oriented semantic exploration. Advances in Neural Information Processing Systems, 33, 2020.
[24] D. S. Chaplot, D. Gandhi, S. Gupta, Gupta, and Salakhutdinov. Learning to explore using active neural slam. arXiv preprint M0Xi::2004.0555:: 2020.
[25] C. Li, F. Xia, R. Martín-Martín, M. Lingelbach, S. Srivastava, B. Shen, K. E. Vainio, C. Gok- men, G. Dharan, T. Jain, et al. igibson 2.0: Object-centric simulation for robot learning of everyday houschold tasks. In Conference on Robot Learning. PMLR, 2022.
[26] A. Das, S. Datta, G. Gkioxari, S. Lee, D. Parikh, and D. Batra. Embodied question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1-10, 2018.
[27] D. Gordon, A. Kembhavi, M. Rastegari, J. Redmon, D. Fox, and A. Farhadi. Iqa: Visual question answering in interactive environments. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4089 4098, 2018.
[28] Y.-.. Liao, X. Puig, M. Boben, A. Torralba, and Fidler. Synthesizing environment-awwe aware activities via activity sketches. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 6291-6299, 2019.
[29] D. Trivedi, J. Zhang, S.-H. Sun, and J. Lim. Learning to synthesize programs as interpretable and generalizable policies. Advances in neural information processing systems, 34:25146 25163, 2021.
[30] R. Wang. J. Mao, S. J. Gershman, and J. Wu. Language-mediated, object-centric representation learning, arXiv preprint arXiv:2012.15814, 2020.
[31] Y. Bisk, A. Holtzman, J. Thomason, J. Andreas, Y. Bengio, J. Chai, M. Lapata, A. Lazari- dou, J. May, A. Nisnevich, et al. Experience grounds language. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), 2020.
10

---

[32] M. Prabhudesai, H.-Y. F. Tung, S. A. Javed, M. Sieb, A. W. Harley, and K. Fragkiadaki. Embodied language grounding with 3d visual feature representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2220-2229, 2020.
[33] K. He, G. Gkioxari, Dollár, and R. Girshick. Mask r-cnn. In Proceedings of the IEEE international conference computer vision, pages 2961-2969, 2017.
[34] M. Ding, Y. Huo, H. Yi, Z. Wang. J. Shi, Z. Lu, and P. Luo. Learning sopth-guided convolutions for monocular 3d object detection. In CVPR, pages 1000-1001, 2020.
[35] M. Ding, B. Xiao, N. Codella, P. Luo, Wang, and L. Yuan. Davit: Dual attention vision transformers. In ECCV, 2022.
[36] C. Kong, D. Lin, M. Bansal, R. Urtasun, and S. Fidler. What are you talking about? text-to- image coreference. In CVPR, 2014.
[37] B. A. Plummer, L. Wang, C. M. Cervantes, J. Caicedo, Hockenmaier, and S. Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In 2CCVV0015
[38] C. Matuszek, N. FitzGerald, L. Zettlemoyer, L. Bo, and D. Fox. A joint model of language and perception for grounded attribute learning. In ICML, 2012.
[39] A. Karpathy and Fei-Fei. Deep visual-semantic alignments for generating image descrip- tions. In CVPR, 2015.
[40] J. Mao, J. Huang, A. Toshev, O. Camburu, A. L. Yuille, and K. Murphy. Generation and comprehension of unambiguous object descriptions. In CVPR, 2016.
[41] H. Zhang, Y. Niu, and S.-F. Chang. Grounding referring expressions images by variational context. In CVPR, 2018
[42] J. Yang, H.-Y. Tung, Y. Zhang, G. Pathak, A. Pokle, C. G. Atkeson, and K. Fragkiadaki. Souualy--roundde library of behaviors for manipulating diverse objects across diverse config- urations and views. In 5th Annual Conference on Robot Learning, 2021.
[43] Z. Chen, P. Wang. L. Ma, K.-Y. K. Wong, and Q. Wu. Cops-ref: A new dataset and task on compositional referring expression comprehension. In Proceedings of the IEEE/CVF Confer- ence Computer Vision and Pattern Recognition, pages 10096, 2020.
[44] J. Mao, C. Gan, P. Kohli, J. B. Tenenbaum, and J. Wu. The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision. In ICLR, 2019.
[45] Z. Chen, J. Mao, Wu, K.-Y. Wong, B. Tenenbaum, and C. Gan. Grounding physical concepts of objects and events through dynamic visual reasoning. In ICLR, 2021.
[46] J. Mao, F. Shi, J. Wu, R. Levy, and J. Tenenbaum. Grammar- grounded lexicon learning. Advances in Neural Information Processing Systems, 2021.
[47] B. Bergen and J. Feldman. Embodied concept learning. In Handbook of Cognitive Science, pages 313-331. Elsevier, 2008
[48] K. M. Hermann, F. Hill, S. Green, F. Wang, R. Faulkner, H. Soyer, D. Szepesvari, W. M. Czarnecki. M. Jaderberg, D. Teplyashin, et al. Grounded language learning in a simulated 3d world. arxiv, 2017.
[49] Z. Chen, K. Yi, Y. Li. M. Ding, A. Torralba, J. B. Tenenbaum, and C. Gan. Comphy: Compo- sitional physical reasoning of objects and events from videos. In International Conference Learning Representations, 2022.

---

[50] M. Ding, Z. Chen, T. Du, P. Luo, J. Tenenbaum, and C. Gan. Dynamic visual reasoning by learning differentiable physics models from video and language. Advances in Neural Informa- fion Processing Systems, 34, 2021.
[51] M. Feng, Z. Li, Q Li, L. Zhang, X. Zhang, G. Zhu, H. Zhang, Y. Wang, and A. Mian. Free- form description guided 3d visual graph network for object grounding in point cloud. In ICCV, 2021.
[52] J. Roh, K. Desingh, A. Farhadi, and D. Fox. Languagerefer: Spatial-lanuaage model for 3d visual grounding. In Conference on Robot Learning. PMLR, 2022.
[53] Achlioptas, Abdelreheem, F. Xia, M. Elhoseiny, and Guibas. Referit3d: Neural listen- for fine-grained 3d object identification in real-world scenes. In ECCV, 2020.
[54] M. Lewis, Liu, Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and
L.. Zettlemoyer. BART: Denoising sequence-to-seqquene pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online, July 2020. Asso- ciation for Computational Linguistics. doi: 10. 100. l/aclantholll Sentthooogy.org222222 acl-main.700 703.
URL https:
[55] Meelfy. cntorch_pretraineee pretrained_BERT.
2019. URL https://githin http://hhwb..cco/eeaaat//tttooo
[56] C.-C. Carbon. Understanding human perception by human-made illusions. Frontiers human neuroscience, 8:566, 2014.
[57] D. Regan. Human perception objects. Sunderland, MA: Sinauer, 2000.
[58] J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv:1607.06450. 2016.
[59] D. Hendrycks and K. Gimpel. Gaussian error linear units (gelus). arXiv preprint 1rX1v:1606.08415, 2016.
[60] H. W. Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97, 1955.
[61] E. A. Maguire, N. Burgess, J. G. Donnett, R. S. Frackowiak, C. D. Frith, and J. Keeffe. Knowing where and getting there: a human navigation network. Science, 280($365):211-224 1998.
[62] C. Godard, O. Mac Aodha, M. Firman, and G. Brostow. Digging into self-supervised monoc- ular depth estimation. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3828-3838, 2019.
[63] T. Zhou, M. Brown, N. Snavely, and D. G. Lowe. Unsupervised learning of depth and ego- motion from video. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1851-1858, 2017.
[64] A. Kendall and Y. Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems, 30, 2017.
[65] A. Sethian. A fast marching level set method for monotonically advancing fronts. Pro- ceedings of the National Academy of Sciences, 9314):19111155, 1996. ISSN 0027-8424. doi: 1ot:10.10730 100..000//mss... 93.4........... 15911. URL https: h/ww::::/www.... pnas. cttcc//otttt////////////
[66] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri. Learning spatiotemporal fea- tures with 3d convolutional networks. In Proceedings of the IEEE international conference computer vision, pages 4489 2015.
12


Reviews:
[{"review_id": "cY1EODTaZJS", "reviewer": "Reviewer_Zyvv", "summary": "The paper proposed a system which learns depth in an unsupervised manner and semantic segmentation in a self-supervised manner from object segmentations from Mask R-CNN and task demonstrates and language instructions. Using the resulting predicted depth and semantic segmentations, the approach builds an uncertainty-aware semantic 3D map which can continually update information from new views. This map can be used for downstream task solving or question answering.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n - Interesting approach to fusing semantic and spatial uncertainty into one unified 3D scene representation for planning.\n - Impressive quantitative results on Alfred benchmark.\n\nWeaknesses:\n- Key assumption for self-supervised semantic segmentation matching may not apply on other simulators or real robots. This assumption is also not addressed in the limitations section.\n- Some training details unclear from writing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall an interesting system which combines depth and semantic segmentation self-supervised learning with 3D map building for task solving. While their quantitative results demonstrate impressive performance on the ALFRED benchmark, I'm doubtful whether their key assumption allows the system to work on real robots. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": " - A key limitation of this work are their core assumptions, which is that \"1) If a subgoal completes, ... until the robot drops it\" (L137-143). This is an artifact of the AI2THOR environment, and may not generally be applicable in other simulators and in the real world. \n- From section 3.2, it is not clear how the self-supervised semantic model is trained. For instance, after reading the paper, I still have questions about why hungarian matching over a randomly initiallized MLP might make sense during the beginning of training, and how is the MLP's supervision signal derived in a self-supervised manner. Please provide more details.\n- When the robot is deployed in home level tasks, achieving high enough resolution using voxel representations may cause prohibitive memory usage. While this 3D representation makes it straight forward to aggregate semantic and spatial uncertainties on a grid, it is also not scalable. \n- Does it make more sense to allow ECL-Oracle access to ground truth segmentation and depth rather than to ground truth segmentation and depth supervision? This would help gauge ECL's performance if its self-supervised depth and semantic segmentation was perfect.\n- Mistake in L231 about objects that appear infrequently in the environment.\n- I'm curious to see exposure frequency v.s. concept learning accuracy of object classes to back up statements in L229 - 232."}}, {"review_id": "CNEQM-4zRN", "reviewer": "Reviewer_XMVt", "summary": "This paper presents Embodied Concept Learner (ECL), a self-supervised framework for grounding concepts and following instructions in 3D environments. The framework is set in the ALFRED instruction following environment. Instead of relying on ground-truth segmentations, ECL makes connections between instructions, observations, and actions to learn object-centric concepts like “knife” and “soap bottle”. Similarly, for estimating depth, the method uses an unsupervised-learning approach. Together, the learnt concepts and depth predictions are fused with a Bayes filter to build a spatial-semantic map of the scene. This map is used with an executor module to solve instruction following tasks. The experiments include evaluations on the ALFRED benchmark with metrics like success and path-weighed success on seen and unseen scenes. While the proposed approach does not achieve state-of-the-art performance, it achieves compelling results without using any ground-truth segmentations or depth. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n+ Overall, the idea of using self-supervised learning to learn object-centric concepts to aid embodied agents is an interesting and novel approach. Most existing methods for the ALFRED challenge make heavy use of ground-truth segmentation masks and perfect depthmaps from the simulator. Such supervision is often hard to obtain in real-world robotics where it’s difficult to pre-define a set of object categories, or obtain noiseless depth estimates from onboard sensors. As such, ECL takes a good step towards realizing approaches that have been prevalent in the Embodied AI community for instruction following tasks. \n\n+ The experiments include a good set of comparisons to prior work on the ALFRED benchmark. Additionally, the results also include ablations investigating various aspects of ECL like Bayesian Filtering, word embeddings, and an analysis on common failure cases. If the authors open-source their implementation, these results should be easily reproducible.\n\n+ The results in Section 4.2 and 4.3 are interesting. Without any explicit object-level supervision, the learnt concepts can be re-used to query questions like “how many sofas are in the room?”. Future works could follow up these investigations by studying the generalizability of bottom-up perceptual representations that are learnt not through explicit supervision, but from environmental interactions.\n\n**Weaknesses**\n\n- One major concern is that ECL’s approach to self-supervised concept learning might be a bit specific to the ALFRED environment. As described in Section 3.2, the training data is composed of frames from before and after interacting with objects. These interactions are observed through discrete changes in image observations, e.g. picking up an object immediately zaps the object to the agent’s visual frame in the center. Such obvious visual cues might not be apparent in real-world settings, especially if the agent is interacting with objects through a manipulator. \n\n- Section 3.2 is missing details on how exactly the concept learner is trained. The word embeddings and visual representations are fine-tuned with MLP layers, but they are matched with the Hungarian algorithm. How is gradient descent used with the Hungarian algorithm? And what is the training loss? Without these details, it’s hard to understand how exactly ECL works in practice. Similarly, is the Program Executor (in Section 3.4) a hand-coded policy specifically designed for ALFRED tasks? If there is no learning or planning involved here, again, such hand-coded policies might be specific to ALFRED-only tasks. \n\n- One minor concern regarding the prose is the use of frequent analogies comparing ECL’s approach to “human-like” and “baby-like” learning. While it’s great that ECL is inspired by theories in cognitive science, and a lot of its core ideas are rooted in cognitive development, some references like “Humans … group[ing] pixels regions without knowing their semantics” and making an “agent behave like a baby” with a Bayes filter for mapping – is unsubstantiated. These analogies could be rephrased or included with proper citations to cognitive science literature to back-up the claims. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "ECL presents a novel approach for learning object-centric concepts without explicit segmentation masks or depth supervision. ECL’s contributions might be limited to ALFRED tasks, but nonetheless they might be broadly useful to the embodied AI and robotics community. \n\n**Post Rebuttal**  \nThese responses address most of my concerns. I am still a bit skeptical about some of the cogsci claims made in the paper like \"humans ... [group]ing pixels\" and equating Bayes-filter mapping to how humans map and navigate. I checked out the references the authors mentioned, and I couldn't find any evidence of \"pixel\" representations or Bayesian-filtering. But overall, I think the contributions here are still of interest to the embodied-AI/robotics community, and so I will keep my score of Weak Accept.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "See above on the concern regarding the approach being specific to ALFRED and missing details. ", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "7rCoIxOXCPw", "reviewer": "Reviewer_TjRQ", "summary": "This paper addresses the problem of embodied concept learning, whose aim is to learn concepts through interaction in an interactive 3D environment. Specifically, this work focuses on learning to ground visual concepts from human demonstrations and language instructions and enabling embodied instruction following. To this end, the paper proposes a framework that parses an instruction into a sequence of subtasks, grounds visual concepts based on language descriptions, leverages learned concepts to estimate the depth and construct semantic maps, and fulfills each subtask with a crafted rule-based policy. The experiments on the ALFRED benchmark show that the proposed framework outperforms baselines when no depth and semantic label is provided in terms of success rate and goal-condition success. I believe this work explores a promising research direction (i.e. embodied concept learning) and presents an effective framework to address the problem. ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Paper strengths and contributions\n**Motivation and intuition**\n- The motivation for studying embodied concept learning is convincing.\n- Studying embodied concept learning from and for embodied instruction following task seems like a decent choice.\n\n**Technical contribution**\n- Utilizing an objects proposal network to obtain detected objects makes sense and seems to be effective\n- Using the Hungarian maximum matching algorithm seems effective for matching detected objects and extracted word embeddings.\n- The map constructor that builds up semantic maps based on estimated depths and grounded visual concepts seems to incorporate the information well.\n\n**Clarity**\nThe overall writing is very clear. The authors utilize figures and examples well to illustrate the ideas. \n\n**Experimental results**\n- The presentation of the experimental results is very clear.\n- The experimental results on ALFRED show that the proposed framework (1) outperforms baselines when no depth or semantic label is available, (2) generalizes better between test-seen and test-unseen, and (3) achieves competitive performance when depth supervision is used.\n- The qualitative results in both the main paper and the appendix are well-analyzed and interesting.\n\n**Ablation study**\nAblation studies justify the effect of the language representation in concept learning and the mapping strategy in map construction. \n\n## Paper weaknesses and questions\n\n**Oversell**\nI feel the authors oversell the paper a little bit. After reading the abstract and the introduction, \nI thought the proposed framework learns from scratch from only interactions and human demonstrations.\nYet, word embeddings from a pre-trained BERT model are utilized which can provide relationships among objects and an object proposal network is leveraged which has learned low-level visual features and how to group pixels.\nIn my opinion, the authors can tone down the key motivation a little bit so that the readers do not get disappointed.\n\n**Instruction-program pairs**\nThis work fine-tunes a pre-trained BERT model to convert an instruction to a program (i.e. a sequence of subtasks) using paired data from the subtask sequence annotations in ALFRED. I wonder if the instruction parser can learn from the task completion signal when there is no instruction-program pair available, or if paired data is just required.\n\n**Instruction parser's accuracy**\nI wonder how accurate the instruction parser is. It would be great if this information can be provided.\n\n**Training object proposal network**\nHow was the object proposal network trained? Was it trained on data from ALFRED? I wonder how accurate it has to be to allow the whole framework to work. Also, is it fine-tuned during the learning?\n\n**Learning continuous concepts**\nThis work presents an effective framework to ground visual concepts by matching them to language descriptions in a \"discrete\" manner. I wonder how we can learn continuous concepts or even dynamic concepts.\n\n**Deterministic (rule-based) policy**\nThis work incorporates a simple deterministic policy. I wonder how this can scale up to environments that require longer-term exploration.\n\n**Related work**\nI feel the authors can include the following papers to make the related work more comprehensive:\n- Synthesizing Environment-Aware Activities via Activity Sketches: it generates program sketches from instructions. \n- Learning to Synthesize Programs as Interpretable and Generalizable Policies: it discusses interpretability and generalizability. \n\n====== After rebuttal ======\n\nI thank the authors for the response and the revision, which addresses of my concerns. I have carefully read other reviews and decided to keep my original rating.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I believe this work explores a promising research direction (i.e. embodied concept learning) and presents an effective framework to address the problem. While I have a few minor concerns stated above, I enjoy reading this paper and would love to see this paper included at the conference.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "Described in the Strengths And Weaknesses section.\n", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "-oGYfKFn9s", "reviewer": "Reviewer_dHJH", "summary": "This paper presents embodied concept learner that can ground the objects included in the instructions and constructs a semantic map. Grounding is achieved by a minimum cost assignment problem between visual features and word embeddings. The semantic map is constructed by assigning the object probabilities to 3D space. Based on the semantic map and parsed instructions, the agent can predict actions.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n- ECL achieved high performance in unseen test tasks of ALFRED. This result indicates that the ECL has high generalizability. \n\n# Weaknesses and suggestions\n- First of all, I have a question about the purpose of this study written in lines 47-56. This study aims to learn knowledge from expert demonstrations without supervision. However, this study uses the labels in ALFRD, and this is considered different from learning by a baby. \n- For a similar reason, if demonstrations mean the labels in ALFRED in this study, I think these are not demonstrations but dis ambiguated labels. \n- Sec 3.2: How were the MLP that projects visual representation into semantic space of word learned? If this projection is already learned, this leads that the agent already has knowledge about the objects and their names. \n-  Figure 3: I do not understand the difference between learned encoding and word embedding. \n- Lines 229-231: typos?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I think this paper should be revised to be accepted. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "Described in the strengths and weaknesses section.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)