Grounded Compositional Generalization with Environment Interactions
Authors: Yuanpeng Li
Venue:  (2021)
Keywords: compositional generalization
grounding

Abstract: In this paper, we present a compositional generalization approach in grounded agent instruction learning. Compositional generalization is an important part of human intelligence, but current neural network models do not have such ability. This is more complicated in multi-modal problems with grounding. Our proposed approach has two main ideas. First, we use interactions between agent and the environment to find components in the output. Second, we apply entropy regularization to learn corresponding input components for each output component. The results show the proposed approach significantly outperforms baselines in most tasks, with more than 25% absolute average accuracy increase. We also investigate the impact of entropy regularization and other changes with ablation study. We hope this work is the first step to address grounded compositional generalization, and it will be helpful in advancing artificial intelligence research.


Under review as a conference paper at ICLR 2021
GROUNDED COMPOSITIONAL GENERALIZATION WITH ENVIRONMENT INTERACTIONS
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper, we present compositional generalization approach in grounded agent instruction learning. Compositional generalization is an important part of human intelligence, but current neural network models do not have such ability, This is more complicated in multi-modal problems with grounding. Our proposed approach has two main ideas. First, we use interactions between agent and the environment to find components in the output. Second. we apply entropy regular- ization to learn corresponding input components for each output component. The results show the proposed approach significantly outperforms baselines in most tasks, with more than 25% absolute average accuracy increase. We also investi- gate the impact of entropy regularization and other changes with ablation study, We hope this work is the first step to address grounded compositional generaliza- tion, and it will be helpful in advancing artificial intelligence research. The source code is included in supplementary material.
INTRODUCTION
Compositional generalization is key skill for flexible and efficient learning. Humans leverage compositionality to create and recognize new combinations of familiar concepts (Chomsky) 1957] Minsky 1986). Though there are many progresses for machine learning and deep learning in various areas recently (LeCun et al., 2015), current main learning algorithms are not able to perform com- positional generalization, and require many samples train models. Such efficient learning even more important when machines interact with the environment for grounding. because interactions are usually slow.
Machine learning has been mostly developed with an assumption that training and test distributions are identical. Compositional generalization, however, kind of out-of-distribution generaliza- tion (Bengio 2017), where training and test distributions are different. During training, dataset does not contain the information of the difference, so it can only be given as prior knowledge. In compo- sitional generalization, sample is combination of several components. Test distribution changes as test samples are new combinations of seen components in training. For example, if we can find "large apple" and "small orange" in some environments, then we can also find "large orange" among multiple objects in a new environment.
The recombination is enabled when an output component depends only on the corresponding input components, and invariant of other components (please see Section 4.1 for more details). So there aspects consider. What are the components in output, and how to find the corresponding input signals. We propose to use interactions between agent and the environment to define output components. This is analogues to model-free reinforcement learning (Sutton & Barto 2018). where an agent does not have an environment model, but leans to act at each step during the interactions with the environment. Then we use entropy regularization (Li et al. 2019 Li & Fisner 2019) to leamn the minimal input components for outputs.
We evaluate the approach with gSCAN dataset (Ruis et al., 2020). which is designed to study com- positional generalization in grounded agent instruction learning. Please see Figure] | for examples. The results show the proposed approach significantly outperforms baselines in most tasks, with more than 25% absolute average accuracy increase, and the high accuracy indicates that the pro- posed approach addresses the designed grounded compositional generalization problems in these tasks. We also look into the impact of entropy regularization and other changes with ablation study.

---

Under review as a conference paper at ICLR 2021
(a) Walk to small cylinder.
(b) Walk to a large blue circle.
Figure I: Examples of gSCAN dataset. The dataset evaluates compositional generalization ability in grounded instruction learning. The agent needs to understand command and the environment to take sequence of actions. Training and test data have different distributions and tasks require different kinds of compositional generalization. Please refer to Section 5 for more details.
We hope this work will be helpful advancing grounded compositional generalization and artificial intelligence research.
The contributions of this paper can be summarized as follows.
This is the first work to enable accurate compositional generalization in grounded instruc- tion learning problem, serving for analyzing and understanding the mechanism.
The novelty of this paper is to find that the combination of environment interaction and entropy regularization helps the generalization.
RELATED WORK
Compositional generalization research has a long history, and recently there are increasing focus on this area. SCAN dataset (Lake 2019) was proposed to study compositional generalization in instruction learning. It maps a command sentence to sequence of actions. This dataset has a property that input words and output actions have direct correspondence. Though some NLP tasks, such as machine translation, have similar property, not all problems to the setting. Also this dataset does not contain environment for agent take actions.
SCAN dataset inspired multiple approaches (Russin et al., 2019 Andreas 2019 Li et al. 2019
Lake 2019 Gordon al. 2020: Liu et al. 2020). Some of them lead general techniques for compositional generalization. For example, entropy regularization (Li et al., 2019) is proposed to avoid redundant dependency input, and it
idea of the approach in this paper.
Compositional generalization has applications in various fields such as question answering (An- dreas al.) 20166] Hudson & Manning 2019 Keysers et al. 2020), counting (Rodriguez & Wiles 1998 Weiss et al. (2088), systematic behaviour (Wong & Wang 2007 Brakel & Frank (209)). and hierarchical structure (Linzen al., 2016). Another related work independent disentangled
representation (Higgins et al. 2017 Locatello et al., (219), but they do not address compositional
generalization. Compositionality also helpful for reasoning (Talmor et al. 2020) and continual
learning (Jin et al. 2020 Li al. 2020).
Grounded SCAN (gSCAN) dataset was proposed to introduce environment and grounding to agent instruction learning with compositional generalization (Ruis et al. [2020). It has command sentence as input and sequence of actions as output. However, the input command does not tell the specific way act, but agent needs to understand the environment and take corresponding actions. This also avoids direct mapping between input words and output actions.
Different approaches have been proposed to address this problem. As compositional generalization requires prior knowledge for distribution change, these approaches correspond to different ways to provide the prior knowledge. Andreas (2019) uses linguistic knowledge to augment training data.
2

---

Under review as a conference paper at ICLR 2021
Kuo al. (2020) uses external syntactic parser and WordNet. In this paper, apply the prior knowledge for the interactions of agent and the environment.
3 PROBLEM DESCRIPTION
gSCAN dataset contains episodes of commands and actions, and each episode has an input and an output. The input includes command and environment. A command is sequence of words. An environment contains an agent and set of objects. An agent has its position and initial direction. An object has its position and attributes of color, shape and size. The output contains sequence of actions (Figure
We hope to extract prior knowledge from environment interactions. We first notice that agent should know whether it is going to change position. This means we can break an episode to a sequence of steps, where each step corresponds
action for position change. An agent should also know the change of directions between steps. So we separate the direction change and the manner of action in step. Therefore, we have three output for each step: direction, action and manner. We further convert the environment to be agent centered, and rotate the environment to make agent facing forward. We also assume automatic collision prevention. If the agent tries to push or pull an object to collide with other objects or boarder, then it stops. This makes us focus on addressing grounded compositional generalization problem.
In summary, when we consider agent interaction with the environment, we can convert the problem a set of step-wise label prediction problems with multiple outputs. Input contains command, environment, state, and output contains direction, action and manner.
APPROACH
In this section, we describe the approach for grounded compositional generalization. As we use en- tropy regularization in different modules, we first introduce it, then move the model architecture.
4.1 ENTROPY REGULARIZATION
The difficulty of compositional generalization is that there might be incorrect dependency between input and output components. For example, when "red" and "square" do not appear together in any training sample, a model might learn that square is not red. However, this causes errors for compositional generalization in test. To avoid such case, we hope the representation of shape not influenced by input information of color.
Entropy regularization (Li al. 2019 Li & Eisner 2019) aims reducing entropy of represen- tation to avoid dependency redundant input components. Given an representation the La norm and add normal noise to each element the representation. This decreases the channel capacity, so that the entropy for the representation reduces. We then feed the noised representation to the next layer, and add the norm to loss function.
compute
Cariginal + \L)
EntReg(x) + aN(0,)
where a is weight of noise, positive for training and zero for inference.
A representation can be fed to multiple networks, requiring different regularization for each input node. So we design entropy regularization layer, where we achieve non-linear mapping by expand- ing each node z; to a vector hę E RR with ReLU activation, and maps it back to node with linear activation, then apply entropy regularization on y to get y'. The input x and output ERL(x) v/ have the same size.
h4 (I.
(T., H),
= f1,5 (h,, 1),
=
We write ff( K) for feed-for ferw-forward neural network with æ as input and K as output size. We use EntReg for word embeddings and ERI for environment inputs.
3

---

Under review as a conference paper at ICLR 2021
q. query
w": score + a:: attention map
t: type rep.
r: attended value rep.
command
0: value rep.
(a) Command module (CM).
I......V
CM,
wf: score
: score sum a attention map
2.0: attributes
pt target position
rP: positions
(b) Grounding module (GM).
CM
p: target position
z: concatenated rep. - j: prediction
state
(c) Prediction module (PM).
Figure 2: Flow charts for core part each module the proposed approach. Command module and grounding module based attention mechanism the main architecture, and prediction module concatenates inputs and use feed- corwafr network. More details can be found Algorithm
4.2 MODEL ARCHITECTURE
Input contains one-hot representation of a command x with sentence length n and vocabulary size Vc. and sequence of m objects. We denote object attributes x" with vector size V and object positions x! with vector size VP. We also has a binary state 8 E (0. }} indicating whether is the first step of episode. Output has three types. Direction yd has Ctd classes, action y has C classes, and manner y" has C classes.
x- if.,.... " E RR x² = xi. =f........ Fm
CRRV"M xm
x = ..... s E R VPmm
The model has the following modules, as summarized in Figure and Algorithm 1
Command module (CM) This module takes command xf as input and returns representation with size K, CM(x", K). is expected to be the embedding of certain type of keyword, e.g. action, color, etc. To enable compositional generalization, we hope to separate representations of types (e.g. color) and values (e.g. red). We use types for attention maps and values for attended values in attention mechanism. For new combination of values, the types are still recognizable, so that attention maps are correct and the corresponding values are extracted (I.i et al. 2019). We design that each word of in command x has two embeddings for type t and value 2/ with corre- sponding embedding matrices, E, E & E, E R where ky and k, are embedding sizes, respectively. We then apply entropy regularization both f and to reduce redundant dependency.
== EntReg (fa=") E ke
t = EntReg (Faa") E An
We use attention mechanism with a query q E R as learnable parameters, keys t, values v, and temperature 7 E R. We compute score w and attention map a.. The attended value is fed to feed-forward network with output size K.
wh qt E" a Softmax(u"s (wo (////////)//)) R."
va E R
(u, K) R
Grounding module (GM) This module finds target in the environment according to the command.
We use N command modules for queries. For each query r., have an dedicated entropy regu- larization layer on attributes 2 because different queries correspond to different type of attributes, and other attributes would be redundant. We then compute score w.. For i 1,..., N.
r CM(2 Vo) V
ERL(x.) IR. VV= Vm
w r: E m...
4

---

Under review as a conference paper at ICLR 2021
The proposed approach. Input includes command x², state 8, object attributes x and object positions wP. K an embedding size. Please see Section for more information.
Algorithm
Command module
Grounding module Input: x², xx" x
Prediction module
Input: x
Input: x², ,,", s. K
Output: embedding
Output: p: target position
Output: ÿ: prediction
I: EntReg( (E,x")
1: for i -I,..., N do
1: p- = GM(x x², x²)
2: EntReg( E,,x")
2: r; CM Vo)
for each output type i do
3: Nº qt
3: x, ERL(x*)
3: hy = CM(2,, K)
4: a Softmax (wo /T9)
4: W riao
for each node j do
5: u = va
5: end for
5:
Pt. ERL(p)
6: ff K)
6: T Σ w
6:
Pi.j [KT P ST|7
7: 0 = Softmax (attmax(w///a0 /T0)
7:
l = H (26.j.
8: p x a
8:
end for
j/i Softmax(4:)
10: end for
The scores are added as wº. Attention map is computed by Softmax with temperature 7 E IR and is applied to get the attended object position
w
RT
a = Softmax (w (19 E R",
p x a E RV..
t=1
Prediction module (PM) Prediction module takes command, environment and state as input, and outputs prediction. We have three separate prediction modules for direction, action and manner, respectively. Modules correspond to different keywords but share the same grounded target and state. So for each prediction module, use one command module extract a keyword is with size K for the prediction. We also have environment module for target object position p. and all the prediction modules share as an input. There is a input of state 8, and this is also shared input for each prediction module.
We build a dedicated feed-forwart neural network from input to each output node without weight sharing. In each network of output type i and node j. we use entropy regularization layer for target position p. This is because different output nodes may need to be computed from different input components, and other components node to input nodes (more discussion in Section(6.11 Then all the inputs are concatenated to form a vector z with size T. K VP
redundant,
hope reduce dependency of each output
h; = CM(2", KKERK,
P/. = ERL(p) e RV",
z = IK7
aT Pij. s
ITIT E RR.
We then feed it to another feed-forward network to get output node 4.jj They are concatenated to form a logit Li, and we use Softmax to output ii- C2 is the number of classes for the output type.
lij nh, (z..) 1) R,
l.......,,4.c RC, t Softmax(44 RC
We use cross entropy and the norms for entropy regularization with weight training objective.
EXPERIMENTS
We use all eight tasks in gSCAN for experiment. Figure I shows examples of two gSCAN com- mands in different environments. We run experiments to show that the designed compositional generalization problems are correctly addressed. The tasks are summarized follows.
A: Random from the same distribution as training data. does not contain compositional gener- alization, and is used to compare with other tasks as reference.
B: Yellow squares has a new way of referring to a type of object, e.g.. "small square" in training, but "yellow square" only in test.
5

---

Under review as a conference paper at ICLR 2021
[ Seq2seq
GECA
Semantti
Proposed
100
Accuracy (%)
50
A: Random
B: Yellow squares
C: Red squares
D: Novel direction
E: Relativity
F: Class inference
H: Adverb to verh
G: Adverb k:
Figure 3: Bar graph of accuracy results. Seq2seq and GECA are from Ruis et al. (020) Semantic from Kuo al. (2020). The results shows that the proposed approach effectively address most of the tasks, and task G is analyzed in discussion section.
C: Red squares has combination of attribute and object, e.g., "red" and "square" appears separately in training, but "red square" appears only in test.
D: Novel direction has target object in new direction, e.g., down left direction only in test.
E: Relativity newly has objects of specific sizes as "small" or "large". e.g., refer size 2 as "large" (a sample does not contain larger sizes) only in test.
F: Class inference requires inferring object properties, e.g., the object size decides the number of pull or push actions to take.
G: Adverbs requires learning adverbs, e.g., "cautiously", with one few samples.
H: Adverb to verb has new combination of adverb and verb, e.g., "walking" and "while spinning" both appear in training, but "walking while spinning" appears only in test.
The dataset statistics and more details of settings in model design and training can be found in Appendix[A A We evaluate accuracy on episodes. An episode is correct when all the steps are correct. Following Ruis et al. (2020), we run each experiment three times, and report mean and standard deviation (note that their sum may be above 100%).
The results are shown in Figure 3 and more details in Table I The proposed approach has signif- icantly higher accuracy than the baseline methods most of the tasks. It does not work in Exper- iment G, and we discuss about it in Section 6..3 This indicates that the approach can effectively address most of the designed compositional generalization problems. Especially in Experiment D: Novel direction and Experiment H: Adverb to verb, there are more than 94% and 78% absolute im- provements, respectively. Experiment requires recombining different directions, and experiment H requires recombining verbs and adverbs. These tasks are standard compositional generalization problems with recombination seen components.
DISCUSSIONS
In this section, we discuss and analyze more details to have better understanding of the approach. We will first run ablation study find what factors enable the improvement. We then analyze the grounding ability the proposed approach. We also discuss the one-shot learning problem. We further apply the approach on target length task.

---

Under review as a conference paper at ICLR 2021
Table Details of accuracy results (%) for experiments. We report mean and standard deviations.
Seq2seq
GECA
Semantic
Proposed
A: Random
97.69 0.22
87.60 + 1.19
97.32
100.00 0.00
B: Yellow squares
54.96 + 39.39
95.35
99.35 + 0.84
C: Red squares
23.51 21.82
78.77 + 6.63
80.16
98.69 + 1.85
D: Novel direction
0.00 + 0.00
0.00 + 0.00
5.73
100.00 + 0.00
E: Relativity
35.02 2.35
33.19 + 3.69
75.19
99.72 + 0.38
F: Class inference
92.52 + 6.75
85.99 - 0.85
98.63
100.00 = 0.00
G: Adverb & =
0.00 + 0.00
0.00 = 0.00
11.94
0.00 - 0.00
H: Adverb to verb
22.70 = 4.59
11.83 + 0.31
21.95
100.00 + 0.00
6.1 ABLATION STUDY
We conduct ablation study to understand what factors of the algorithm important. One change of the approach entropy regularization (EntReg). We also have structure change to apply the regularization (ERL). We have element-wise prediction for each output node (NodeOut). We design experiments by removing each of these factors.
The results in Table show that the ablation experiments have lower accuracy than the proposed approach. This means all the changes are important for the approach. The reduction is especially significant for Experiment D: Novel direction. In this task, each value of the output depends on different input components, because different output values have priorities in the setting. When target appears on back, back right or back left, the agent turns around. When appears on right, the agent turns right. Setting agent faces horizontal positive direction, this means turning around node depends only on one coordinate (horizontal negative), but turn right node depends on two coordinates (horizontal is zero and vertical is negative). This is different from previous composi- tional generalization where all values of an output component depend on the same input component. Though this particular case may be solved in other ways, it is general that different output values may depend on different input components, and the proposed method is able to address the problem.
Table 2: Accuracy results (%) for ablation study. The ablation experiments have lower accuracy than the proposed approach. This means all the changes are important for the approach.
Proposed
No EntReg
No ERL
No NodeOut
A: Random
100.00 + 0.00
100.00 = 0.00
100.00 + 0.00
95.51 6.35
B: Yellow squares
99.35 + 0.84
94.60 4.37
74.50 + 0.57
95.73 = 5.20
C: Red squares
98.69 + 1.85
08.27:1.3
94.72 + 2.96
96.40 + 5.09
D: Novel direction
100.00 0.00
92.44
13.29 + 14.63
0.00 = 0.00
E: Relativity
99.72 + 0.38
99.69 0.43
96.57 + 3.17
95.98 5.67
F: Class inference
100.00 0.00
100.00 1.000
99.99 + 0.01
95.76 1 6.00
G: Adverb & =
0.00 + 0.00
0.00 0.00
0.00 + 0.00
0.00 + 0.00
H: Adverb to verb
100.00 + 0.00
100.00
100.00
0.00
96.67 = 4.71
6.2 GROUNDING
We run visualization to show that the approach works in the expected way. The grounding finds correct object according to descriptions in command. A common way for grounding, which is also used the proposed approach, to use attention. So we evaluate whether the correct target object is attended.
The results in Table 3 show that the proposed approach has good grounding ability in all the tasks. Note that it also works Experiment G, where the final prediction is not accurate. This means the errors in Experiment G are not caused by grounding, and the proposed approach addresses the designed grounding problems in all the tasks of gSCNN
7

---

Under review as a conference paper at ICLR 2021
Table Grounding accuracy results (%) for the proposed method. Grounding works well in all the tasks.
A: Random
100.00 + 0.00
E: Relativity
100.00 + 0.00
B: Yellow squares
100.00 + 0.00
F: Class inference
100.00 + 0.00
C: Red squares
98.91 + 1.54
98.68 + 0.88 G: Adverb A =
D: Novel direction
100.00 = 0.00
H: Adverb to verb
100.00 = 0.00
6.3 ONE-SHOT LEARNING
The gSCAN experiment result shows that Experiment G: Adverb does not work well for methods without external lexical information. This experiment was designed as few-shot or one-shot (A: 1) learning task, however it is different from standard settings. Conventionally, one-shot learning has only one sample for a class, but the frequency or weight of the sample is not fixed. For example, in Lake & Baroni (2018), the one-shot sample is repeated to be 10% of the whole dataset. Changing the weight one-shot samples is also common human learning, as we focus on one special sample and learn from it. However, in experiment G, the frequency of the one-shot sample is one, while there are more 300,000 other training samples.
This not only extremely lower the contribution of the sample to prediction loss, but also decrease the chance to be included in a mini-batch for training. To some extent, this requires the algorithm to partially address catastrophic forgetting (Kirkpatrick et al. 2017). This is because when the last mini-batcher do not contain the sample, the model should not forget it. So this experiment contains more challenges than core compositional generalization, and they beyond the scope of this paper.
6.4 TARGET LENGTII TASK
Ruis et al. (2020) also contains separate dataset for target length task. It requires the model to perform on the target action sequence longer than those training. This not the focus of this
paper, but we run experiments for references.
The result in Table 4 shows that, for the proposed approach, the accuracy does not drastically de- crease when the evaluation target length increases. This might be because of using step-wise predic- tion, which is less influenced by the length of an episode. The result indicates an additional benefit of using environment interactions.
Table 4: Accuracy results (%) for target length experiments. Seq2seq is from Ruis al. 2020
Semantic is from Kuo et al. (2020). The proposed approach avoids drastic decrease when the length increases.
Target length
Seq2seq
Semantic
Proposed
15
94.98 + 0.12
93.43
94.45 + 4.56
16
19.32 = 0.02
90.88
96.28 + 3.36
17
1.71 0.38
87.92
96.04 + 3.25
218
< 1.00
56.50
97.64 + 2.06
7 CONCLUSION
We propose an approach to address compositional generalization in grounded agent instruction learning. We use interactions between agent and the environment to define output components, and entropy regularization to reduce redundant dependency on input. We achieve significant im- provements most of gSCAN tasks, and the high accuracy indicates that the proposed approach addresses the designed grounded compositional generalization problems in these tasks. In ablation study, we show the effectiveness of entropy regularization and other changes, and look into differ- ent aspects of the approach. We hope this work will be a step towards addressing compositional generalization grounded language learning and general artificial intelligence.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: system for large- scale machine learning. In 12th {USENIX} Symposium on Operating Systems Design and Imple- mentation ((OSDI) 16), pp. 265-283, 2016.
Jacob Andreas. Good-enough compositional data augmentation. arXiv preprint arXiv:1904.09555, 2019.
Jacob Andreas, Mareus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Yoshua Bengio. The consciousness prior. arXiv preprint 1rXiv:1709.0000 08568, 2017.
Philémon Brakel and Stefan Frank. Strong systematicity in sentence processing by simple recurrent networks. In 3/th Annual Conference of the Cognitive Science Society (COGSCI-2009), pp- 1599-1604. Cognitive Science Society, 2009.
Noam Chomsky. Syntactic structures. Walter de Gruyter, 1957.
Jonathan Gordon, David Lopez-Paz, Marco Baroni, and Diane Bouchacourt. Permutation equivari- ant models for compositional generalization language. In International Conference on Learning Representations, 2020. URL
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 3-vae: Learning basic visual concepts with a con- strained variational framework. In International Conference on Learning Representations (ICLR), 2017.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6700-6709, 2019.
Xisen Jin, Junyi Du, and Xiang Ren. Visually grounded continual learning of compositional seman- tics. arXiv preprint 20XXv:0005.007855 2020.
Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Marc van Zee, and Olivier Bousquet. Measuring compositional generalization: com- prehensive method on realistic data. In International Conference Learning Representations, 2020. PRLLitttss:/opeereeee
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6900 2014.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-335 2017.
et al. Overcom-
Yen-Ling Kuo, Boris Katz, and Andrei Barbu. Compositional networks enable systematic general- ization for grounded language understanding. arXiv preprint arXiv:2008.02742. 2020.
Brenden Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to-sequeeee recurrent networks. In International Conference Machine Learning, pp. 28722873322 2873-2882, 2018.
Brenden M Lake. Compositional generalization through meta seuuence-to-sqqeeee learning. In Advances in Neural Information Processing Systems, pp. 9788-9798, 2019.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning- nature, 521(5553) 2015.

---

Under review as a conference paper at ICLR 2021
Xiang Lisa Li and Jason Eisner. Specializing word embeddings (for parsing) by information bottleneck. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and 9th International Joint Conference on Natural Language Processing, pp. 2744-2754, Hong Kong, November 2019. URL httt:httssss jhu htlccs...u..uu//a.e jaapsa/apnssss//////ppppp Best Paper Award.
2019
Yuanpeng Li. Liang Zhao, Jianyu Wang, and Joel Hestness. Compositional generalization for prim- itive substitutions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNILP), pp. 4284 4293, 2019.
Yuanpeng Li, Liang Zhao, Kenneth Church, and Mohamed Elhoseiny. Compositional continual language learning. In International Conference on Learning Representations, 2020.
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. Assessing the ability of Istms to learn syntax- sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521- 535, 2016.
Qian Liu, Shengnan An, Jian-Guang Lou, Bei Chen, Zeqi Lin, Yan Gao, Bin Zhou, Nanning Zheng, and Dongmei Zhang. Compositional generalization by learning analytical expressions. arXiv preprint arXiv:2006. 20:::20000.00.7, 2020.
Francesco Locatello, Stefan Bauer, Mario Lucie, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In International Conference Machine Learning, pp. 4114- 4124, 2019.
Marvin Minsky. Society of mind. Simon and Schuster, 1986.
Paul Rodriguez and Janet Wiles. Recurrent neural networks can learn to implement symbol-sensitive counting. In Advances in Neural Information Processing Systems, pp. 87-93, 1998.
Laura Ruis, Jacob Andreas, Marco Baroni, Diane Bouchacourt, and Brenden M Lake. A benchmark for systematic generalization in grounded language understanding. arXiv preprint arXiv:2003.05 161, 2020.
Jake Russin, Jason Jo, and Randall C O'Reilly. Compositional generalization in a deep seq2seq model by separating syntax and semantics. arXiv preprint arXiv:1904.00 09708, 2019.
Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Alon Talmor, Oyvind Tafjord, Peter Clark, Yoav Goldberg, and Jonathan Berant. Teach- ing pre-trained models to systematically reason over implicit knowledge. arXiv preprint arXiv:2006.06609 2020.
Gail Weiss, Yoav Goldberg, and Eran Yahav. On the practical computational power of finite preci- sion mns for language recognition. arXiv preprint arXiv:1805. 185.00080000 2018.
Francis CK Wong and William SY Wang. Generalisation towards combinatorial productivity lan- guage acquisition by simple recurrent networks. In 2007 International Conference on Integration of Knowledge Intensive Multi-Agent Systems, pp. 139-144. IEEE, 2007.
10

---

Under review as a conference paper at ICLR 2021
A EXPERIMENT SETTINGS
We implement the model as follows. We use one layer feed-forward neural network for ff and fr with = 4. f has ReLU activation and fr has linear activation. In command module, we use k. 32 and k. 32. fīg is grounding module, layer feed-forward network. The first and second layers have 32 hidden nodes with Relu activation. The third layer has one output node with linear activation. For target length task, the setting the same except N = 8 in grounding module.
layer feed-forward network with linear activation. 7= 1. In K ft., is three
N = 3 and 7 0.1. In prediction module,
In training,
À and 10,000 training steps. Each mini-batch has 256 samples selected from training data uniformly at random with replacement. Gradient is clip by norm of We use initial learning rate 0.001, which exponentially decays every 100 steps by factor of 0.96. TensorFlow (Abadi et al. 2016) is used for implementation.
0.1. We use Adam optimizer (Kingma & Ba 2014) with
The sample size of gSCAN dataset is summarized in Table
Table 5: Sample size of gSCAN dataset.
Task
Samples
Train
367,933
A: Random
19,282
B: Yellow squares
18,718
C: Red squares
37,436
D: Novel direction
88,642
Relativity
16,808
F: Class inference
11,460
G: Adverb k
112,880
H: Adverb verb
38,582
B FEW-SHOT LEARNING
The results of other few-shot learning settings in Experiment G summarized in Table 6 More discussions for the task can be found in Section 6.3
Table 6: Accuracy results (%) for few-shot learning experiments. Seq2seq is from Ruis et al. 2020
Semantic is from Kuo et al. (2020).
Seq2seq
Semantic
Proposed
Adverb = 0.00 = 0.00
11.94
0.00 + 0.00
Adverb k- 0.47 + 0.14
10.17
0.00 + 0.00
Adverb A: = 10
2.04 + 0.95
33.28
0.00 + 0.00
Adverb A 15
40.78
0.00 + 0.00
Adverb & 50
4.63 + 2.08
67.44 + 28.27
11


Reviews:
[{"review_id": "wlufE3AkFG", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new regularization method that constrains the mapping between the inputs and output spaces for achieving compositional generalization in simple grounded environments like gSCAN. The problem is interesting and important and the paper is corroborated by good experiments with 25% accuracy increase and also generalization to longer commands. However, the paper has  clarity issues with descriptions that are sometimes vague or not precise enough and quite frequent language mistakes. It also doesn't discuss almost at all existing works and mention the only very briefly, making it harder to judge the strength of the new approach as there's not enough context.\n\nIn addition, the idea presented feels somewhat too specific to the particular gSCAN task. For instance, it considers particularly disentangling the representation of each step to direction, action and manner components. I would hope that a general approach will discover a disentangled representation that allows compositional generalization on its own, rather than being hand-engineered for the particular dataset, especially given its relative simplicity. \n\nFor the entropy regularization idea, while it may allow for compositional generalization, it may reduce the model ability to capture trends in the training data, and so it may produce too \"extreme\" representations that can’t account for correlations within the data, and therefore I suspect it may not work well for more complex problems beyond gSCAN.\n\nComments and questions\n- The related work sections gives background about the subject and used dataset but has only one short like about competing approaches. It will be good to move the general text from the related work section to the introduction section and instead add a bit more detailed description about prior approaches for the problem and how they differ from the new method.\n- The experiments section doesn’t provide any details about the baselines either, so overall a more detailed comparison to existing approaches or putting the paper in the context of prior work is really missing.\n- Page 2: \"Another related work is independent disentangled representation (Higgins et al., 2017; Locatello et al., 2019), but they do not address compositional generalization.\" -> Why aren’t they? The main advantage of disentangled representation is their ability to generalize to combinations of properties outside the training distribution. For instance, if you encode separate features for color and shape, you may learn to generalize to any combination of them even if the training data didn’t cover all of them, since your representation inherent compositional structure that separates out these two properties may prevent capturing spurious correlations of the two properties and encourage generalization to combinations of them.\n- The description of the task is not clear enough. A more formal/mathematical definition of the task will be useful, especially if letters are presented for e.g. the input command, output sequence etc it will be easier to refer back to them later in the paper. Also further description/ a couple of examples on what the actions and the manners are will be useful for those not familiar enough with gSCAN.\n- \"We also assume automatic collision prevention… This makes us focus on addressing grounded compositional generalization problem.\" - is it fair to assume that or does it simplify the problem? Are alternative approaches assume that? How is it useful?\n- Page 3: \"For example, when \"red\" and \"square\" do not appear together in any training sample, a model might learn that square is not red. However, this causes errors for compositional generalization in test. To avoid such case...\" - Do we want to avoid such cases? A model that will learns that there is no correlation at all between pairs of properties will not work in the real world. Rather than avoiding learning the correlations it will be useful if the model will be still able to learn them but at the same time allocate some smaller probability for the case of combinations that are less common.\n- The description of entropy regularization isn’t completely cleared to me. What are x_i and y_i. Does each y_i depend on all x or only xi?\n- It sounds like basically entropy regularization reduces the capacity of the representation by adding noise + l2 loss on the activations. This sounds like a quite general regularization technique but is unclear to me why this encourages compositionality in particular. Further explanation of that will be helpful.\n- A small comment, since the generalization to length work well but one-shot learning isn't and is still an open problem, it may make more sense to put the length subsection first and then the one-shot learning one.\n\nSome Typos\n- What are the components in output -> in the output\n- other changes with ablation study -> with an ablation study\n- to understand command and the environment -> the command and\n- redundant dependency on input -> on the input\n- Grounded SCAN (gSCAN) dataset -> the Grounded SCAN dataset\n- but agent needs -> but the agent need or but agents need \n- of agent and the environment -> of the agent\n- to change position -> to change its position\n- on addressing grounded compositional generalization problem -> addressing the problem\n- Input contains command -> the input \n- Output contains -> the output\n- Information of color -> of the color\n- design entropy regularization layer -> a/the layer\n- finds correct object -> find the correct object\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "gS69bTanOC4", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper aims to improve compositional generalization of grounded language learning methods. As far as I understood, this is achieved by (a) crafting a task-specific architecture with noise addition in certain spots and (b) changing the output format. Unfortunately, I did not understand many details that would be crucial to properly review this paper. I found the paper to be very unclearly written, see details below.\n\n1. As far as I understand the key contribution of the paper is supposed to be using “interactions between the agent and the environment to find components in the output”. The place in the paper that seems to explain what this is about is Section 3. But since this section does not explain what the original action space in the task is and does not show example trajectories, I found it impossible to understand what the \"components in the output\" and \"interactions between the agent and the environment\" are. \n2. The entropy regularization is also explained poorly. I understand what EntReg is. But to explain ERL the paper refers to “nodes x_i”. I do not think the paper clearly explains what these “nodes” are, and how this is related to the possibility that “a representation can be fed to multiple networks”.\n3. I struggled to understand the specific architecture that is explained in Algorithm 1, but one thing that struck me is that unlike the baselines that the paper compares to, it uses an object-centric representation, whereas the baselines are taking images (albeit symbolic) as inputs. This distinction alone in principle might explain the difference in the results. \n4. To make Algorithm 1 more understandable it could be helpful to visualize what the N command modules are and what they are supposed to do. A figure with an example that explains the motivation for the architecture would be of great help. \n\nUnfortunately, this concludes my review: I was not able to understand enough in order to make more substantial comments. I would encourage the authors to improve the paper in the following aspects: \n- clear explanation of the task, including input and output format\n- clear presentation of the model, including a visual motivating example\n- clear comparison to the baselines that takes into account the difference in input representations. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "HbyH9KWl65U", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper proposes a new model for the gSCAN dataset (Ruis et al. 2020) which is a synthetically-generated dataset that challenges models to generalize to new compositions of attributes and objects in an instruction. The paper proposes to use \"entropy regularization\" as a way to enforce that spurious correlations between input tokens and output actions are not learned. The proposed model shows promising results on the gSCAN benchmark, achieving nearly 100% accuracy on all but one task.\n\n########\n\nReasons for score: \n\nWhile the paper achieves impressive performance on the synthetic gSCAN dataset, the paper's framing of the problem, discussion of related work, and description of the model are vague and confusing. As a reviewer, I am left with many questions and confusions about the setup and scope of this paper, and my review depends on clarifications from the authors during the discussion period.\n\n########\n\nStrengths:\n1. The paper achieves impressive performance on the gSCAN dataset across all but one task, and describes intuitions on why the remaining task failed.\n1. The paper also achieves impressive results on the target length task when compared to existing systems.\n1. The proposed model is relatively simple.\n\n########\n\nWeaknesses:\n\n1. My main concern about the paper is its framing of the problem and use of vague terms, especially in the introduction and abstract. See below for more suggestions as well. The paper should be more clear about the scope of compositional generalization it is targeting, especially early in the paper. Specifically, it seems to be targeting novel compositions of attributes and objects in an input instruction. However, there are other dimensions of compositional generalization which the paper does not cover. For example, generalization to new compositions of the environment itself (i.e., novel combinations of the objects, attributes, and spatial relations, as evaluated in datasets like ALFRED (Shridhar et la. 2020), Room to Room (Anderson et al. 2018)), or novel deep compositional structures in the output space (as in semantic parsing, e.g., Finegan-Dollak et al. 2018, in comparison to shallow output space compositions such as new manner/direction combinations). This paper should clarify the dimensions along which it is evaluating compositional generalization, and contrast that with other forms of compositional generalization. Some other related work which evaluates compositional generalization and are missed include VQA with Changing Priors (Agrawal et al. 2017) and CLEVR CoGenT (Johnson et al. 2017). Especially for CLEVR CoGenT, there are several systems which achieve near perfect accuracy on that compositional generalization task, such as NS-VQA (Yi et al. 2019), which are not compared against.\n1. The evaluation is only performed on a single synthetic dataset. While it performs well on this benchmark, evaluation on other grounded instruction following tasks, such as some of the examples above, or even on non-instruction tasks such as image question answering, would make the results more convincing. \n\n\n########\n\nMajor questions for the authors about the paper setup :\n\n1. I am confused about where \"entropy regularization\" comes into play in the model. As it is written it seems to be used in two places: (1) in the actual forward-pass, the only difference is that it adds noise to an input (and the noise is zero at evaluation time), and (2) during optimization, it also minimizes the L2 norm of these layers' activations. Is this correct? \n1. In the ablation setting, it's unclear to me what exactly is being ablated. If EntReg appears both in optimization as L2 regularization and in the forward pass as noise, then are these ablated independently? What is ERL replaced with when ablated? And what exactly is NodeOut -- it's not defined anywhere earlier in the paper, if it is the most critical component of the approach (as indicated by Table 2), this is concerning.\n\nOther clarification questions for the authors:\n1. What is the intuition of the EntReg component actually solving the problem of learning spurious correlations? How does it actually limit the correlations (if this is the intent) between components if it is not considering some relationship between two components?\n1. What does it mean for compositional generalization to require prior knowledge on distribution change? \n1. Am I correct in understanding that the command module will always result in the same output while the agent is acting in the environment? Nothing about the command module seems to change as it is not conditioned on an environment observation.\n1. How does the model know the types and values of the words in the command? \n1. How do \"yellow squares\" and \"red squares\" actually differ in the compositionality they require?\n1. Does \"relatively\" test new attributes that were never seen or referred to during training? Or just new ways of referring to these attributes?\n1. What's an example of \"class inference\"?\n1. How is attention evaluated? Automatically?\n\n########\n\nSuggestions to clarify the presentation of the paper and the approach:\n\n1. The related work should contrast the proposed approach or evaluation setup with the cited works. E.g., how does the proposed approach differ from the approaches used on SCAN?\n1. \"entropy regularization\" is a more general term than it is used in this paper, so I would suggest renaming it. E.g., it is often used to prevent entropy collapse in reinforcement learning by adding an additional term in the optimization that optimizes for higher entropy output distributions. \n1. I am confused about why the term \"entropy\" is used here, because I don't see the relationship between the function EntReg and entropy (i.e., H(x) where x is a distribution). EntReg here just seems to add some noise to an input x, which is not necessarily a distribution. \n1. Several terms are vague/undefined. The term \"component\" is vague without a formal definition of the task being studied. Later on, I am also confused on what \"a change of direction between steps\" means, and why there is a direction, action, and manner. The example in Figure 1 doesn't show the output sequences or examples of different manners, and I only have an idea of what \"manner\" means when I read the gSCAN paper or later on in this paper that manner might include spinning, being slow/fast, etc. So, I would suggest, e.g., showing the actual possible space of outputs.\n1. The proposed approach should be described more concretely in the introduction before the results of it are described.\n1. Several terms used in the model description are vague and should be defined: what is a \"node\" x_i (Section 4.1)? What is the difference between ERL and EntReg (my reading is that ERL is an MLP with EntReg between the layers)? What is a \"keyword\" (as opposed to just a \"word\")?\n\nSuggestions on the description of model / results:\n1. The model seems to be described formally three times: in Algorithm 1, Figure 2, and Section 4.2. Its description could be more concise (e.g., by removing Algorithm 1).\n1. The experiments section should contrast the proposed approach with the baselines/other systems in a meaningful way. \n1. I know this is coming from the gSCAN dataset, but the names of the 8 tasks are very confusing. If it's possible to augment these with a short keyword description that accurately describes the forms of generalization they are evaluating, this would make things a lot easier to read. Particularly, upon reading the results table, the \"yellow squares\" and \"red squares\" settings aren't meaningfully different, and I have no idea what they are testing differently.\n1. If Figure 3 is just a bar graph version of Table 1, it could probably be removed as it is doesn't add any information over Table 1. It's missing the numbers in the bars, the x-axis is hard to read, and it's unclear to me whether this is graphing the mean, max, etc.\n1. The meaning of $k$ should be defined before Figure 3 and Table 1 are presented.", "extra_scores": {}, "extra_text": {}}, {"review_id": "6KGX_TYymlX", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThis paper tries to address a very important problem, compositional generalization in grounded agent instruction learning. It proposes to use interactions between agent and the environment to define output components, and entropy regularization to reduce redundant dependency on input. It shows significant improvements in most of gSCAN tasks. The paper also has an ablation study that investigates the effectiveness of entropy regularization and other factors.\n\nStrengths\n\nGrounded compositional generalization is a key challenge to the AI community. This paper proposes entropy regularization and shows some good results compared with Seq2seq and GECA are from Ruis et al. (2020). Semantic is from Kuo et al. (2020). \n\nWeaknesses\n\nThe paper is very hard to read. Notations is Section 4 is very confusing. \n\n\nIt is not clear there is any novelty in the model architecture, command, grounding and prediction modules. If not, the paper should clearly cite relevant papers.\n\n\nThe paper claims to be \"the first work to enable accurate compositional generalization in grounded instruction learning problem, serving for analyzing and understanding the mechanism\". It is not clear this is true given Ruis et al. (2020). Semantic is from Kuo et al. (2020).\n\n\nThe paper fails to make a connection to the two papers. It only says \"we apply the prior knowledge for the interactions of agent and the environment.\".\n\n\nSome qualitative results in the main text or appendix would have helped to illustrate the proposed approach and offer insights on why the method is effective.\n\n\nDecision\n\nSince the paper is poorly written, not put in the proper context of the related papers, and the novelty seems to be limited to entropy regularization, my decision is rejection. I would be open to revise my decision if the authors make clear of their methods and contributions.\n\n=====POST-REBUTTAL COMMENTS========\n\nThe authors did not make a good effort to address my comments and failed to update the paper. Therefore, I maintain my original decision.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an approach to training language instruction following agents that aims to improve their compositional generalization., by means of an entropy regularization method to reduce redundant dependency on input.

All four expert reviewers agreed that the paper is not ready for publication in its current form. Of biggest concern is the fact that the reviewers could not interpret the exposition of the method, so were unable to be sure exactly how the method worked. This can be addressed in a future submission by clearer presentation. 

Another concern was that the authors only consider a single benchmark, and fail to situate the work relative to other grounded language learning tasks and datasets. Thus, reviewers were concerned about the generality of the method, and suspected it may be too specific to the gSCAN setup. 

That said, the reviewers were all impressed by the strong results on the gSCAN benchmark. It strikes me that there is some interesting insight here that can be derived from this impressive performance, that may also be applicable to other grounded language learning settings. However, to make the paper acceptable for publication the authors must do a much better job of communicating how their method works, what that specific insight is and how it is relevant beyond the gSCAN dataset (ideally via direct experimentation in other settings).
Decision: Reject