Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: 
Summary
---

(motivation)
Consider SCAN, a synthetic task where setences like S1="jump twice and run left" are supposed to be translated into action sequences like A1=JUMP JUMP LTURN RUN. One might replace the word "jump" in S1 with "walk" then translate to get A2=WALK WALK LTURN RUN. If instead S1 is translated into A1 and then the action JUMP is replaced with the action WALK then we should still get the same A2. Such a translation model is equivariant to permutations of "jump" and "walk".

This paper aims to
1) define a general notion of compositionality as equivariance,
2) build a model which is compositional in this general sense, and
3) apply the model to SCAN.

(approach - theory)
This work considers this kind of compositionality as equivariance to group actions. Previous work (Kondor & Trivedi, 2018) viewed convolution as equivariance to actions by translation groups. This work views language compositionality as equivariance to actions by permutation groups applied to a set of similar words (e.g. verbs in SCAN).

(approach - model)
The paper proposes G-Embed, G-RNN, G-Attention, and G-Conv (not new) layers that are equivariant to word permuatations (e.g., switching "jump" and "walk"). It then composes these modules in a fairly standard fashion to build a new G-seq2seq model which is invariant to group actions.

(experiments)
Experiments apply a G-seq2seq model to the SCAN tasks, comparing to strong baselines. G-seq2seq requires slightly more knowledge (a set of related words like verbs) than all the baselines, but less knowledge than Lake 2019.
1. G-seq2seq outperforms all baselines except Lake 2019 (unfair comparison) on basic compositional tasks ("Add Jump" and "Around Right").
2. Like other models, G-seq2seq fails on the "Length" task, though it is still among the best performers.


Strengths
---

The theory of compositionality as invariance to actions by permutation groups is new, interesting, and could turn out to be significant.

The proposed models are also new, interesting, and could be significant.

Experiments on SCAN verify that the proposed models work about as expected, sometimes beating strong baselines in the process.


Weaknesses
---

It's hard to know what the impact of this paper will be because 1) it's unclear whether this model can generalize to more useful domains and 2) the presentation may turn some readers away. While neither of these issues can really be solved, I think paper could be substantially better in both aspects. Corresponding suggestions:
1) How expensive is this? It seems quite expensive because the representation size scales with the number of permutation of the set of words equivariance is with respect to. How will it scale to larger problems in terms of computation/memory costs (especially larger vocab sizes)? What knowledge is required for applying this method to new domains--i.e., how do I choose a set of permutation equivariant verbs in general? More discussion of these issues may help increase the impact of the paper.
2) See next section.


Presentation Weaknesses / Points of Confusion / Missing Details
---

To mimic a typical decoder RNN there should be another input which copies the word \tilde{w}_{t-1} from the previous iteration as input, somehow fused with the attention feature \tilde{a}_t. How does the G-RNN know what the last word it generated was?

The notation $g^{-1} w$ in the first equation of section 4.1: I think $\psi^i$ is supposed to take an integer as input but $g^{-1} w$ is a permutation applied to a function. I'm not sure how to apply permutations to functions like w and it doesn't seem like the output should be an integer in any case so I find this notation confusing.

Taking a step back, I find some of the notation (e.g., previous point) a bit confusing. This makes it hard for me to get the main point. I think the idea is that equivariant models can be achieved by tracking a representation (e.g., via rows of the G-Embed matrix) for (almost?) every member of the acting group.

It may help the presentation to more frequently demonstrate the general concepts with examples, though doing so may be in conflict with the general nature of the paper's theoretical contribution. I'm sure this is a familiar tradeoff, but from my perspective the paper would probably be more impactful if the presentation leaned more on examples.

Equation numbers would be a really great addition. I found it hard to reference some of the material in writing my review.

"and the use of algebraic computation"
* This seems specific to the chosen example whereas the rest of the sentence is trying to be general.


Suggestions
---

* This seems related to [1], which uses group theory to define a notion of disentangled representation similar to compositionality. That may inspire future work and would be useful to mention in the related work.

* Why didn't performance on SCAN get to 100%? It would be useful to spend some time addressing points of failure for the model other than compositionality.

* The G-RNN doesn't have a bias. It's not necessary, but it may be interesting to describe why this design choice was made.


[1]: Higgins, Irina et al. “Towards a Definition of Disentangled Representations.” ArXiv abs/1812.02230 (2018): n. pag. 


Preliminary Evaluation
---

Quality: The theoretical contributions make sense and the experiments show they lead to useful models.
Clarity: The technical parts of the paper are somewhat unclear, but the rest of the paper is well written.
Significance: As discussed in the Weaknesses section this could turn out to be very significant or not significant at all, but that's true for a lot of good research.
Originality: The general notion of equivariant neural networks and good performance on SCAN are novel.

Overall, this is a very clear accept.

Post-Rebuttal Update
---

There was a lot of agreement between reviewers, though we came to slightly different conclusions about ratings. Though there is significant uncertainty about the impact of this work, I still think 8: Accept is the most appropriate rating. Overall, the other reviews and author responses only increased my confidence that this paper should be accepted.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The paper presents an architecture that captures equivariance to certain transformations that happen in text, like synonym words and some simple transformation over word order.  

* General comments: 
  
Increasing compositional generalization using equivariance is a very interesting idea. Sections 1-3 are well written and the solution of modeling the translation function as a G-equivariant function is well motivated. 

Section 4 is far less clear. In its current form, it is very hard to understand the model construction as well as the design choices. This section should be significantly improved in order for me to increase my score. A direct by-product of the confusing writing is that the experiments cannot  be reproduced.

The experiments show improvement in one out of four tasks, where the single phrase “Around right” is held from the training set. There are no examples, not qualitative analysis, no ablation experiments. Overall, more evidence needed to convince that the approach is useful. In addition to deeper error analysis, the authors can hold out other phrases (e.g., “around left”, and many others). 

* Specific comments which I hope the authors address:

1. To the best of my understanding, the authors do not explicitly specify the group G that they want to be invariant to. Is it a product of a few cyclic groups? (a cycle for each set of words that are interchangeable?)

2. The authors suggest using G-convolution, i.e. the group convolution on G. This is in contrast to the (arguably) more popular choice of using linear layers that are G-equivariant (as in, for example,  deep sets (Zaheer et al. 2017), Deep Models of Interactions Across Sets (Hartford et al. 2018),Universal invariant and equivariant graph neural networks (Keriven and Peyré ) and in general convolutional layers for learning images).
I have several questions regarding this choice:
2a. Can the authors discuss the differences/advantages of this approach over the approach mentioned above? It seems like the approach mentioned above will be more efficient (as there is no need to sum over all group elements)
2b. In order to use G-convolution, one has to use functions defined on G. Can the authors explain how they look on the input words as functions on G?
2c. How is the G-Conv actually implemented? 
2d. Can the authors provide some intuition to what this operator does? 

3. Is the whole model G-equivariant? The authors might want to clearly state this. To the best of my understanding, this is the main motivation of this construction.

4. It might be helpful for readers that are not familiar with deep learning for NLP tasks to provide a visualization of the full model (can be added to the appendix)

5. Why are words represented as infinite one-hot sequences? Don’t we assume a finite vocabulary? This is pretty confusing.

6. As a part of the G-RNN the authors apply a G-conv to the state h_{t-1}. What is the dimension of this hidden state? How does G act on it? 

7. Please explicitly state the dimensions of each input/output/parameter in the network (this can be combined with the illustration above illustration)

* Minor comments:

Section 4.1 pointwise activation are in general equivariant only to permutation representations
Page 2 - typo - ‘all short’-> ‘fall short’ 

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This work focuses on learning equivariant representations and functions over input/output words for the purposes of SCAN task. Basically, the focus is on local equivariances (over vocabulary) such that the effect of replacing and action verb like RUN in the input with the verb JUMP causes a similar change in the output. However, effects requiring global equivariances like learning relationship between "twice" and "thrice", or learning relationships between different kinds of conjunctions are not handled in this work. For learning equivariant functions over vocabulary, group convolutions are used at each step over vocabulary items in both the sequence encoder and decoder.  The results on SCAN task are impressive for verb replacement based experiments and improve over other relevant baselines. Also, improvement is shown on another word replacement task ("around right"), which requires learning corresponding substitutions in output based on the word changes in the input. As expected, for experiments that require global equivariances or no equivariance (simple, length), the difference ion performance is not very pronounced over other baselines.
While this paper does show that modelling effects of word substitution can be handled by the locally equivariant functions, it still cannot account for more complex generalization phenomena which are likely to be much more prevalent especially for domains dealing with natural language that are other than SCAN. Therefor, I think the applicability of the proposed equivariant architectures is rather limited if interesting.
review_title: Official Blind Review #1

Meta-review: This paper proposes an equivariant sequence-to-sequence model for dealing with compositionality of language. They show these models are better at SCAN tasks.

Reviewers expressed two major concerns:
1) Limited clarity of section 4 which makes the paper difficult to understand.
2) Whether this could generalize to more complex types of compositionality.

Authors responded by revising Section 4 and answering the question of generalization. While the reviewers are not 100% satisfied, they agree there is enough novel contribution in this paper. 

I thank the authors for submitting and look forward to seeing a clearer revision in the conference.
Decision: Accept (Poster)