Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper focuses on modelling invariances or symmetry between various components for solving tasks via convolutions and weight sharing.
The proposed tasks are toyish in nature although they do give insights into importance of modeling symmetry for better generalization. The first task is a symbol substitution which considers a permutation in source symbols and maps them to either "ABA" or  "ABB" categories i.e binary classification. While this task does require generalizability, it is surprising that the mlp and recurrent net baselines are so much inferior (basically random) to the convolution baseline. While this shows efficacy of modeling symmetry, I'd be curious about performance graphs as the training data increases in size.
The second task is an artificially created task inspired from the SCAN dataset. The task is to translate a verb-number pair into number repetitions of the verb. The encoder decoder network uses convolution in the recurrences to capture the notion of generalizability. The input and output space is very small (10 verbs and 10 numbers) but shows superiority of convolution and weight sharing over other baselines. Curiously, the recurrent baseline seems to perform better than 0% accuracy (if still poorly) on the original SCAN task which is much harder than the proposed task in this paper. Maybe, the number of examples (1000) is too small recurrent networks but this makes me a little surprised. More details about the architecture and training procedure for baselines would be helpful to ensure that the comparison is fair across baselines.
The final task is CFG modeling where convolutions are used to model the forget gate of an LSTM which seems to endow the network with PDA like properties and the convolutions are more effective than baselines at modeling this.

Apart from the concerns related to the results mentioned above, my major concern is that the tasks considered are too simple and at least one complicated large-scale task would have strengthened the paper.
Also, for tasks 2 and 3, the motivation behind using convolutions is not as clean as in task 1. So more analysis and insights into model performance, the weights learned, ablation studies etc. would have helped in understanding how the convolutions are modeling the symmetry. This should be informative and tractable because of simplicity of the tasks involved.

Finally, as mentioned above, I still cannot intuitively understand why convolutions in the forget architecture would learn about symmetry related to structured repetition produced by a CFG. Hence, more analysis or a better motivation would have helped. 
review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: The paper investigates the idea of using symmetry and invariance in symbolic reasoning. In particular, it considers models where modeling symbolic symmetries through parameter-sharing help with generalization. The three tasks considered are: 1) rule learning: performs sequence-classification. 2) composition: performs sequence-to-sequence with structured input using encoder-decoder architecture, and 3) context-free language learning using memory. In the first two cases, the convolution (of single width) is used to benefit from the symmetry prior, and in the third task, convolution is applied to stack memory structure. In all cases, the proposed architectures were shown to outperform MLP and RNN.

The paper addresses an important area in deep learning, and the paper is accessible and easy to read. However, there are major issues:

-- I found it challenging to identify a novel contribution. For example, the first task is simply using a single convolution layer followed by pooling for sequence prediction. However, using 1D convolution layers are somewhat wide-spread in NLP. 

-- The paper is oblivious to a large body of related work in the area of relational learning and invariant/equivariant deep learning. Here are some examples: Permutation invariant model for sets [1,2], and the link between parameter-sharing and invariance [3,4] is theoretically studied in several works. Note that convolution with a filter of width one followed by pooling is exactly invariant to the symmetric group. There are related works that extend these ideas to graphs [5,6], and relational learning [7]. Invariance has also been explored as it relates to memory [8]. Another relevant direction to discussions of the paper is the idea of attention in various architectures, such as transformers.

-- There are vague or misleading claims. In particular, for some tasks, it is not clear why the proposed architecture addresses the targeted symmetry. For example, it is not clear why translation invariance in-memory models the structure in a context-free grammar. 


[1] Zaheer, Manzil, et al. "Deep sets." Advances in neural information processing systems. 2017. 
[2] Qi, Charles R., et al. "Pointnet: Deep learning on point sets for 3d classification and segmentation." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017. 
[3] Shawe-Taylor, John. "Building symmetries into feedforward networks." 1989 First IEE International Conference on Artificial Neural Networks,(Conf. Publ. No. 313). IET, 1989.
[4] Ravanbakhsh, Siamak, Jeff Schneider, and Barnabas Poczos. "Equivariance through parameter-sharing." Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017. 
[5] Kondor, Risi, et al. "Covariant compositional networks for learning graphs." arXiv preprint arXiv:1801.02144 (2018). 
[6] Maron, Haggai, et al. "Invariant and equivariant graph networks." arXiv preprint arXiv:1812.09902 (2018). 
[8] Kazemi, Seyed Mehran, and David Poole. "RelNN: A deep neural model for relational learning." Thirty-Second AAAI Conference on Artificial Intelligence. 2018. 
[9] Vinyals, Oriol, Samy Bengio, and Manjunath Kudlur. "Order matters: Sequence to sequence for sets." arXiv preprint arXiv:1511.06391 (2015).

review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: 
======================================== Update after rebuttal =============================================

I have now read the author rebuttal, but my concerns about the paper remain. The training details are not described in anywhere near sufficient detail (optimizer?, batch size?, learning rate?, initialization?, etc). The baseline architectures “recurrent net” or the “multi-layer perceptron” are not described at all, despite my explicit request to that effect. I had also requested to see the source code for the experiments as this would perhaps have illuminated a lot of the details left out in the paper, but the authors have not provided it. I understand that the authors are not required to provide their code, but this should have been a relatively straightforward request in this case given the simplicity of the experiments and as I mentioned in my initial review, it would have been very useful in evaluating the paper. 

In their rebuttal, the authors also claimed that the results in Fig. 1 and Table 1 are training results (that even though the architecture is "innate", the weights are learned), but I'm concerned about this claim. I happen to be doing some experiments along these lines at the moment, and it is not trivial at all to get such crisp results as those shown in Fig. 1 & Table 1 in these kinds of experiments (even when the architecture is correctly specified). Again, it would have been very helpful if the authors had either provided their source code or had described their experimental setup in sufficient detail to allow the reproduction of these results. 

Given these concerns, I have decided to keep my score as it is.

========================================================================================================

This paper addresses an important problem: systematic generalization in neural networks. However, the paper is very confusing and I have some serious concerns about the models and the results presented in sections 3 and 4. Here are the main issues:

1) In section 3, there are only 10x10=100 possible combinations in this composition task. Yet, the paper says “we randomly sample 1000 such translation pairs, choose three combinations and remove all instances of them from the training data and then exclusively test on unseen pairings of command and modifier.” How can you sample 1000 pairs out of a possible 100 combinations? Also being able to generalize to 3 held-out combinations out of 100 is not very impressive. On the contrary, it is almost trivial.

2) No details are given about the “recurrent net” or the “multi-layer perceptron” baselines in section 3. What are these models? The fact that they have exactly zero accuracy is a bit suspicious, especially given the almost trivial nature of the task in section 3. Previous works reported perfect or near perfect accuracy with similar baselines in similar tasks (see e.g., Lake & Baroni, 2018).

3) I'm afraid the proposed model in section 3 also doesn’t make sense to me. It is explicitly acknowledged (Appendix B) that y is an M+1-dimensional vector, g is an Nx(M+1) matrix. Then by all accounts, the convolution of these should be an N-dimensional vector. Yet, somehow, h_t+1 in Equation 2 manages to be an NxM matrix. How? Please clarify this. If possible, making the source code available would be very helpful.

4) What is the semantics of x and y in section 3? What exactly are they supposed to be doing? This is not explained in the paper beyond a vague description.

5) Similar problems arise in section 4. The task is not explicitly described in the text. We only learn from Appendix C that it is actually to predict the next symbol. The task description mentions “strings of length 15, 17, 19, 21, 23 and 24,...” (p. 5). But, the grammar in Fig. 3 can only generate odd length strings, it cannot generate a string of length 24. Is this a typo?

6) Again in section 4, I have no idea how the proposed model is actually supposed to work. The motivation for the model and its description are not clear at all.

7) The model in section 2 is hand-coded. It is not shown that it can actually learn this solution from data. What happens if the sequences are longer or if the rules are different, for example? Then you have to hand-code a completely different architecture.

8) Which brings me to another important issue I have with this paper (and with similar papers): this whole set-up is very misguided in my mind. I think the real problem is not to come up with an architecture that would generalize systematically in a very specific (and usually toy) problem. It is to come up with an architecture that would learn to generalize systematically in a much broader set of problems. The learning aspect in sections 3-4 is a step in the right direction, but there’s no evidence that the models proposed there can learn to generalize in any task other than the very specific tasks they were designed for (if they can actually do that).
review_title: Official Blind Review #3

Meta-review: Thanks for clarifying several issues raised by the reviewers, which helped us understand the paper.

After all, we decided not to accept this paper due to the weakness of its contribution. I hope the updated comments by the reviewers help you strengthen your paper for potential future submission.
Decision: Reject