Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposed a new approach composed of multiple components for systematic generalization on language tasks. It achieves good results on multiple datasets including SCAN, PCFG, and a task for arithmetic reasoning. 
strength_and_weaknesses: **Strengths**: 
* **Domain**: the paper explores an important and growing domain of systematic generalization.
* **Experiments and results**: The paper presents experiments in multiple complementary datasets covering parsing, semantic reasoning, compositional generalization and arithmetics. It achieves good results across multiple tasks, and compares to multiple suitable baselines. The descriptions in the experiments section of the tasks and the baselines are also clear, well-structured and easy to follow. 
* **Modeling section**: The discussion in the modeling section is interesting and gives solid motivation for the approach presented in the paper, and the section is structured in a good and clear way. The model is also discussed from a theoretical perspective, and formally defines its properties (equivariance and recursiveness). 
* **Related work, background and context**: the paper does a good job contextualizing the approach, providing the necessary general background, and covering the relevant related fields. I recommend moving the related work section to an earlier point in the paper (before the modeling section). 
    
**Weaknesses**:
* **Related work comparison to specific recent approaches**: More details on how the new approach compares to recent models on the explored tasks would be useful.
* **Complicated approach**: The approach is a bit complicated and consists of multiple different modules that are responsible for different aspects of the problem. We know that models such as transformers do amazingly well on language tasks being more uniform, and so I recommend examining whether the model could be simplified without negatively and potentially also positively impacting its performance. 
* **Undiferentaible**: The NSR model isn’t end-to-end differentiable, making training more challenging. Having multiple units that interact through hard predictions (without a gradient) could also lead to cascade errors, where early mistakes in early modules prevent the following ones to give the right predictions.I recommend exploring ways to make it differentiable, either by relaxing some of the constraints or using techniques such as gumbel-softmax etc. 
* **Synthetic data**: The model is evaluated on synthetic data only. I would highly recommend exploring e.g. logical problems in natural language such as word problems, logical questions (e.g. LSAT), etc etc, to strengthen the paper.
* **Suggestion: focusing on the reasoning modules**: I also recommend exploring the parts of the model related to parsing and reasoning independently of the perceptual module since that is the central part of the paper, so would be good to highlight that, and also would broaden the range of datasets that could be explored.
* **Evaluating each component independently**: Providing more details on the performance of each component independently and performing ablation studies is critical for models with multiple components, rather than only having results on downstream task performance.
clarity,_quality,_novelty_and_reproducibility: **Clarity**: The writing of the paper’s abstract could be improved to increase the clarity, but the other sections’ writing quality is good. The multiple diagrams and visualizations  are clear and helpful to convey the ideas of the paper, and the key concepts (e.g. the analogy among different problems having underlying tree structure, and the NSR high-level pipeline). I recommend moving some of the visualizations in the appendix to the main paper.

**Novelty**: The proposed approach is new and the paper explores an important underexplored domain.

**Reproducibility**: Adding more details such as hyperparams and finer modeling decisions for the different component would be useful to increase the paper’s reproducibility.

summary_of_the_review: The paper explores an important direction and show nice experiments but I believe the main issue is the model that could be improved as well as some updates to the writing, and therefore overall at this point recommend weak rejection, but at the same time I wish to encourage the authors to work on improving the paper further to make it better!
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes Neural-Symbolic Recursive Machine (NSR) as a means for neural networks to learn compositional rules through 3 modules, one for processing perception, syntactic parsing and semantic reasoning. These modules made up the Grounded Symbol System (GSS) and are jointly learned through a deduction-abduction algorithm. The authors also showed that the NSR modules confer equivariance and recursiveness to the network’s representations. The experiments showed that NSR outperforms baselines on semantic parsing, string manipulation and arithmetic reasoning tasks.
strength_and_weaknesses: Strength:
The proposed idea is sound and shows superior performance vs some baselines.
Theoretical analysis is done to show proposed NSR confers equivariance and recursiveness.

Weaknesses:
Lack of comparison versus more recent baselines such as Ontanón et al. (2022) 

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and well organized. For a reader who is not very familiar with this field, it is not easy to tell how the proposed NSR is novel and different from existing work. This would be addressed by better contrasting NSR against prior work in the “Related Work” section. The proposed design of NSR using 3 modules to process perception, syntactic parsing and semantic reasoning to confer systemic generalization is intuitive and the experimental results show superior performance versus baselines. The proposed deduction-abduction algorithm to address the model’s training is sound and design choices are also well-explained.
summary_of_the_review: Overall, the paper proposed a sound and intuitive approach for a model’s systemic generalization which is a timely contribution given how important generalization is for deep learning models. One area for improvement is more comparison against more recent baselines and prior work from the ‘data augmentation’ and ‘symbolic scaffolding’ categories. Though I am not an expert in this field, the contributions seem substantial enough for me to lean toward an accept.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new model -- Neural-Symbolic Recursive Machine (NSR), that uses a Grounded Symbol System as its core mechanism. The model includes three components: a perception module, a parsing module, and a program induction module. The three modules are trained in an end-to-end fashion without intermediate supervision. The system benefits from the inductive bias of equivariance and recursiveness. It shows a strong compositional generalization ability on several standard benchmarks, including SCAN, PCFG, and HINT. 
strength_and_weaknesses: Strength:
1. The proposed NSR achieve impressive generalization results on several datasets.
2. The three-module design is intuitive and general enough to be applied to different tasks and modalities.
3. Experiments on SCAN and PCFG show that NSR has good interpretability.

Weaknesses
1. The NSR is only tested on synthetic tasks. The limitation of applying NSR in real-world tasks is not discussed either.
2. The paper is difficult to follow, important diagrams and figures are put in the appendix.
clarity,_quality,_novelty_and_reproducibility: The method section is difficult to follow, important figures and algorithms that are essential for understanding the algorithm are put in the appendix. This also makes the model difficult to reproduce. 
summary_of_the_review: Overall, the paper proposed an interesting new framework for achieving compositional generalization. Experiment results confirm the effectiveness of the model in synthetic tasks. However, the proposed method is fairly complicated, which shed doubt on whether the method can be generalized to real-world tasks. The author didn't provide any proof or discussion on this question.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces a ‘neural-symbolic recursive machine’ (NSR) to learn compositional rules from limited data that can be applied to unseen combinations in various domains. SotA is reported to be achieved on SCAN, PCFG, and HINT, which suggests at good domain-generalizability.
strength_and_weaknesses: Strengths

- NSR is evaluated on three benchmarks from different domains, which demonstrates a good degree of generalization, and appears to achieve SotA on all three.
- The related work of Sec 4 is fairly comprehensive in topic, and clarity.
- The visualization of Fig 3 is fairly convincing

Weaknesses

- Although this is relatively minor, there are more recent alternatives to Chen & Manning (2014) for dependency parsing, including Mrini et al (2020) and a few based on deep biaffine attention.
- Also relatively minor, but it would be preferable to explore the space of program induction to a greater degree than in Sec 2.2, as this is a fairly active area of research.
- The baselines are adequate, but other more recent permutations of some of these exist (and it it probably not necessary to include seq2seq at this point). For example, is it possible to compare NSR against the approach in Ontañón et al (2022; arXiv:2108.04378v2), whom are cited?

Minor

- The forced correspondence to neuroscience always seems a bit naive, and may be removed? E.g., it’s easy to find studies that show that syntax and semantics are linked anatomically in a way that may counter Miller et al (2001), and connectionism vs symbolism is somewhat of a forced dichotomy.
- Please do another round of editing, as some grammatical errors exist throughout (e.g., “… and are thus difficult to transfer such capability …”)
- There should be a citation on the “have been shown” claim in Sec 2.3
- Additional detail should be added to the lemmas of Sec 2.4. E.g., the broad claim of the existence of a neural network in Lemma 1 is not in doubt but either a citation should be added to ‘provably’ or the abstracted proof should be added, as there is already a precedent for repeating previously established facts or approaches, from Sec 2.3
- It seems that ‘Theorem 3’ is meant to be ‘Lemma 3’ but, regardless, further explanation is advised.

clarity,_quality,_novelty_and_reproducibility: - the work is very clearly described, but with a few gaps. The deduction-abduction algorithm specifically applied here provides much of the novelty.
- With regards to reproducibility, standard tools like DreamCoder are used in some cases, and other approaches are fairly clear or very well-established (Sec 2.3). The appendices provide additional detail.
summary_of_the_review: 
- This is a valuable extension of other work in this area, although it is only somewhat hampered by some vagueness, and potential strawmen of baselines, as opposed to direct comparisons with specific work.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a neural architecture suitable for learning compositional generalization consisting of 3 modules: perception, dependency parsing and program induction. Since the architecture is not end-to-end differentiable, a sampling-based inference method is proposed. A number of theoretical results regarding the expressiveness and generalization (equivariance and recursiveness) are presented. Experimental results show that the approach performs on par with previous approaches on SCAN (semantic parsing), while outperforming them on PCFG (string manipulation) and HINT (arithmetic reasoning). Despite these results, the main weakness of the paper is that the evaluation is only done on synthetic tasks. Due to the complexity of the proposed approach (consisting of multiple components with the risk of cascading errors), there is a risk that it could be tailored to specific problems while not generalizing beyond those. Therefore it cannot be concluded from the current evaluation that the approach will generalize on tasks with real data. 
justification_for_why_not_higher_score: The paper proposes a promising approach to improving compositional generalization in neural models, and therefore might be of interest to some researchers, but the approach requires additional experimental verification on non-synthetic tasks.
Decision: Reject