Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper revisits the problem of neural network's systematic generalization ability from the perspective of meaningful learning, or more specifically, semantic linking. Based on this view, they propose two data augmentation methods from either the inductive learning perspective or deductive learning perspective. They train different model variants with such augmented data. The empirical results on SCAN, GEO, and ADV show that models can behave systematically. They further group some data augmentation methods on the machine translation task and semantic parsing task into the inductive or the deductive category, and show these augmentation methods can bring benefit in real data.

main_review: Strengths:
1. The interpretation of system generalization from the psychological concepts of meaningful learning and semantic linking is really interesting and reasonable. 
2. The author designed extensive experiments to prove the concepts and also show the effectiveness on real data - machine translation and semantic parsing.

Weaknesses:
1. Although the author is well motivated by some psychological concepts such as meaningful learning, semantic linking or prior knowledge, I had a hard time to draw the connection between these concepts to targeted systematic generation tasks (e.g., SCAN). I think the author should give a more concise definition of these concepts, and improve the examples in Figure 1 with better explanation.
2. Although the perspective of inductive and deductive learning sounds fancy and reasonable, the introduced data augmentation methods are quite simple, and a bit frustrating. 
3. It seems this paper only focuses on a specific type of generalization - lexical variants, while most of the systematic generation research focuses on compositionality. The paper seems to overclaim and not be clear at the early part of the paper.
summary_of_the_review: This paper provides a new perspective of systematic generalization, and gives empirical results to prove that methods based on this perspective can improve model's generalization on toy and real tasks. However, the explanation of the new perspective is not clear enough, and the proposed methods look pretty simple and only solve simple generalization problems.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper introduces an interesting idea of improving the systematic generalization ability via meaningful learning. Through providing augmented data for inductive learning and deductive learning, the sequence-to-sequence model can be more generalizable to compositions of new concepts. It tests on real data to provide evidence of the efficacy. 
main_review: Strengths:

1. The data augmentation method is interesting, bringing semantic linking to bridge the new and old concepts. Deductive learning and inductive learning provide two learning directions for specific-to-general and general-to-specific.

2. It designs and conducts comprehensive experiments for analyzing how the semantic linking in inductive learning and deductive learning affects the models. The ablation study shows how various primitives impact the model.

Weaknesses: 

1. The technical contribution is weak. It proposes the augmentation approach using the semantic link but does not study how to better use the semantic link. I would like to see more insights into how to design the model beyond these traditional sequence-to-sequence models.

2. It talks a lot about inductive learning and deductive learning but does not provide learning approaches in the deductive style.

3. Some arguments and claims are not appropriate and make it harder to understand the approaches. For example, the two methods for injecting the semantic link are actually rules with or without context. The inductive learning and deductive learning story make it ambiguous on whether it is a learning strategy or data augmentation strategy. I suggest the proper tone should be augmenting the data and then explaining the learning process with augmented data in two paradigms, but not listing the inductive/deductive learning at the beginning, which makes it very confusing.

summary_of_the_review: The technical contribution is relatively weak and the arguments and stories of this paper make it hard to understand the specific contribution of each part.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper considers the problem of learning novel words from a few examples. The authors name their approach as "meaningful learning," which, at a high level, means that we should relate the new word with existing words. The concrete technique they proposed is to use domain-specific rules to generate new data that contains novel words based on the existing examples.
main_review: Overall I think the presentation of the algorithm can be further improved by walking through a concrete example, especially for 2.2 and 2.3. For example, when the authors says, "getting prompt by replacing words with slot mark", it will be great if we can get a concrete example showing what's exactly the prompt. Meanwhile, the arrows in Figure 1 is kind of confusing. For example, why is the arrow from "prior knowledge" to the bottom middle box the same type of arrow as the one from the bottom middle box to "new compositions?"

My primary concerns about this paper are the following.

1. The learning setup contains very strong prior knowledge about the domain, which, sometimes can be wrong. For example, in real-world datasets because of pragmatics, "red" and "green" might not be always interchangeable: "you have a green light."
2. If we view the algorithm just as a data augmentation technique, I believe, at least for the cases studied in this paper, it has been covered by many existing techniques:
- https://arxiv.org/abs/2011.09039 This paper contains results for a number of heuristic augmentation techniques.
- https://arxiv.org/abs/1904.09545 
- https://arxiv.org/abs/2010.03706 Learning to do data augmentation.
- https://aclanthology.org/2021.findings-acl.307.pdf (replacing a subtree with a new tree that has the same tag)

The authors discussed a bit about some related works in the second paragraph in Section 4, but I am still not seeing enough contribution compared with existing methods. For example, can I interpret the authors' comments on existing replacement-based augmentation as:

Existing methods use automatic ways to find possible replacements to augment the data (e.g., from a few occurrences, trying to guess that jump_0 and jump can be used interchangeably), however, if we know more about the domain, we can just write Python programs to tell the model that jump_0 and jump can be used interchangeably (this paper)?

If I'm going to be harsh, I would like to argue that in the settings studied in this paper (See Table 12 in the appendix), all these tasks can be performed simply by replacing "jump_0" with "jump" ("Houston city" with "New York city", etc) during test time. This assumes the same kind of knowledge as the authors' proposed technique.

3. I am not sure about the significance of having multiple splits in Table 2. Why is this setting meaningful? If we already know the heuristic rules, then what's the scenario where we need to exclude primitive training samples of the same length as their variant samples...?
Note that these generalization tests are all "easier" than the one we have in SCAN (e.g., jump generalization, where there is only one single example of the novel word "jump")

summary_of_the_review: The paper is decently written and the statements are well supported. However, I failed to see the significance of the paper nor its contribution compared with other existing papers.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paperintroduce semantic linking for systematic generalization through the analysis of inductive and deductive learning from a meaningful learning perspective. They show that both prior knowledge and semantic linking play a key role in
systematic generalization, which is in line with the so-called 'meaningful learning theory'. Interesting results are attained from
SCAN to real data.
main_review: Merits:
* This paper focuses on an important problem - systematic genneralization and compositional generalization in particular.
* The experimental evaluation is sound including both synthetic and NMT tasks. The evaluation protocal and the design of three tasks are valuable for followups.
* The empirical performances are encouraging and interesting.

Flaws:
* The model depends heavily on the prior knowledge about the structure of the task but essentially a more realistic approach towards compositionality is concerning how to induce gramatical rules over raw natural language (considering the birttleness of many neural-symbolic systems).
* Considering there are already many works using data augmentation to improved compositional generalizaiton [1,2], the algorithmic novelty only combining with semantic linking is quite doubleful in my opinion. It is interesting to unify existing (data-augmentation) frameworks but the efforts are not solid enough.
* The inductive and deductive processes are not very well justified. For example, in symbolism, inductive learning induce (logical) rules explicitly by composing predicates. But in the paper, it seems the model only use rules to generate extra parrallel data for training.
* It would be nice to discuss more on the connections between the intruiging performance in synthetic dataset like SCAN and the systematic generalization capability over real-world cases [3, 4]. Considering there are many works on compositional 
whether they are only exploiting dataset biases is an open question and suits the title of this work.
* The claims in the paper are a bit vague like 'meaningful learning' (seems like a cognitive concept but not very clear to many authors). And the experiment part is well-written but the previous parts are not as good as it. It would be much better to have an algorithm for introducing the method.

[1] Learning to Recombine and Resample Data for Compositional Generalization

[2] Good-Enough Compositional Data Augmentation

[3] The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers

[4] Can Transformers Jump Around Right in Natural Language? Assessing Performance Transfer from SCAN

summary_of_the_review: The paper has some merits but the overall claims are not well justified. The work can be improved after a major revision and narrow down the claim a bit.
review_title: null

Meta-review: This paper attempts to rationalize data augmentation techniques for compositional generalization by evoking the principle of meaningful
learning which posits that learning new concepts builds on previously learned concepts (which learners already understand). So for compositional generalization, this means that a model exposed to some new concepts in the test set, should link them to known concepts which have been already attested in the training set. The links between concepts are presumed to be semantic, e.g., hyponyms, hypernyms, or lexical variants. Ideally, a model should perform semantic linking on its own, however the authors do not propose a linking mechanism. Rather they investigate data augmentation as a way of exposing a model to semantic links and then explore whether different operationalizations of semantic linking enable the model to generalize better. Inductive learning is a bottom-up approach, where links are created from general to specific concepts, whereas deductive learning is a top-down approach where links are created from specific to general concepts. Experimental results indicate that inductive learning works better. 

The reviewers had the following issues with the submission (a) the technical contribution is not very strong (the idea of data augmentation is not new, although the authors' meaningful learning perspective is) (b) semantic linking seems to be able to handle only cases pertaining to lexical generalization (even though the authors include examples with structural generalization in their splits, there is no reason why semantic linking could handle these cases); (c) it would be more interesting/useful  to learn the linking than assume it is given. The authors did their best to respond to the criticism, but ultimately addressing the criticism is future work. I would also recommend to take a look at this dataset which might be useful for machine learning experiments: https://arxiv.org/abs/2105.14802
Decision: Reject