Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper presents a prototype-based method for data augmentation based on a generative model without rule/template based requirements. The generative model creates new input-output pairs from training fragments (recombination: rewrite model conditioned on multiple examples), and samples in low-density places (rare words) of the training data (resampling). Empirical results show that the in combination recombination and resampling perform on par with a recently introduced rule-based method, GECA.
Experiments are conducted on two compositional generalization tasks: SCAN and sigmorphon. 

The paper is very clearly written and motivated, doing a good job in presenting the recent and past pertinent literature. The problem addressed is of great interest, and the two proposed contributions of resampling and recombination are likely to be useful to further research in data augmentation. 

The extension of prototype models to multiple examples is a promising step, but depending on the task leaves open questions. 
The empirical results on SCAN are strong. The results on Sigmorphon are strong in the sense of obtaining comparable accuracy to simple rule-based approach, which itself is very simple and has many incorrect examples it constructs, but do not clearly outperform it.
Granting resampling is a contribution, wouldn’t the proper comparison be GECA resampling with the recomb-1 or -2, since those include resampling as well? In that case the Sigmorphon performance is much closer to each other for the two methods. 

Taking the performance jump from 1 to 2 prototypes on SCAN as potential for further jumps, why restrict to n=2? And related, why do you think -2 outperforms -1 in some instances on Sigmorphon?  


review_title: Clearly written and motivated contribution to data augmentation with solid empirical results

final_score: rating: 7: Good paper, accept
review_content: 
Summary:

 
The paper proposes an interesting approach to systematically generate new examples and augment the training data with these examples. The goal is to target rare and unseen sequences of text or instructions with this augmentation. 
In particular, it proposes learning to copy parts of the reference examples. The approach is based on prototype-based models where every training example can be explained by at least one other example and a parametric rewriting operation. They show that these models do not perform well when facing complex (and rare) composition events and propose a recombination addition to address this issue.
 
I like the idea of systematically augmenting training data targeting rare and unseen subsequences. I give an "accept" to this work because of its novelty and contribution (see pros below). 
My minor concern is about the impact and/or usefulness of this work when dealing with real-world datasets and more complicated tasks as well as some clarity issues (see cons below). Hopefully, the authors can address my concern in the rebuttal period. 

 ##########################################################################

Pros:

1. The paper addresses one of the interesting and important shortcomings of current neural models: the ability to generalize to rare and unseen sequences. I find this problem important to investigate and applicable to many areas of research.

2. The proposed approach is flexible and practical to use. The design of the prototype-based data augmentation method is reasonable and interesting. 

3. This paper provides comprehensive experiments, including both qualitative analysis and quantitative results, to show the effectiveness of the proposed framework. 

 
##########################################################################

Cons: 

 
1. In Table 2, we see F1 score for morphological analysis. It is not entirely clear to me why the results in the NOVEL section of the table is very close to ALL section. NOVEL shows model accuracy on examples whose exact tag set never appeared in the training data and I expected a bigger gap in the performance.

2. There is a big gap between the performance of 1- and 2-prototype models. Do the authors know what explains this gap? Have the authors explored the higher order of prototype recombinations? 

3. What are the challenges of applying this approach to real-world data sets for instance in machine translation? I will suggest the authors discuss the implications and possible shortcomings of such an approach when dealing with natural (and potentially long) sequences of text. The definition of unseen subsequences and compositional learning will be more complex there. 

4. It is not clear to me what the authors mean by "hints". Is the complete sequence counts as a hint? Or only the subsequence that was identified as rare?

 
#########################################################################

Some typos: 


(1) Relevant literature that was not mentioned in this paper:
         https://arxiv.org/abs/1705.00440
         https://arxiv.org/abs/1801.02929

(2) Equation (31): tex formatting issue.

(3) Typo in page 15 section G1: Fig. Table 2 shows -> Table 2 shows


review_title: Interesting work with nice findings. Would be valuable to know the applicability to real-world data

final_score: rating: 7: Good paper, accept
review_content: Summary:
* Motivated by the fact that certain datasets require modeling compositional phenomena, the lack of flexibility of highly structured models, and the strong performance of large unstructured models on unstructured data, this paper approaches the problem of getting unstructured models to generalize on compositional data.
* Prior work showed that a simple rule-based data augmentation approach could allow unstructured models to generalize on compositional data. This paper demonstrates that a learned data augmentation strategy can be as effective at encouraging generalization as a rule-based one.

Contributions:
* Extends the prototype+edit model (Guu et al 2018) to a recombinator model with multi-source copy attention.
* Proposes a resampling scheme for upweighting rare examples.
* Obtains results comparable to strong rule-based data augmentation baseline GECA on two datasets, demonstrating that the combination of both resampling and recombination is effective.

Strengths:
* Clearly written. The method is simple and is broken down cleanly into recombination and resampling.
* Ablation studies support the need for both recombination and resampling.
* Sufficient performance to support the claim. The approach manages to match the performance of GECA with a learned method, while remaining more flexible.

Weaknesses:
* Neighbourhood heuristics are one of the last remaining applications of manual rules in the method, but seem necessary for computationally feasible training.

Decision: Accept
* Problem is important: Whether unstructured models can generalize on structured data has implications for whether or not to move towards more structured models. This paper provides experimental evidence that unstructured models can generalize on structured data with a data augmentation procedure that uses fewer manually specified rules than previously shown. This provides a path forward by continuing to iterate on the augmentation procedure.

Questions:
* Given that GECA is feasible on both datasets, would there be benefit to combining the examples from the learned augmentation strategy with those from GECA? In other words, do the different augmentation strategies result in orthogonal improvements?
* Why does resampling hurt GECA on SCAN?

Suggestions:
* Both datasets are quite small. The story could be strengthened by demonstrating the method scales better than GECA by applying it to a larger dataset as well, such as a translation dataset.

Nit:
* w is overloaded to both be a value of d (eqns 12, 13, 17) as well as the weighting function (eqns 1, 18)

review_title: Shows unstructured models can generalize on compositional data via a learned data augmentation procedure

final_score: rating: 6: Marginally above acceptance threshold
review_content: ####Summary: 
To tackle situations where compositionality is mostly required at inference time, the paper proposes a novel data augmentation method with an RNN based generator (recombination); to make the generator generate highly compositional patterns, the paper proposes a resampling method. The methods have been tested on two benchmarks focusing on the issue, SCAN and morphological analysis. The system performs on par with recently proposed GECA for SCAN and favorably to GECA on morphological analysis. 

The two datasets have some “toy flavor”, while SCAN great favors example combination (with recomb-2 performs much better than recomb-1), recomb-1 seem to perform better for morphological analysis dataset, leaving questions about how to choose the exact models in general. 

####Pros: 
The paper proposes the first RNN based neural generator to perform data augmentation for “extreme” compositionality inference. The paper has explained and empirically showed that this learned generator needs a resampler. With these two elements, the approach performs on par with recently proposed GECA (where the data is not augmented via a neural generator) on two datasets.

####Question: 
1. Comparison with GECA: I can read from the paper that the performance is on par with GECA. However, I am unable to grasp nuances, leaving important questions untouched such as: In what scenarios do we expect the model to perform better than GECA? In experimental details, the slightly better performance in Table 2, can it be attributed to finer generation powered by the RNN generator? Why does Recomb-2 perform less well than GECA in SCAN? 

2. A uniform framework for resampling
Different recombinations perform more or less favorably across different datasets. While the exact choice depends on the dataset characteristics, a framework will be more attractive if it can perform well on different scenarios. Could the authors list some possible approaches to automatically choose this hyper-parameter please?

####Minor Comments: 
The paper mentions in several places symbolic scaffolding without citations, literature is certainly rich here, e.g. [1,2] are papers integrating symbolic constraints for semantic parsing. There are also neural architectures that particularly target to ensure some symbolic famous properties such as [3]. 

The authors say in the introduction that the approach (Andreas, 2020) is task specific which seems not correct. In fact, it can be applied to a large range of NLP problems (e.g. all the experiments in this paper compares to the approach GECA))

In section 3, it says that “the use of a continuous latent variable appears to make no difference”, I would suggest to precise “make no difference” as “make no difference in prediction performance“ as the latent variable can facilitate some generation control shown in (Guu et al. 2018). 

In this paper, GECA is first introduced in section 5. I would recommend to put the citation around it (Andreas, 2000) although previously cited.

[1] Sequence-based structured prediction for semantic parsing, Xiao et al. 2016
[2] A syntactic neural model for general-purpose code generation, Yin and Neubig 2017
[3] Making Neural Programming Architectures Generalize via Recursion, Cai et al. 2017

####Authors have engaged in the discussion, clarified questions about the paper and addressed comments in its newest revision. I have consequently revised my score from 5 to 6. 
review_title: neural data augmentation for compositionality; not clear for its pros and cons compared to its non-neural counterpart.

Meta-review: The paper addresses generalization to compositions of rare and unseen sequences. It proposes an unstructured data augmentation, that achieves comparable generalization to structured approaches (e.g. using grammars). The idea is based on recombining prototypes and oversampling  in the tail. 

The paper provides a novel approach to an important problem. All four reviewers recommended accept. 


Decision: Accept (Poster)