Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: 
In this paper, the authors propose to use a Maximum-Entropy Rewarded Reinforcement Learning framework to select training data for NLP tasks, the goal of which is to maximize generalization. The authors experiment with A2C and SAC and experimental results show that the proposed framework could outperform several baseline approaches.


strength_and_weaknesses: 
Strength: In general this paper is clearly written and easy to follow. The experimental results seem to confirm the validity of proposed method.

Weakness: I suggest the authors include more justifications when selecting certain models of NLP tasks. For example, in sentiment analysis experiments, why the authors choose to use a CNN classifier as the model?
clarity,_quality,_novelty_and_reproducibility: 
This paper is easy to follow. The authors provide the detailed information for reproduce the experiments.
summary_of_the_review: 
I think the authors propose a novel method of using entropy in selecting data.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors introduce a novel framework, called Maximum-Entropy Rewarded Reinforcement Learning (MERRL), which can select training data to cover more possible queries that may appear in unknown worlds.
strength_and_weaknesses: Strengths:
In INTRODUCTION, the authors first proved the relationship between `training set entropy’ and `f1 score and oov’ through experiments, and explained the importance of training set entropy to model generalization. The motivation and logic of the paper is clear.

Weaknesses:
The authors only proved the role of entropy in selecting data, but this paper does not elaborate on the motivation and advantages of introducing complex reinforcement learning to train a policy network.

Further Comments:
1.	The authors use training set entropy as a reward to train a policy network for data selection. How is it different from directly using entropy and selecting data through threshold? What are their advantages and disadvantages?
2.	In the code provided in the supplementary materials, the policy network is first trained on the data set to be selected. Does the policy network need to be retrained on the data set to be selected each time? Considering the generalization of reinforcement learning, this will limit the universality of the algorithm.
3.	In Figure 5, the initial reward of SCA is much higher than A2C and VPG, and the trend is different from that of A2C and VPG. What is the reason for these phenomena?  Is the value of reward related to the quality of data selection?
4.	There are some clerical errors in the paper. For example, in second page, it should be “Pr(“nor”) = 1/2”. It is recommended that the author read the full text to correct such problems.

clarity,_quality,_novelty_and_reproducibility: The writing is good while the novelty is not enough.
summary_of_the_review: The authors only proved the role of entropy in selecting data, but this paper does not elaborate on the motivation and advantages of introducing complex reinforcement learning to train a policy network.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method for optimal training set selection with the goal of maximizing generalization to multiple unknown target domains for NLP tasks. One of the goals of the method is to perform data selection on the training set only without knowledge of any target domain. To achieve this, the paper proposes a Maximum-Entropy Rewarded Reinforcement Learning (MERRL) framework, which seeks to maximize the entropy of the training data. The proposed method rests on the key assumption that the optimal training data for generalizing to many different tasks simultaneously is one that maximizes the training data's entropy and hence is not "latched on" to any one specific domain. The authors experiment with two RL algorithms (A2C) and SAC and two measures of entropy: observational entropy (OE) and prediction entropy (PE) and find SAC combined with PE to be be optimum. On four NLP benchmarks the authors show that their proposed approach outperforms several previous approaches for this task, including ones that specifically use the target dataset for optimal training set selection.
strength_and_weaknesses: Strengths:
Novelty: The proposed method addresses an open under-researched and impactful problem of optimal generalization from seen to unseen domains. It presents the novel and intuitive insight of maximizing the entropy of the training dataset to improve generalization on target tasks. The paper also proposes a straight-forward and simple method based on RL to maximize this uncertainty measure. 

Soundness: All aspects of the proposed method are theoretically sound and correctly motivated.

Clarity: The paper is very-well written and very clear.

Experiments: The proposed method shows significant and impressive performance improvements over all previous methods that it is compared against in the paper, and even those that use target datasets on four different datasets. The authors have also clearly empirically ablated their design choices for the type of RL algorithm and the uncertainty measure to use.

Weaknesses:
1. One obvious weakness of the proposed method is its high computational cost, which in turn stems from the use of the RL algorithm. RL is widely known to be sample inefficient. The use of the SAC versus the A2C algorithm mitigates this somewhat in the proposed design. While the authors allude to this this briefly in their paper, this weakness could be acknowledged and discussed more in the paper.

2. Another widely used strong baseline method for optimal sample selection with known target datasets is the following: Ren et al., Learning to reweight examples for robust deep learning, ICML 2018. However, the authors have not compared against it. It would have been interesting to see this comparison.

3. The proposed method could be made more impactful if the authors had shown it to also be applicable to other domains besides NLP, for example to vision as well.
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is very clear and well-written.

Quality: The quality of presentation, theoretical soundness of the technique, the experiments and of the significance of the results is good.

Originality: Within the context of the prior work presented in this paper for the task of NLP for optimal domain generalization, the proposed method seems novel. I am not an expert in this field of NLP and hence I am not deeply familiar with the specific literature in this field beyond what the authors have cited. However, the idea if using RL for optimal training data generation is not entirely new and has been used for example in prior works in the related field of computer vision in the following works:

(a) Ruiz et al, Learning To Simulate, ICLR 2019 (https://openreview.net/forum?id=HJgkx2Aqt7)
(b) Mishra et al, Task2Sim: Towards Effective Pre-training and Transfer from Synthetic Data, CVPR 2022 (https://openaccess.thecvf.com/content/CVPR2022/papers/Mishra_Task2Sim_Towards_Effective_Pre-Training_and_Transfer_From_Synthetic_Data_CVPR_2022_paper.pdf).

Reproducibility: Good! The authors have provided code (which I did not run) and have described the method is sufficient detail.


summary_of_the_review: Overall, I am leaning positive about this paper based on the material presented in it with the caveat that I am not an expert in the field of NLP and hence not familiar with its most current literature. I am aware of closely related works from the field of computer vision that have previously employed ideas of using RL or meta-learning for optimal training set selection/synthesis. I would like to hear from the authors about how they position their method within the context of these existing closely related-works in the vision domain; what their thoughts are on the limitations of using RL for their task; and how generalizable they think their technique would be to other problem domains besides NLP.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: 
Strengths

* the paper has a clear motivation and addresses an important topic (data selection for out-of-domain generalisation);
* the paper is very clear and proposes a technically sound solution;
* their original method outperforms reasonable baselines across a few benchmarks (text classification, named-entity recognition, and language modelling -- in the sense of next word prediction with evaluation via perplexity).

Weakness

* RL is sample inefficient (I'd take that as a weakness only if some other sample efficient baseline existed, see remarks below) and details about the computation cost of the method were missing;
* Some unusual architectural choices (e.g., sentiment classification should be done on top of a large pretrained model; language models are not trained from scratch these days, rather, they would be fine-tuned on your selected data); 
* NLP-only benchmarks;
* Benchmarks are not representative of NLP as a whole, yet the paper keeps referring to NLP as a whole. I don't think a single paper can touch upon enough NLP tasks to make a general claim about NLP, thus my advice is to nuance the language in the paper. 

Remarks

* there were some reservations about RL (in particular it's sample inefficiency) that I am choosing to ignore in light of the rebuttal which further characterised the computational cost of the method;
* the reviewers requested two missing comparisons (against ranking on entropy and thresholding, and against Ren et al), both sufficiently addressed in the rebuttal;

The paper is executed reasonably well and I see no obvious technical flaws. Some reasons for rejection might be the unusual architecture choices (the paper's findings would be more readily available to a wider audience if experiments had exploited modern pretrained components throughout) and the overly general language around "NLP" when the investigation only covers 3 little corners of NLP.
justification_for_why_not_higher_score: * some architecture blocks simply do not speak to a modern NLP audience (e.g., CNNs for sentiment classification, LM trained from scratch on selected data);
* the paper could nuance most uses of the word/acronym "NLP" in its narrative.
justification_for_why_not_lower_score: * plausible and original premise, incorporated reasonably well into an algorithm for data selection
* elegant use of RL to circumvent combinatorial search
* overall, nicely executed paper
* the rebuttal addressed most criticism gathered in the first round
Decision: Accept: poster