Learning Uncertainty for Unknown Domains with Zero-Target-Assumption
Authors: Yu Yu
Hassan Sajjad
Jia Xu
Venue: ICLR 2023 poster (2023)

Abstract: We introduce our Maximum-Entropy Rewarded Reinforcement Learning (MERRL) framework that selects training data for more accurate Natural Language Processing (NLP). Because conventional data selection methods select training samples based on the test domain knowledge and not on real life data,  they frequently fail in unknown domains like patent and Twitter. 
Our approach selects training samples that maximize information uncertainty measured by entropy, including observation entropy like empirical Shannon entropy, Min-entropy, R\'enyi entropy, and prediction entropy using mutual information, to cover more possible queries that may appear in unknown worlds. Our MERRL using regularized A2C and SAC achieves up to -99.7 perplexity decrease (-43.4\% relatively) in language modeling, +25.0 accuracy increase (+40.0\% relatively) in sentiment analysis, and +5.0 F1 score increase (+30.8\% relatively) in named entity recognition over various domains, demonstrating strong generalization power on unknown test sets.

Published as a conference paper at ICLR 2023
LEARNING UNCERTAINTY FOR UNKNOWN DOMAINS WITH
Yu Yu Hassan Sajjad Jia Xu'
"School of Engincering and Science, Stevens Institute of Technology 'Faculty Computer Science, Dalhousie University
yya508stevens.edu jaajjad@dal.coo jxu70@stevens.edu
ABSTRACT
We introduce our Maximum-Entropy Rewarded Reinforcement Learning (MERRL) framework that selects training data for accurate Natural Lan- guage Processing (NLP). Because conventional data selection methods select training samples based the test domain knowledge and not on real life data, they frequently fail in unknown domains like patent and Twitter. Our approach selects training samples that maximize information uncertainty measured by entropy, in- cluding observation entropy like empirical Shannon entropy, Min-entropy, Rényi entropy, and prediction entropy using mutual information, cover more possible queries that may appear in unknown worlds. Our MERRL using regularized A2C and SAC achieves up -99.7 perplexity decrease (-43.4% relatively) in language modeling, +25.0 accuracy increase (+40.0% relatively) in sentiment analysis, and +5.0 Fl score increase (+30.8% relatively) in named entity recognition over vari- ous domains, demonstrating strong generalization power unknown test sets.
INTRODUCTION
We introduce novel training set selection method that does not require target-domain information to improve out-of-domain Natural Language Processing (NLP) model accuracy. Machine learning is data-driven process whose success relies highly on the data in use. System performance is typically measured on specific test set, however, in reality, the test domain is often oblivious during model training. resulting in critical performance gap between laboratory findings and language use in the real world. For example, we often observe that a system that relies on human parity results generates surprising errors in real-life use scenarios.
Some work has been done in augmenting or selecting data (Wang et al., 2022) to address this dis- crepancy. Data optimization be expensive and error-prone for general domains (Jha al. (2220) Thus, conventional approaches choose critical in-domain data that may work well for a pre-defined
target domain (Moore & Lewis 2010, Kirchhoff & Bilmes 2014 van der Wees et al. 2017 Fan et al., 2017 Qu et al. 2019; Liu et al.) 2019 Kang et al. 2020). However, there are two prob- lems with fomain-specific data selection: First, shifting data toward one target domain may fail in the source and other domains. Second, when target domains are unknown, as in the case of most real-world applications, we do know what future data to receive before model launches.
In our study, we select training data without using target-domain information to achieve learning generalization. Our data selection objective is maximize the uncertainty of the training data. Specifically, we use entropy to measure the uncertainty based on the principle of maximum entropy, which states that subject to known constraints, the probability distribution that best represents the current state of knowledge is the one with the largest entropy (Jaynes 1957 Katz [967] Hernando cet al., 2012). Therefore, a system with the largest remaining uncertainty contains the least extra biases uncalled-for assumptions and is ideal for modeling distributions for unknown test domains.
To that end, we propose to measure the amount uncertainty in our observational data and in our model prediction output. As observation entropy, use Shannon Entropy. Rényi Entropy, and Min Entropy on the n n-gram relative frequency of all sentences in the dataset instead of one sentence to model the dependency among sentences. As prediction entropy, compute the mutual information

---

Published as a conference paper at ICLR 2023
(8)
Selected Cate A.
(b)
X03
002 PAR
Environment
Daak
NLP moded 3F
Po
032
F1 Score
200
DO0 Far
200
400
Meder Cardle Soue.rl Replay (i,
Roward
kumopy Predictor Cranpy ih?)
Furcttion
Training Set Entropy
Training Set Entropy
Figure I: (a): Maximum-Entropp Rewarded Reinforcement Learning framework. (b): Higher train- ing set entropy, better learning generalization, w.r.t F1 score and OOV.
between the neural network input and its latent representation to quantify how well the information compressed according to the Information Bottleneck principle. In this way, our approach makes possible to model inter-dependencces among samples that are critical to improve learning but often neglected (Steinwart et al. 2009; Zhelezniak et al. [2019] Fan et al. 2017).
Putting things into NLP context, may ask: "Why does higher entropy of the training dataset lead to a more generalized learning ability of an NLP model?" Consider a toy example of three sentences (To be. Not to be. To be not be.] with frequencies of the words "or" (1), "to" (4), "be" (4), "not" (2). Although "to" occurs more often, "not" represents the opposite meaning and contributes more the Shannon entropy value. As a hypothetical example, we assume these four words compose the full vocabulary of our world. Now consider that each word is a sample, i.e., Pr("to") = 11 Pr("or") = 11: Pr("bee)
11 and Pr("nnot)) and B, where subset A selects "to" four times, which has a unigram entropy of 0.16, while subset B selects "to", "or", "be", and "not" each one time, which has a unigram entropy of 0.49. The entropy of subset B is higher than subset A, and the (maximum) out-of-vocable smaller than subset A (for a random test), suggesting more generalized data for training that results in more accurate predictions. This observation denotes that increasing the entropy of training data helps build a generalized machine learning model.
2 11 Suppose there are subsets A
(OOV) of subset B is
Moving from the above hypothetical example, in a real dataset, does higher entropy also indicate better learning generalization, specifically fewer OOV words, and higher prediction accuracy? Fig- ure (bbb) shows our named entity recognition (NER) task results the CONLL2003 dataset (Sang) & Meulder 2003) with one in-domain and five out-of-domain (OOD) test sets, details in Appendix. We observe that the unigram entropy the training subset negatively correlates (Pearson correlation coefficient: -0.94) to the OOV of six test sets and strongly positively correlates to the in in-domiin and out-of-domain test F1 scores (Pearson correlation coefficient: 0.80). This result indicates that the subset with higher entropy is more likely to generalize on a new test domain with lower OOV rate and higher F1 score, demonstrating that the training set optimization using entropy can effectively enhance prediction accuracy unseen domains.
Knowing that a training set with higher entropy leads to more generalized learning, how can we optimize the subset to maximize the information content without any larget domain assumption? In general, the subset selection optimization problem is computationally intractable, and we use reg- ularized Advantage Actor Critic (A2C) (Mnih et al. 2016) and Soft Actor Critic (SAC) (Haarnojal et al. 2018) to approximate the set optimization. As illustrated in Figure (a)), our method equipar- titions the training data into mini-batches and simultaneously learns policy network to select data sequentially and two Q networks to estimate future returns with our entropy rewards. MERRL has the advantages of low variance, monotonic policy improvement, sampling efficiency, and signifi- cantly outperforms data selection baselines (Ma et al. 2019 Liu et al. 2019 Aharoni & Goldberg 2020).
Our work contributes four important components to ongoing work on learning generalization:
1. Maximizing uncertainty measured by entropy for learning generalization without target domain assumptions;
2

---

Published as a conference paper at ICLR 2023
2. regrooy-regulaiized A2C and SAC reinforcement learning algorithms with entropy rewards for training subset optimization that is typically computational intractable;
3. A data selection framework MERRL by modeling training sample dependency that demon- strates significant improvement in NLP accuracy and generalization on various tasks and domains
The rest of the paper is organized as follows. In Section 2, we introduce the MERRL in detail. Then in Section 3, we empirically verify the generalization and accuracy improvement using MERRL. We discuss related work in Section 4 and conclude the paper in the last section.
2
METHOD
Below, we describe our MERRL framework in detail, including problem definitions (Section 2.1 the proposed framework (Section 2.2 the training algorithms (Section 2.3 and the entropy-based reward functions (Section 2.4
2.1 DEFINITIONS
In training set optimization, we formalize the components of the environment as illustrated in Fig- ure (a), including training dataset, NLP model F, and reward function R. The train- ing set denoted as = {x;}²= where z; is a sentence (document) and n is the training set size. We shuffle and randomly partition X' into T disjoint data batches (Liu et al., 2019) so that X = =T = [B1, B2, ..., Br), with B, = (P(tt-)m|T++ [e- (),, C.n|TT n| is the integer division of n by T, and S If mod (n, * O, then the last batch has variable size of mod (n,T)) and collects the remaining sentences.
MERRL selects a subset of data from each mini batch in sequence. The series of selection can be viewed as sequential decision-maie (MDP), consisting of four elements: set of states S, a set of actions A, a transition function R. Given MDP (S, A, P, R),
process and can be modeled by Markov decision process
P S A x S [0, oo), and reward function R S the goal of a reinforcement learning system, or an agent, is to learn an optimal policy function æ. which is a mapping from the set of states S perceived from the environment or formally π + A (Uc-Cetina et al. (022). In our data selection context, the MDP elements (S. A. P. R) are specified as: the observation space S E B xd where [B,] is size a batch and d is the sentence (document) embedding dimension; the action space .A E R 1B:1; the uniform transition function P which gives the next state; and the entropy-based reward functions R (details in Section 2.4
of actions A.
2.2 MERRL FRAMEWORK
In our reinforcement learning (RL) setting, the policy interacts with the environment over number of discrete time steps T. and stores the collected experience (s, a, r) into the replay buffer. After some fixed time, the replay buffer samples tuple and updates the Q networks and policy network respectively. At each time step / E 'T, the policy receives batch of sentence embeddings from the environment and selects a subset of data. Then, the environment gives the next state 81+1 and scalar reward Tt the agent. The reward r, measures how good the selected data is. The return is the total discounted accumulated reward R. Σ}= 7rre4ş from time step to terminal time step T with discount factor y E [0, 1]. Our goal is learn an optimal policy π to maximize the expected return from each state 8t-
Each time step contains eight steps as shown in Figure I (a): At step 1, an encoder (e.g. an embed- ding layer in LSTM, or an encoder in transformer) inside the NLP model transforms the batch of raw data B, into a batch of (document) embeddings, denoted as 8t. Next, at step and step 3, the policy outputs action a, along with the selected data B., Specifically, the policy takes the state 8, as input and outputs probability distribution for st, that each sentence is associated with proba- bility representing how likely it is going be selected. The selected subset B3, is then obtained by Bernoulli sampling each sentence in the state st. The result of Bernoulli sampling is represented as an action vector at, where each value in it is either 0 or representing each sentence in the batch not
3

---

Published as a conference paper at ICLR 2023
being or being selected. At step 4, finetuned by the selected subset B At step 5, the scalar reward designed reward functions R (which we give definitions in Section 2.4). Next in step 6, the tuple of (st, de. r')) is stored in the replay buffer. After some fixed time steps, at step 7, sample previ- ously stored tuple to update the two Q networks. Finally, at step 8, we take the minimum between the outputs of two Q networks given the sampled (se, ag, re) to update the policy network with regard to the objectives expanded in next section 2.3
obtain B,, the NLP model F well encoder g
(st, a+) is calculated by
2.3 ENTROPY-BASEE BASED TRAINING ALGORTHTS
We draw on two algorithms to estimate the policy function ग,, the on-policy Advantage Actor Critic (A2C) with entropy regularization and off-policy Soft Actor Critic (SAC).
2.3.1 A2C WITH ENTROPY REGULARIZATION
The A2C algorithm maintains policy function Ty and value function Ve.. builds on the vanilla policy gradient method that directly optimizes the policy function by performing gradient ascent on Vo log x ( Rx, which is an unbiased estimate of Vo[[[R.] Intuitively, increases the log probability of the sampled action, weighted by the return R, (Uc-Cetina et al. (2021). The value function Vo, is used as a baseline scaling the policy gradient to reduce the variance in the optimization process with the objective E, [(re ? (Schulman et al. 2015 Mnih et al., 2015). With the baseline, the gradient of policy function becomes Ve lol (ae (t) where A is estimated by the difference between the the empirical return R, and value function ((se) as E/ " + dimensional action space, we refer to the maximum entropy objective (Ziebart 2010) which aug- 8x to encourage exploration of diverse behaviours and stabilize training(Mmih al. 2016 Schulman man et al. 2017).
(sz)). To enhance the robustness of the policy in the face of high-
ments the standard reinforcement learning objective with an entropy term H(
Consequently, the parameters of policy function # and value function 0g are updated by:
0c+1 0, + a aV log ss) + (1)
(1)
(o(1+1) = Gyt aV.... (r+ V(s,))²
(2)
where a is learning rate, H is the entropy policy स, and 3 controls the trade-off between exploita- tion and exploration.
2.3.2 SAC
Though A2C with maximum entropy objective improves the stability of training, it suffers from poor sample efficiency. In contrast, SAC (Haarnoja et al.) 2018) uses a replay buffer to reuse past experiences to reduce sample complexity. To this end, SAC maintains soft Q-function Q (t, ae) and a policy function no (a 8t), where á and A are the parameters for these networks respectively. The soft Q-function parameters can be optimized with the objective of soft Bellman residual:
jQ( (B D
(Q((s, az) (r(st, + 7Bs4i1up (V((++1)))))
(3)
where the parameters Ø are obtained as an exponentially moving average of Φ, and the soft state value function V is defined below following the SAC for discrete action settings (Christodoulou
2
V(s:) = n (x.))(())))))) (Q)) - (000)))))330o((((t log(m(s:)))
The policy parameters are updated towards the exponential of the new Q-function with the KL- divergence objective, and it can be further transformed to the following form for the discrete action
settings:
( = EE,DD [st) [3 (og((T - (s))
(5)
In practice, SAC maintains two soft Q-functions Q and Q and substitutes the soft Q-functions in equation 3 and equation 5 with min(Q01, Qa2) to mitigate bias (Fujimoto al. 2018).
4

---

Published as a conference paper at ICLR 2023
2.4
ENTROPY-BASED REWARD FUNCTIONS
We introduce two classes of reward functions from the angle of syntactic heuristics of training data and the theory of information bottleneck (2.4.2).
2.4.1 1
2.4.1
OBSERVATION ENTROPY
Although there is no consensus what the best in-domain data for generalization, experiments (Adila & Kang 2022) find models latch on syntactic heuristics, like the overlap of words between in-
domain and out-of -iiistrriitiiitoo sentences make predictions. |Ruder & Plank (2077) demonstrates extracting word entropy as heuristic features to select training data favors domain adaptation in
NLP. Based on these findings, we follow classic count-based methods (Song al. 2012 Ruder & Plank 2017 Parcheta et al. 2018: Tevet & Berant 2020), or N-grams, as an indicator of how good the selected data is. Specifically, we apply Shannon Entropy (Shannon 1948). Rényi Entropy [Rényi 1961) and Min Entropy (Smihh (011)) I as reward functions in our reinforcement learning framework. All entropy measures are computed on word n-gram relative frequency on all sentences in the dataset.
For set G with M sentences, and each sentence z; containing Jj words, we define the empirical set entropy as the sum of n-gram entropy:
M
H(G)
Σ h(zgn)
M(gg))
log Dogp(a ta
where p(ar)t"=)) is the relative frequency of n-gram from word j to + n of sentence T; and parameter of Rényi Entropy controlling the order of entropy. Especially, when approaching 1, Rényi Entropy is equivalent as Shannon Entropy; when a approaching infinity, Rényi Entropy converges Min Entropy.
For set G with M training examples, we define the k-th order interpolated set entropy as linear combination of n-gram entropy from until k, weighted X, where Σ Xn 1. For example, if & = 3, then it combines unigram, bigram, and trigram set entropy with weight ' X2. and Ag. respectively:
M
H'G Σ Am
h(z;n),
(6)
We use the 2-nd order interpolated set entropy our default setting in the following sections.
2.4.2 PREDICTION ENTROPY
From the information theory perspective, the Information Bottleneck (IB) principle indicates the mutual information between the input of neural network and its latent representation needs to be well-compressed to generalize well on out-of-domall data (Tishby et al., 2000; Tishby & Zaslavsky, 2015). Specifically, IB seeks obtain a latent representation Z such that the mutual information between input X' and 2, denoting as [(x; 2) is minimized, and the mutual information between Z and output y, denoting I(Y; 2), ), maximized. Formally, is implemented by minimizing the following Lagrangian:
minimize (I(X; XI()2
Intuitively, the smaller mutual information I(X; 2) is, the better 2 compresses Y. the less likely Z learns spurious correlations with X, the more robust representation Z is. However, since is high dimensional, the exact computation of mutual information I(X'; Z) is intractable. Following ZZao al. 2020)) that bounds I(X;; Z) with the prediction entropy H()),, approximately minimize I(X: 2) 2)br by empirically caleulating the prediction entropy H()):
5

---

Published as a conference paper at ICLR 2023
auto
beautyy
Food
instruments
office
computer
tools
phones
grocery
jewelry
outdoor
avg
ALL
56.48
54.03
56,02
59.15
56.84
55.17
53.95
$2.79
$3.06
56.63
55.81
55.63
RAND
57.07
56.93
59.55
59.28
56.16
58.04
53.77
54.81
57.92
57.30
56.89
MTL
59.46
55.89
60.73
61.88
61.87
56.20
62.50
53.93
57.86
58.98
58.23
58.86
PLM
63.45
52.35
68.37
56,09
73.48
34.34
22.32
51.38
49.06
76.09
56.17
35.04
COS
56.67
VPG
73.43
62.97
77.31
79.42
78.41
63.04
81.25
61.26
70.78
73.29
70.11
71.94
AZC-DE
77.79
63.60
81.87
84,30
83.02
67.86
86.61
62.16
72.93
76.75
74.43
75.82
A2C-PE
81,20
83,96
82.07
86,
62.69
73.80
76.71
75.03
75.95
SAC-OE
78.39
66.83
81.58
83.43
82.21
68.37
87.50
61.01
73.36
76.85
75.03
75.87
SAC-PE
78.53
67.00
81.98
84.37
83.96
68.46
87.50
62.99
73,95
76.95
75.22
76.19
4%
19.07
11.11
21.02
22.26
22.09
11.65
25.00
9,06
14.84
17.49
16.99
17.33
Table 1: Sentiment analysis accuracy [96] on amazon unprocessed domains. Baselines PLM (Ma et al. 2019).
COS (Aharoni & Goldberg 2020) and VPG (Liu et al. 2019) use test/target domain data of each column, while methods outperform all of them without using any target domain knowledge. Last row: absolute improvement hetween SAC-PE and best domain generalization method MTL (Blanchard al. 2027
2) I(X:D) H(O) -((())) -H())
(8)
|
H() ~ -
Σ
Ps (, loglloggg log (z.;))
(9)
where Py (m;;; 0) is the predicted probability of label V/ of sample 2i. given the model 0, and |Y]
the set of all labels. Adopting this observation into our context, minimize I(X; 2) using
-H()) as the reward select training data within mini-batch that can learn the optimal latent representation for out-of-distribution generalization.
3 EXPERIMENTS
We describe our experimental details and demonstrate that MERRI. improve baselines in three NLP applications among various out-of-distribution domains, including two classification tasks, senti- ment analysis, named entity recognition, and one generation task of language modeling, without any out-of-domain knowledge. For each task, we experiment with two reinforcement learning algo- rithms to train the data selector, well as three reward functions, i.e. A2C-OE denotes A2C with entropy regularization rewarded by Observation Entropy. We give a list of hyperparameters used in MERRL in appendix A.2.
3.1
NLP EXPERIMENTS
Baselines We compare our methods with six baselines: 1) ALL The models are trained on all in-domain training data; RAND The models are trained on randomly selected 50% in-domain data; 3) MTL Marginal transfer learning by Blanchard et al. (222)) a domain generalization frame- work using kernel methods to augment feature space. 4) PLM (Ma et al. 2019) that uses the large pretrained language model BERT (Devlin al. 8018b) to learn domain classifier and seleet data according to probability given by the domain classifier. 5) COS (Aharoni & Goldberg 2020) that uses cosine distance measure the distance between an in-domain sentence and the centroid of a target domain cout-of-distribution domain), and select sentences close to the target domain. 6) VPG (Liu et al., 2019) that uses the vanilla policy gradient method choose data from target distribution that resembles in-domain distribution. To be noted, PLM, COS and VPG are all data selection methods requiring the usage of out-of-domain data, while ALL, RAND, MTL and all our methods do not use any out-of-domain knowledge. For the training data size, ALL and MTL use all in-domain training data only; VPG and our methods choose roughly 50% in-domain training data (complete data statistics in Appendix A.4), and we control PLM & COS which both require pre-defined selected data size to select 50% in-domain data.
Sentiment Analysis We use the Amazon product review dataset BBlitzer et al. 2007 for the senti- ment analysis task. Specifically, we use the processed labeled domain data (books, dvd and kitchen) to train task model and the unprocessed 21 domains as test data. We CNN classifier (Kim

---

Published as a conference paper at ICLR 2023
politics
science
music
literature
A1
WikrText-2
Penn Treebank
ALL
26.49
19.84
12.26
16.38
13.92
IWSLT"17
Bio 21
IWSLT
Bio 21
RAND
17.47
328.23
254.64
147.03
117.17
MTL
28.47
22.47
13.49
18.97
15.68
RAND
515.22
456.78
234.14
157.39
PLM
26.81
22.31
14.04
18.29
16.00
PLM
$54,36
441.29
233.80
154,30
12,99
19.05
14.49
COS
410.15
314.81
190.54
169.16
VPG
28.74
22.65
12.68
19.24
15.81
VPG
311.76
229.53
143.45
88.86
OZC-OE
24.29
23.48
14.32
20.17
T6.02
A2C-
228.96
152.38
134.87
69.33
A2C-PE
29.52
23.34
15.02
21.07
16.11
A2C-PE
186.09
131.66
132.64
73.08
SAC-OF
29.80
23.91
15.35
21.22
16.78
SAC-OE
198.47
137.85
137.03
70.03
SAC-PE
29.90
23.95
15.42
21.43
16.95
SAC-PE
182.17
129.78
130.48
69.15
Table 2: Left: NER Fl-scores. Right: Language modeling perplexity scores two test domains. First row: source training domain; Second row: test domains. Results are averaged three runs.
camera
comptr
magazns
video
toys
train
VPG
4543
5123
4399
8207
3826
28908
SAC-OE
4490
5076
4298
8108
3739
29538
Table 3: OOV of VPG selected data (Liu ct 2019) and SAC selected data test domains of amazon product review dataset. Last column: training vocab the selected set.
2014) as the sentiment analysis model and pre-train the CNN classifier for two epochs following Liu al. (2019) for fair comparison. Table [ shows the results averaged over five random seeds. Our methods outperform all baselines on all unprocessed amazon domains. It is worth noting that even with the test domain knowledge, baselines PLM or COS fail to select the "right" data for specific do- mains (e.g. tools, computer), on the contrary, our methods can indiscriminately select effective data for all domains. SAC-PE gains an average improvement of 17.33 over the domain generalization baseline MTL which does not out-of-domain knowledge either.
Named Entity Recognition We use the CoNLL2003 English NER dataset (Sang & Meulder, 2003) as in-domain training set and the five domains from CrossNER dataset (Liu et al. (0200) as test sets, which has specialized entity categories for each domain. We finetune the pretrained BERT model (Devlin al., (2088)) on source training set by adding a linear layer on top of the hidden- states output of the last layer and then report the Fs-ccorss on five test sets in left of Table 2 SAC outperforms A2C across all domains and SAC-PE improves the test score on music domain up to 14.3% compared to MTL.
Language Modeling We experiment with two moderate size datasets WikiText-2 Merity et al. 2016) and Penn Treebank. Our bascline is Transformer language model /Vaswani al.) 2017) trained with default hyper-parameters from scratch. The RL loop in Figure (a) initializes the language model be the checkpoint of the pre-trained transformer model. As for evaluation, we report perplexity scores on datasets from different domains, the English side of IWSLT'17 (TED talk) and the English side of WMT Biomedical's 21. The baseline transformer model and language models trained on selected data are updated by the fairseq toolkit (Ott et al. (019)) and stopped until the in-domain validation perplexity score does not improve for 5 epochs. The evaluation results are shown the right of Table 2] The perplexity on two test domains has been largely improved and there is at most 43.4% (decrease from VPG-229.55 to SAC-PE-129.78) relative improvement in the biomedical domain with WikiText-2 in-domain data.
3.2 ANALYSIS
SAC VS. A2C We plot the learning curves of three reinforcement learning algorithms on the left of Figure [ The average reward of SAC is significantly higher than A2C with entropy regularization (shortened as A2C) and VPG. SAC and A2C both converge at around 10000 timesteps, while VPG converges at around 20000 timesteps. Comparing A2C and VPG, it is clearly shown that A2C has a smaller variance than VPG. In short, SAC is the most effective algorithm among the three, and A2C can reduce variance compared to VPG. Particularly, with limited training time budget (e.g. training with 5000 timesteps). SAC can lead to the best performance in training set optimization, which matches empirical results.
7

---

Published as a conference paper at ICLR 2023
2200
SAC A2C
200
iwsit
VPG
bio
Test perplexity
400
Rewards
150
300
250
200
150
5000 xodpo1 Number of Timesteps
1900
Batch size
Figure 2: Left: Leaarnig curve of three reinforcement learning algorithms in three random seeds in the NER task. Right: Smaller batch size B. results in better test perplexity test
Test domain: magazines
PPO selected
VPG selected
2-D visualization of sentence embeddings of the selected dataset by VPG (Liu et al. 2019 (blue) and SAC-OE (red). Middle: SAC-OD OC-OE covers of the test domain, e.g. yellow Right: SAC-OE has larger convex hull volume than VPG.
Figure 3:
Time In theory, training MERRL and training an NLP model by the selected data takes Toudget/ times compared to simply training NLP model using all data from scratch, where 'I is the time steps in Markov Decision Process, or the number of batches, and Toudget parameter of reinforcement learning training steps. Intuitively, Thudget/ 'T is the number of epochs MERRL training, and larger Towdget or smaller will lead to longer training time. In practice, training with all in-domain data in sentiment analysis takes 131 seconds while selecting data with 2000 and T 140) on one Tesla V100 GPU, which is
pre-defined hyper-
SAC-OE takes 1394 seconds (Toudget roughly ten times faster than the bascline VPG to achieve similar average reward. That being said, the problem of computational cost for MERRL remains to be solved and we leave this as future work.
Batch size Unlike previous applications of reinforcement learning in NLP YYon et al. 2020 Fang al. 2017 Wu et al. 2018) which give reward to a single sample/sentence, our reward function measures the informative level of whole set of data. In this case, the observation (state) space is no longer vector, but a batch of vectors. Thus, the batch size [Be] is newly introduced hyper- parameter in our subset optimization problem that affects both action space and state space. While previous work uses larger batch size (BB,1 Z 2000) to improve the stability of reinforcement learning training (Yoon et al.] 2020 McCandlish et al.) 2018), we find that training set optimization can benefit from smaller batch size than large batch size when total training step Toudget is fixed, as shown in the right of Fig 2 The reason be related to designed state space, which is not single vector but a batch vectors so that a larger batch size can directly enlarge the action space into 2 Be| and make the training harder.
Visualization We plot the t-SNE 2D visualizations using the data selected from the training source domains (books, DVD and kitchen) by VPG (Liu et al.) 2019) (blue) and by SAC-OE (red), as well surprising (unknown) test domain (magazines, green dots). We embed each sentence using
8

---

Published as a conference paper at ICLR 2023
the sentence-transte transformer tool (Reimers & Gurevych| 2019). In Figure the middle plot shows the coverage of selected data from SAC-OE (3361 sentences, including 53.3% sentences not overlapping with VPG). While similar in dataset size, blue dots more densely spread, especially the several dense clusters formed by blue points in the bottom part of the left plot. On the contrary, red dots cover more test domain areas than blue dots, especially in those yellow highlighted areas. To gain more intuition, we draw the convex hull (Barber et al. 2013)) for red dots and blue dots respectively, shown on the right side. The red circle encloses the blue circle after removing the outliers from both sets. Furthermore, we compute the out-of- vocabuulayy of all test domains in the Amazon product review dataset and in-domain vocabulary size of VPG and SAC-OE selected set. In Table SAC- OE has significantly lower OOV size among all test domains, while more in-domain vocabulary than VPG. In summary, we infer SAC-OE selected data has a superior generalization ability than VPG since it is capable of selecting a training set with more diverse vocabulary and wider coverage of semantic space.
RELATED WORK
There have been number of influential work Moore & Lewis (2010); Axelrod et al. (201) Ruderr Plank] (2017) on data selection that significantly contributed to today's s NL.P state-of-the-art More recently. Fan et al. (2017), Feng et al. (2018), Quu et al. (2019). Fang et al. (2017) and Liu itt al. (2199) incorporate reinforcement learning with data selection. Another direction examines the potential of large pretrained language models to select data Yuan al. (2020); Aharoni & Goldberg (020); Ma et al. (2019). These work mainly select the training data close to a given target domain for domain adaptation. In contrast, we aim to enhance the model generalization and increase the accuracy on any arbitrary domain. Furthermore, we advance the existing data selection techniques using A2C and SAC that simultaneously optimize the value (Q) network and the policy network for better convergence and lower variance, resulting in higher prediction accuracy and generality.
Beyond the field of NLP. data selection mentation Volpi et al. (2018); Zhang et al. (2020): et al. (2020) and data generation Ruiz et al. (2018 Qiao et al. (2020); Mishra et al. (2022) have been widely used in vision, medical and gen- eral regression tasks. These methods either utilize augmentation operations on images (e.g, flipping, scaling and cropping), or focus on one specific goal (i.e. image style transfer), or include generative models for adversarial training. Thus, it requires further consideration on how to generalize text data when adapting these methods to NLP tasks. Our method puts an emphasis on both characteristies of text data and general prediction entropy which could be directly generalized to other fields.
Killamsetty et al. (202I); Durga et al. (202I), data aug-
Another relevant and emergent line of work is data pruming, which aims at selecting a minimum subset of training data to reduce training costs [Sorscher et al. (2022); Yang et al. (2022). or to enhance the robustness of the model Kaufmann et al. (2022).
CONCLUSION
We introduce Maximum-Entropy Rewarded Reinforcement Learning (MERRL) with observation entropy and prediction entropy to select effective training data that significantly enhances the gen- eralization capability of NLP models. We performed experiments using sentiment analysis, named entity recognition and language modeling across various domains. Without any knowledge of of distribution domains, our method outperforms the CNN, BERT and transformer baselines. Our perimental results show that modeling sample dependency by increasing data uncertainty enhances learning generalization and prediction accuracy.
ACKNOWLEDGMENTS
We appreciate Amazon Alexa Prize, National Science Foundation (NSF) Award No. 1747728, and NSF CRAFT Award No. 22001 fund this research.
9

---

Published as a conference paper at ICLR 2023
REFERENCES
Dyah Adíla and Dongyeop Kang. Understanding out-of-distributioo:: perspective of data dynam- ics. In (Still) Can't Believe It's Not Better! Workshop at NeurIPS 2021, pp. 1-8. PMLR, 2022.
Roee Aharoni and Yoav Goldberg. Unsupervised domain clusters in pretrained language models.
arXiv preprint arXiv:2004.02100 2020.
Amittai Axelrod. Xiaodong He, and Jianfeng Gao. Domain adaptation via pseudo in-domain data selection. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pp. 355-362, Edinburgh, Scotland, UK., July 2011. Association for Computational antho ogy . orgg/oo-gD1110000
Linguistics. BtRLhttts:://
Bradford Barber, David P Dobkin, and Hamnu Huhdanpaa. Qhull: Quickhull algorithm for com- puting the convex hull. Astrophysics Source Code Library, pp. ascl-1304, 2013.
Gilles Blanchard, Aniket Anand Deshmukh, Ürün Dogan, Gyemin Lee, and Clayton D. Scott. Do- main generalization by marginal transfer learning. J. Mach. Learn. Res., 22:2:1-2::::::::: 2021.
John Blitzer, Mark Dredze, and Fernando Pereira. Biographies, bollywood. boom-boxes and blenders: Domain adaptation for sentiment classification. In Proceedings of the 45th annual meeting of the association of computational linguistics, pp. 440-447, 2007.
Petros Christodoulou. Soft actor-critic for discrete action settings. arXiv preprini 1rXiv:1910007000; 2019.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018a. URL
http://arxiv. www. hbsgg//bs/BBB
04805
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800; 2018b.
S Durga, Rishabh Iyer, Ganesh Ramakrishnan, and Abir De. Training data subset selection for re- gression with controlled generalization error. In International Conference on Machine Learning, pp. 9202-9212. PMLR, 2021.
Yang Fan, Fei Tian, Tao Qin, Jiang Bian, and Tie Liu. Learning what data to learn. arXiv preprint arXiv: 1T0.::77000083355 2017.
Meng Fang, Yuan Li, and Trevor Cohn. Learning how to active learn: deep reinforcement learning approach. arXiv preprint arXhv:108.00238. 2017.
Jun Feng, Minlie Huang, Li Zhao, Yang Yang, and Xiaoyan Zhu. Reinforcement learning for relation classification from noisy data. In Proceedings of the aaai conference on artificial intelligence volume 32, 2018.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning. pp. 1861-1870. PMLR, 2018.
A Hernando, A Plastino, and AR Plastino. Maxent and dynamical information. The European Physical Journal B, 858) 2012.
Edwin Jaynes. Information theory and statistical mechanics. Physical review, 106(4):220, 1957.
Rohan Jha, Charles Lovering, and Ellie Pavlick. Does data augmentation improve generalization in nip? arXiv preprint arXiv:2004.150/1 2020.
Xiaomian Kang, Yang Zhao, Jiajun Zhang, and Chengqing Zong. Dynamic context selec- tion for document-level neural machine translation via reinforcement learning. arXiv preprint N0X::20100.43111 14, 2020.
10

---

Published as a conference paper at ICLR 2023
Amnon Katz. Principles of statistical mechanics: the information theory approach. WH Freeman, 1967.
Maximilian Kaufmann, Yiren Zhao, Ilia Shumailov, Robert Mullins, and Nicolas Papernot. Efficient adversarial training with data pruning. arXiv preprint arXiv:2207.0064) 2022.
Krishnateja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, and Rishabh Iyer. Glister: Generalization based data subset selection for efficient and robust learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 8110- 8118, 2021.
Yoon Kim. Convolutional neural networks for sentence classification. CoRR, aass//00..5000 2014.
URL http://arxiv org/ wwwp///..... 5882]
Katrin Kirchhoff and Jeff Bilmes. Submodularity for data selection in machine translation. In Proceedings of the 2014 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 131 2014
Miaofeng Liu, Yan Song, Hongbin Zou, and Tong Zhang. Reinforced training data selection for do- main adaptation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1957-1966 Florence, Italy, July 2019. Association for Computational Linguis- URLpattss http:////cccaattoooooooor///////
tics. doi: 10.18653/v 1/P19-119,
1189
Zihan Liu, Yan Xu, Tiezheng Yu, Wenliang Dai, Ziwei Ji, Samuel Cahyawijaya, Andrea Madotto, and Pascale Fung. Crossner: Evaluating cross domaan named entity recognition. CoRR, abs/2012.04373., 2020.
URL http:: org abs/ 2012. 04373
Xiaofei Ma, Peng Xu, Zhiguo Wang, Ramesh Nallapati, and Bing Xiang. Domain adaptation with BERT-based domain classification and data selection. In Proceedings of the 2nd Workshop
Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), pp. 76-83, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10018663/11/11--11099
URL
Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAl Dota Team. An empirical model of 2018.
large-batch training. arXiv preprint
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint 1rXiv:1609.090 1609.07843, 2016.
Samarth Mishra, Rameswar Panda, Cheng Perng Phoo, Chun-Fu Richard Chen, Leonid Karlin- sky, Kate Saenko, Venkatesh Saligrama, and Rogerio Feris. Task2sim: Towards effective pre-training and transfer from synthetic data. In Proceedings of the IEEE/CVF Conference 2022.
Computer Vision and Pattern Recognition, pp.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforeement learning. nature. 518(7540):529 533, 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-1937. PMLR, 2016.
Robert C. Moore and William Lewis. Intelligent selection of language model training data. In Uppsala, Sweden, July org/
Proceedings of the ACL 2010 Conference Short Papers, pp. 2010. Association for Computational Linguistics. URL pttess://aclanthen P1C-2041
Myle Ott. Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACI-HIL 2019: Demonstrations, 2019.
Zuzanna Parcheta, Germán Sanchis- Trilles, and Francisco Casacuberta. Data selection for nmt using infrequent n-gram recovery. 2018
11

---

Published as a conference paper at ICLR 2023
Fengchun Qiao, Long Zhao, and Xi Peng. Learning to learn single domain generalization. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12556- 12565, 2020.
Chen Qu, Feng Ji, Minghui Qiu, Liu Yang, Zhiyu Min, Haiqing Chen, Jun Huang. and W Bruce Croft. Learning to selectively transfer: Reinforced transfer learning for deep text matching. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, pp- 699-707, 2019.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL https: //aps////rxxvv
org/ /000/199810 10084
Alfréd Rényi. On measures of entropy and information. In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume I: Contributions to the Theory of Statistics, volume 4, pp. 547-562. University of California Press, 1961.
Sebastian Ruder and Barbara Plank. Learning to select data for transfer learning with bayesian optimization. CoRR, abs/ Mes//777.00245 2017. httphttt://arrt org/ aaaaaba//111 /tt///100.000 05246
Nataniel Ruiz, Samuel Schulter, and Manmohan Chandraker. Learning to simulate, arXiv preprint arXiv:1810.02513. 2018.
Erik F. Tjong Kim Sang and Fien De Meulder. Introduction to the conll-2003 shared task: Language- independent named entity recognition. CoRR, CL/3306050, 2003. URL http://arxiv..
hxc/abb/css/03060
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint (rXiv:/506.0244: 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06777 2017.
Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379-42) 423, 1948.
Geoffrey Smith. Quantifying information flow using min- min-entropy. In 2011 Eighth International Conference on Quantitative Evaluation of SysTems, pp. 159-167. IEEE, 2011.
Yan Song, Prescott Klassen, Fei Xia, and Chunyu Kit. Entropy-based training data selection for domain adaptation. In Proceedings of COLING 2012: Posters, pp. 1191-1200, 2012.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. arXiv preprint arXiv:2206, /4486, 2022.
Ingo Steinwart, Don Hush, and Clint Scovel. Learning from dependent observations. Journal of Multivariate Analysis, 175-19:15--194 2009.
Guy Tevet and Jonathan Berant. Evaluating the evaluation of diversity in natural language genera- tion. arXiv preprint (:XXx:2000.009900 2020.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 ieee information theory workshop (inw), pp. 1-5. IEEE, 2015.
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000.
Victor Uc-Cetina, Nicolas Havarro-Guerren Anabel Martin-Gonzalez, Coreeius Weber, and Ste- fan Wermter. Survey on reinforcement learning for language processing. arXiv preprint
14 2021.
12

---

Published as a conference paper at ICLR 2023
Marlies van der Wees, Arianna Bisazza, and Christof Monz. Dynamic data selection for neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 1400-1410, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10. 10.18653/1/D17-1144, URL htttt:://cccaattttttt ogy.
0rq/D17-1147
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. CoRR, abs/1706.03762, 2017.
BRRLhhtp://axx: org/ abs 17766.3766.62..
Riccardo Volpi. Hongseok Namkoong, Ozan Sener, John C Duchi, Vittorio Murino, and Silvio Savarese. Generalizing to unseen domains via adversarial data augmentation. Advances in neural information processing systems, 31, 2018.
Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Philip Yu. Generalizing to unseen domains: A survey on domain generalization. IEEE Transactions Knowledge and Data Engineering, 2022.
Lijun Wu, Fei Tian, Tao Qin, Jianhuang Lai, and Tin-Yan Yan Liu. A study of reinforcement learning for neural machine translation. arXiv preprint 1888::8008088866 2018.
Shuo Yang, Zeke Xie, Hanyu Peng, Min Xu, Mingming Sun, and Ping Li. Dataset pruning: Re- ducing training data by examining generalization influence. arXiv preprint arXiv:2205.09329 2022.
Jinsung Yoon, Sercan Arik, and Tomas Pfister. Data valuation using reinforcement learning. In International Conference on Machine Learning, pp. 10842-10851. PMLR, 2020.
Michelle Yuan, Hsuan-Tien Lin, and Jordan Boyd-Graber. Cold-start active learning through self- supervised language modeling, arXiv preprint arXiv:2010.09533. 2020.
Ling Zhang. Xiaosong Wang, Dong Yang. Thomas Sanford, Stephanie Harmon, Baris Turkbey, Bradford J Wood, Holger Roth, Andriy Myronenko, Daguang Xu, et al. Generalizing deep learn- ing for medical image segmentation to unseen domains via deep stacked transformation. IEEE fransactions on medical imaging, 39(7):2531-2544 2020.
Long Zhao, Ting Liu, Xi Peng, and Dimitris Metaxas. Maximum-entropy adversarial data augmen- tation for improved generalization and robustness. Advances in Neural Information Processing Systems, 33:14435-14444 14447, 2020.
Vitali Zhelezniak, Aleksandar Savkov, April Shen, and Nils Y Hammerla. Correlation coefficients and semantic textual similarity. arXiv preprint arXiv: / /905 07790, 2019.
Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Deep domain-adversaria. im- age generation for domain generalisation. In Proceedings of the AAAÍ Conference on Artificial Intelligence, volume 34, pp. 13025-13032, 2020.
Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegie Mellon University, 2010
A APPENDIX
A.1 NOTATIONS
A.2
HYPERPARAMETERS MERRL
A.3 DETAILS OF THE NER TASK IN INTRODUCTION
We sort 14K training sentences in descending order of entropy and equipartition them into five subsets, where the first subset has the highest unigram set entropy (2069.4), the second subset has the second highest entropy (1078.8) 8.), and so on. We then finetune BERT model (Devlin et al. 20 Sa) on each subset, and compute the Fl scores on six test sets: in-domain CoNLL test set (indomain- test), five distiribibtttoonn domains in CrossNER dataset (Liu et al. 2020): politics (testl), science (test2), music (test3), literature (test4) and AI (test5).
13

---

Published as a conference paper at ICLR 2023
Notation
Meaning
NLP model
X Sr
Training data set
sentence
Training set size
T
disjoint batch index maximum training steps in an episode (epoch)
T
B,
batch
B.
selected batch
encoder
batch state
46
single senteace state
action on batch B, time
ay
single sample batch size
step
[B
policy
reward at time step
R.
total
Q (n..
action value
(*(a))
expected total future return following since time step
b(ss)
baseline function
parameters of policy
(oJ(
os objoctive function policy
L.
epoch number
learning rale
discount factor
Fr
pretrained
(including encoder g)
g.
episode record, including 8t as
samples
cosine dislance of embeddings
number sentences G
sentence length 1--gram entropy
M(;n)
Table Notation table
Hyperparameter
Value
learning rate discount factor
7e
0.99
entropy coefficient
0.001
value function coefficient
0.5
R.SPPrpp epsilon number of steps (Tmudget)
1e -
10000
batch size (NER) batch size (sentiment)
100
500
batch size (language modeling)
500
Table 5: Hyperparameters of MERRL
A.4 N-GRAM ENTROPY ALGORITHM
See algorithm
Algorithm N-gram set entropy
Input: Dictionary dyaas dui, days that stoees unigram entrupy.bigram entropy. trigram entropy for all samples in source training set, butch training samples {(=:) M. } with size M: ratio a, y 6 1).
Output: Reward value of set N N-gram entropy H (G)
1: Initialize (G)
2: Initialize unigram set entropy, bigram set entroy,, trigram set entrpy: (G) 0. h2( 0, Ag) =
3: for all
do
4: Obtain sentence entropy den,[e], ds.[s), dar (s), for
5: Update unigram entropy. bigram entropy, trigram set entropy:
(y(G) = hy(G) + d.,.((s)
h2(G) = h2( + ds.[s]
hs(G) hs(G) der.s5):
6: end For
7: H(G) (G) ())()))))h))+)))))))))))))) +
8: return (G)
14

---

Published as a conference paper at ICLR 2023
A.5
DATA STATISTICS
See table 6
Task
Source 6000
Method
Selected
Sentiment
A2C-RE
3023
(Amazon)
A2C-SE
3287
A2C-ME
3052
SAC-RE
3019
SAC-SE
3361
SAC-ME
3102
NER
14040
A2C-RE
7436
(CoNLL2003)
A2C-SE
7764
A2C-ME
7208
SAC-RE
SAC-SE
7125
SAC-ME
7225
LM
36718
A2C-RE
18207
(wikiText-2)
A2C-SE
18068
A2C-ME
18135
SAC-RE
18230
SAC-SE
18329
SAC-ME
18329
42068
A2C-RE
21156
(pennTreebank)
A2C-SE
21120
A2C-ME
20956
SAC-RE
20969
SAC-SE
21043
SAC-ME
21157
Table 6: MERRL selected data statistics
A.6 OOV OF MERRL SELECTED DATA
We show the full result of OOV of selected data in table 7
Domain
OOV of VPG
OOV of SAC-RE
apparel
2836
2784
auto
1611
1579
baby
3273
3246
beauty
2965
2946
camera
4543
4490
phones
2178
2164
computer
5123
5076
food
2460
2437
gracery
2180
2104
health
4041
4007
jewelry
1635
1618
magazines
4399
4298
music
9033
8975
instruments
824
office
949
939
outdoor
2329
2293
software
4772
4740
sports
4563
4523
tools
151
141
toys
3826
3739
vidco
8207
8108
Table 7: Out-ol-vocabulary of VPG-selected data Liu et (0119) and SAC-selected data on test
domains of amazon product review dataset.
15

---

Published as a conference paper at ICLR 2023
Domain
All
Rand
Threshold
Md
PLM
COS
VPG
A2C-SE
A2C-RE
A2C-ME
SAC-SE
SAC-RE
SAC-ME
apparel
49.43
50.47
50.77
47.65
$1.05
51.25
$1.07
50.62
50.95
50.30
51,66
50.27
50.85
auto
56.48
$7.07
60.35
59.46
63.45
53.94
73.43
77.79
78.39
76.58
78.39
78.53
77.49
haby
50.42
$1.09
48.18
50.80
52.48
50.90
52.81
52.51
52.55
52.37
52.66
53.16
52.95
beauty
54.03
$4.93
55.89
62.97
65.60
65.38
67.00)
65.61
50.30
50.47
50.05
50.04
50.65
50.02
50.08
49.65
49.86
51.10
50.65
phones
52.79
53.77
56.79
53.93
51.58
45.33
61.26
62.16
62.69
61.27
61.01
62.99
61.57
computer
55.17
56.16
57.80
56.20
34.54
41.34
63.04
67.86
67.88
66.25
68.37
68.46
66.73
food
56.02
56,93
65.64
60.73
68.37
68.66
77.31
81.87
81.20
79.74
81.58
81.98
80,50
grocery
53.06
54.81
62.13
57.86
49.06
68.11
70.78
72.93
73.80
71.88
73.36
73.95
76.27
health
49.87
49.65
51.10
49.85
jewelry
56.63
57.92
68.18
58.98
76.09
65.70
73.29
76.75
76.71
75.57
76.85
76.95
76.27
magazines
50.78
50.56
50.76
50.42
50.53
50.30
50.44
50.55
50.97
50.72
50.94
51.94
51.44
music
50.05
50.27
50.85
50.08
49.50
50.07
50.22
50.03
$0.06
49.64
49,97
50.97
50.55
instrs
59.15
59.55
71.98
61.88
56,09
63 90
79.42
84,30
83.96
82.36
83,43
84.37
82.91
office
56.84
59.28
61.87
75.48
63.22
78.41
83.02
82.07
80.98
82.21
83.96
81.25
outdoor
55.81
$7.39
56.54
58.23
70.11
75.03
72.52
75.03
75.22
73.60
software
50.45
50.73
53.30
51.90
52.57
52.25
52.51
52.25
53,24
52.77
sports
51.08
51.13
50.59
49.97
52.17
49.52
50.25
50.32
50.67
52.10
50.78
52.17
49.16
tools
55.95
56.04
78.35
62.50
22.32
41.96
81,25
86.61
86.92
85.11
87,50
87,50
86.30
toys
50.07
50.82
49.70
50.20
52.92
52.80
51.87
50.17
50.90
49.95
51.00
53.25
51.10
video
51.65
50.72
52.30
50.08
51.87
51.27
50.62
50.20
50.80
50.70
50.96
52.47
51.25
Table 8: Sentiment analysis accuracy [%] on unknown domains.
A.7 SENTIMENT ANALYSIS
We show the full test and accuracy result for sentiment analysis in table
16


Reviews:
[{"review_id": "jilrlSg4Fr", "reviewer": "Reviewer_xud3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for optimal training set selection with the goal of maximizing generalization to multiple unknown target domains for NLP tasks. One of the goals of the method is to perform data selection on the training set only without knowledge of any target domain. To achieve this, the paper proposes a Maximum-Entropy Rewarded Reinforcement Learning (MERRL) framework, which seeks to maximize the entropy of the training data. The proposed method rests on the key assumption that the optimal training data for generalizing to many different tasks simultaneously is one that maximizes the training data's entropy and hence is not \"latched on\" to any one specific domain. The authors experiment with two RL algorithms (A2C) and SAC and two measures of entropy: observational entropy (OE) and prediction entropy (PE) and find SAC combined with PE to be be optimum. On four NLP benchmarks the authors show that their proposed approach outperforms several previous approaches for this task, including ones that specifically use the target dataset for optimal training set selection.", "summary_of_the_review": "Overall, I am leaning positive about this paper based on the material presented in it with the caveat that I am not an expert in the field of NLP and hence not familiar with its most current literature. I am aware of closely related works from the field of computer vision that have previously employed ideas of using RL or meta-learning for optimal training set selection/synthesis. I would like to hear from the authors about how they position their method within the context of these existing closely related-works in the vision domain; what their thoughts are on the limitations of using RL for their task; and how generalizable they think their technique would be to other problem domains besides NLP.", "strength_and_weaknesses": "Strengths:\nNovelty: The proposed method addresses an open under-researched and impactful problem of optimal generalization from seen to unseen domains. It presents the novel and intuitive insight of maximizing the entropy of the training dataset to improve generalization on target tasks. The paper also proposes a straight-forward and simple method based on RL to maximize this uncertainty measure. \n\nSoundness: All aspects of the proposed method are theoretically sound and correctly motivated.\n\nClarity: The paper is very-well written and very clear.\n\nExperiments: The proposed method shows significant and impressive performance improvements over all previous methods that it is compared against in the paper, and even those that use target datasets on four different datasets. The authors have also clearly empirically ablated their design choices for the type of RL algorithm and the uncertainty measure to use.\n\nWeaknesses:\n1. One obvious weakness of the proposed method is its high computational cost, which in turn stems from the use of the RL algorithm. RL is widely known to be sample inefficient. The use of the SAC versus the A2C algorithm mitigates this somewhat in the proposed design. While the authors allude to this this briefly in their paper, this weakness could be acknowledged and discussed more in the paper.\n\n2. Another widely used strong baseline method for optimal sample selection with known target datasets is the following: Ren et al., Learning to reweight examples for robust deep learning, ICML 2018. However, the authors have not compared against it. It would have been interesting to see this comparison.\n\n3. The proposed method could be made more impactful if the authors had shown it to also be applicable to other domains besides NLP, for example to vision as well.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "TVZ9p0uANAi", "reviewer": "Reviewer_wzkA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors introduce a novel framework, called Maximum-Entropy Rewarded Reinforcement Learning (MERRL), which can select training data to cover more possible queries that may appear in unknown worlds.", "summary_of_the_review": "The authors only proved the role of entropy in selecting data, but this paper does not elaborate on the motivation and advantages of introducing complex reinforcement learning to train a policy network.", "strength_and_weaknesses": "Strengths:\nIn INTRODUCTION, the authors first proved the relationship between `training set entropy’ and `f1 score and oov’ through experiments, and explained the importance of training set entropy to model generalization. The motivation and logic of the paper is clear.\n\nWeaknesses:\nThe authors only proved the role of entropy in selecting data, but this paper does not elaborate on the motivation and advantages of introducing complex reinforcement learning to train a policy network.\n\nFurther Comments:\n1.\tThe authors use training set entropy as a reward to train a policy network for data selection. How is it different from directly using entropy and selecting data through threshold? What are their advantages and disadvantages?\n2.\tIn the code provided in the supplementary materials, the policy network is first trained on the data set to be selected. Does the policy network need to be retrained on the data set to be selected each time? Considering the generalization of reinforcement learning, this will limit the universality of the algorithm.\n3.\tIn Figure 5, the initial reward of SCA is much higher than A2C and VPG, and the trend is different from that of A2C and VPG. What is the reason for these phenomena?  Is the value of reward related to the quality of data selection?\n4.\tThere are some clerical errors in the paper. For example, in second page, it should be “Pr(“nor”) = 1/2”. It is recommended that the author read the full text to correct such problems.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6geY0WwWeJ", "reviewer": "Reviewer_aHZE", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "\nIn this paper, the authors propose to use a Maximum-Entropy Rewarded Reinforcement Learning framework to select training data for NLP tasks, the goal of which is to maximize generalization. The authors experiment with A2C and SAC and experimental results show that the proposed framework could outperform several baseline approaches.\n\n", "summary_of_the_review": "\nI think the authors propose a novel method of using entropy in selecting data.", "strength_and_weaknesses": "\nStrength: In general this paper is clearly written and easy to follow. The experimental results seem to confirm the validity of proposed method.\n\nWeakness: I suggest the authors include more justifications when selecting certain models of NLP tasks. For example, in sentiment analysis experiments, why the authors choose to use a CNN classifier as the model?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster