Optimizing Data Usage via Differentiable Rewards
Authors: Xinyi Wang
Hieu Pham
Paul Michel
Antonios Anastasopoulos
Graham Neubig
Jaime Carbonell
Venue:  (2020)
Keywords: data selection
multilingual neural machine translation
data usage optimzation
transfer learning
classification

Abstract: To acquire a new skill, humans learn better and faster if a tutor, based on their current knowledge level, informs them of how much attention they should pay to particular content or practice problems. Similarly, a machine learning model could potentially be trained better with a scorer that “adapts” to its current learning state and estimates the importance of each training data instance. Training such an adaptive scorer efficiently is a challenging problem; in order to precisely quantify the effect of a data instance at a given time during the training, it is typically necessary to first complete the entire training process. To efficiently optimize data usage, we propose a reinforcement learning approach called Differentiable Data Selection (DDS). In DDS, we formulate a scorer network as a learnable function of the training data, which can be efficiently updated along with the main model being trained. Specifically, DDS updates the scorer with an intuitive reward signal: it should up-weigh the data that has a similar gradient with a dev set upon which we would finally like to perform well. Without significant computing overhead, DDS delivers strong and consistent improvements over several strong baselines on two very different tasks of machine translation and image classification.

Under review as a conference paper at ICLR 2020
OPTIMIZING DATA USAGE VIA DIFFERENTIABLE REWARDS
Anonymous authors Paper under double-blind review
ABSTRACT
To acquire a new skill, humans learn better and faster if tutor, based on their current knowledge level, informs them of how much attention they should pay to particular content or practice problems. Similarly, machine learning model could potentially be trained better with a scorer that "adapts" to its current learning state and estimates the importance of each training data instance. Training such an adaptive scorer efficiently is a challenging problem; in order to precisely quantify the effect of data instance at a given time during the training, it is typically necessary to first complete the entire training process. To efficiently optimize data usage, we propose reinforcement learning approach called Differentiable Data Selection (DDS). In DDS, we formulate a scorer network as a learnable function of the training data, which can be efficiently updated along with the main model being trained. Specifically. DDS updates the scorer with an intuitive reward signal: it should up-weigh the data that has a similar gradient with a dev set upon which we would finally like to perform well. Without significant computing overhead, DDS delivers strong and consistent improvements over several strong baselines on two very different tasks of machine translation and image classification."
INTRODUCTION
While deep learning models remarkably good at fitting large data sets, their performance is also highly sensitive to the structure and domain of their training data. Training on out-of-domain data can lead worse model performance, while using more relevant data can assist transfer learning. Previous work has attempted to create strategies to handle this sensitivity by selecting subsets of the data to train the model on (Jiang & Zhai, 2007; Wang al.; Axelrod et al., 201 1: Moore & Lewis, 2010), providing different weights for each example (Sivasankaran et al., 2017: Ren et al., 2018 8), or changing the presentation order of data (Bengio et al., 2009; Kumar et al., 2019).
However, there are several challenges with the existing work on better data usage strategies. Most work data filtering criterion or training curriculum rely on domain-speciff knowledge and hand- designed heuristics, which can be sub-optimal. To avoid hand designed heuristics, several works propose to optimize a parameterized neural network to learn the data usage schedule, but most of them are tailored to specific 2018), learning a curriculum learning strategy for NMT (Kumar et al., 2019), and actively selecting data for annotation (Fang et al., 2017: Wu al., 2018). Fan et al. (2018) proposes more general teacher-student framework that first trains a teacher network to select data that directly optimizes development set accuracy over multiple training runs. However, because rumning multiple runs of training simply to train this teacher network entails an n--old increase in training time for n runs, this infeasible many practical settings. In addition, in preliminary experiments we also found the single reward signal provided by dev set accuracy at the end of training noisy to the extent that we were not able to achieve results competitive with simpler heuristic training methods.
cases, such handling noisy data for classification (Jiang et al.,
In this paper, we propose an alternative: a general Reinforcement Learning (RL) framework for optimizing training data usage by training scorer network that minimizes the model loss on the development set. We formulate the scorer network as a function of the current training examples only, making it possible to re-use the model architecture which is designed and trained for the main task.
We will make the code publicly available upon acceptance.

---

Under review as a conference paper at ICLR 2020
Thus, our method requires heuristics and is generalizable to various tasks. To make the scorer adaptive, we perform frequent and efficient updates of the scorer network using a reward function inspired by recent work learning using data from auxiliary tasks (Du et al., 2018; Liu et al., 2019b), which use the similarity hetween two gradients as a measure of task relevance. We propose to use the gradient alignment between the training examples and the dev scorer network, as illustrated in Figure 1. We then formulate our framework as an optimization problem found in many prior works such meta-learning (Finn al., 2017), noisy data filtering (Ren et al., 2018 8), and neural architecture search (Liu et al., 2019a), and demonstrate that our proposed update rules follow direct differentiation of the scorer parameters optimize the model loss on the dev set. Thus we refer to our framework as "Differentiable Data Selection" (DDS).
reward signal for a parametric
We demonstrate two concrete instantiations of the DDS framework, one for more general case of image classification, and the other for more specific case of neural machine transla- tion (NMT). For image classification, we test on both CIFAR-10 10 and ImageNet. For NMT, we focus on a multilingual setting, where we optimize data usage from multilingual corpus to improve the performance on a particular language. For these two very different and realistic tasks, we find the DDS framework brings significant improvements over the baselines for all settings.
Training Data
(x,y)
Scorer y
p(x,y;
Model 0
(c,)
10 Daei)
Figure 1: The general workflow of DDS.
DIFFERENTIABLE DATA SELECTION
2.1 RISK, TRAINING, AND DEVELOPMENT SETS
Commonly in machine learning, we seek to find the parameters 6* that minimize the risk [(, P). the expected value of a loss function f(I, y; 0), where (z,y) are pairs of inputs and associated labels sampled from a particular distribution P(X, ((X,Y))
0" argmin J(0, P) where J)O, E Pe..
Ideally, we would like the risk J() to be minimized over the data distribution that our system sees at test time, ie. Piea (X,, X.Y. Y). Unfortunately, this distribution is unknown at training time, so instead we collect training set Derain = [(xx, ys) 1,.., Numin with distribution Purain Y)- Uniform(Dann and minimize the empirical risk by taking (x, 2) ~ Pumin Y). Since we need a sufficiently large training set Drain to train a good model, is hard to ensure that Pirain (X,, Y) R Pies( Y). In fact, often accept that training data comes from a different distribution than test data. The discrepancy between Parin (X, Y) and Pus((, Y)) manifests itself in the form problems such overfitting (Zhang et al., 2017; Srivastava et al., 2014), covariate shift (Shimodaira, 2000), and label shift (Lipton et al., 2018)
However, unlike the large training set, {(x,, y:) : = I,.... Nevv with distribution Paev Y) which is much closer to (x, Y)2. Since Dave is better approximation of test-time scenario", we can use Daev get reliable feedback to learn to better utilize our training data from Dimin- In particular, we propose to train a scorer network, parameterized by 1b. provide guidance on training data usage to minimize [00 Dov.).
collect relatively small development set Dayy
2.2 REINFORCEMENT LEARNING FOR OPTIMIZING DATA USAGE
We propose to optimize the scorer's parameters & in RL setting. Our environment is the model state 0 and an example (x., 1/). Our RL agent is the scorer network v,, which optimizes the data usage
2 As is standard in machine learning experiments, we make sure that Dae has no overlap with Da or D Details of how we construct the Daw can be found in Appendix A.2 and A.4.
3 For example, in Section 3.2 we would like to use training data from many different languages to improve the performance of particular low-resource language. Here we can gather a small set of Daev from the low resource language, even if we can't I gather a large training set in the language. Moreover, in a domain adaptation setting can obtain small dev set in the target domain, in setting of training on noisy data we can often obtain small clean dey set.
2

---

Under review as a conference paper at ICLR 2020
for the current model state. The agent's reward on picking an example approximates the dev set performance of the resulting model after the model is updated on this example.
Our scorer network is parameterized as differentiable function that only takes as imputs the features of the example (x, y). Intuitively, it represents a distribution over the training data where more important data has higher probability of being used, denoted P(X, Y; t)). Unlike prior methods which generally require complicated featurization of both the model state and the data as input to the RL agent (Fan al., 2018; Jiang et al., 2018; Fang et al., 2017), our formulation is much simpler and generalizable to different tasks. Since our scorer network does not consider the model parameters Or as imput, we update iteratively with the model so that at training step t, P(X, Y; the) provides an up-to-date data scoring feedback for given i,.
Although the above formulation is simpler and more general. requires much more frequent updates to the scorer parameter tb. Existing RL frameworks simply use the change in dev set risk as the regular reward signal, which makes the update expensive and unstable (Fan et al., 2018; Kumar et al., propose novel reward function an approximation to A (I, y) to quantify
2019). Therefore, the effect the training example (r. y). Inspired by Du et al. (2018) (which uses gradient similarity between two tasks to measure the adaptation effect between them, we use the agreement between the model gradient on data (z, y) and the gradient on the dev set to approximate the effect of (zz y) on dev set performance. This reward simply implies that prefer data that moves 0 in the direction that minimizes the dev set risk:
R(x, y)
Aavev (2, v) (; [ Daev)
According to the REINFORCE algorithm (Williams, 1992), the update rule for & is thus dt+1 f 1/2 Vol(x, y: 02-1) (0,, D (xx
R(2,9)
The update rule for the model simply
0₂ <- - V..
(4)
For simplicity of notation, we omit the learning rate term. Full derivation can be found in Appendix A.I. By alternating between Eqn. 4 and Eqn. 3, we can iteratively update 0 using the guidance from the scorer network, and update & to optimize the scorer using feedback from the model.
Our formulation of scorer network P(X, Y: has several advantages. First, it provides the flexibility that we can either sample a training instance or equivalently scale the update from the training instance based on its score. Specifically, we provide algorithm under the DDS framework for multilingual NMT (see Sec. 3.2), where the former is more efficient, and another more general algorithm for image classification (see Sec. 3.1), where the latter choice is natural. Second, it allows easy integration of prior knowledge of the data, which is shown to be effective in Sec. 4.
2.3 DERIVING REWARDS THROUGH DIRECT DIFFERENTIATION
In this section, show that the update for the scorer network in Eqn. 3 can be approximately derived as the solution of bi-level optimization problem (Colson et al., 2007), which has been applied to many different lines of research (Baydin al., 2018; Liu al., 2019a; Ren et al., 2018).
Under our framework, the scorer samples the data by (x. ~ P(X, Y; and & will be chosen so that 6 that optimizes [(, P(X, Y))) will approximately minimize J [e, Parv (X, Y)):
argmin J(0" (1)), Daw) where 0* (v) argmin P(X,x.)) [A(2, 0)]
(5)
The connection between and in Eqn. shows that J(O,, Dae.) differentiable with respect to ab.
Now we can approximately compute the gradient V v Daw)) as follows:
V Dam) JO,, Dam))
V.De(w)
(chain rule)
JO,, Dan)) T. V. (0--1 (o.(0.-1.)))
(substitute O, from Eqn 4)
F Vo, D
V. (VoJ(O:-11,)) v))
(assume V ve R 0)
(6)
--
VoJ(Or, Dav)) (1
-BB..NP(X...0) (Vo)(O,, Dam) Vol(z, 01)) V. log (7,
3

---

Under review as a conference paper at ICLR 2020
Here, we make a Markov assumption that V O, assuming that at step t. given 0 do not care about how the values of & from previous steps led to f1-1. Eqn. 9 leads to a rule to update el using gradient descent, which is exactly the same as the RL update rule in Eqn. 3.
Note that our derivation above does not take into the account that we might use different optimizing algorithms, such as SGD Adam (Kingma & Ba, 2015), to update 0. We provide detailed derivations for several popular optimization algorithms in Appendix A.1.
One potential concern with our approach is that because optimize 1/ directly on the dev set using J(,,, ), we may risk indirectly overfitting model parameters O, by selecting a small subset of data that overly specialized. However this because (1) the influence of dh on the final model parameters Og is quite indirect, and acts as "bottleneck" which has similarly proven useful for preventing overfitting in neural models Grézl et al. (2007), and (2) because the actual implementations of DDS (which we further discuss in Section 3) only samples a subset of data from Durain at each optimization step, further limiting expressivity.
do not observe this problem in practice, and posit that
CONCRETE INSTANTIATIONS OF DDS
We now turn to discuss two concrete instantiations of DDS that use in our experiments: more generic example of classification, which should be applicable to a wide variety of tasks, and a specialized application to the task of multilingual NMT, which should serve an example of how DDS can be adapted to the needs of specific applications.
3.1 FORMULATION FOR CLASSIFICATION
Algorithm 1: Training classification model with DDS.
Input :Desin Daev
Output: parameters 0* Initializer 6g and v/o
for t 1 to num troin de
Sample B training data points æ. ~ Uniform( Duain)
Sample B validation data points x 1/ Uniform (Davv) b Optimize 0
Update 02 <- GradientUddate (c-1. En p(as ti- Vo ((:)))))
b Evaluate O2 on Dáne
Tee dg + 3 EØ= Vol(z')
Optimize
Let dg 1 ΣD [a5 2z V. logp(z.
Updute v4 GraddiettUpdate((ee-.ett
end
Algorithm 1 presents the pseudo code for the training process on classification tasks, using the notations introduced in Section 2. The main classification model is parameterized by 9. The scorer p(X, Y;v) an identical network with the main model, but with independent weights, i.e. p(X, v)) does not share weights with 0. For each example in a minibatch uniformly sampled from Dirain this DDS model outputs a scalar from the data z;. All scalars are passed through a softmax function to compute the relative probabilities of the examples in the minibatch, and their gradients accordingly when applied to 0. Note that our actual formulation of p(X, Y; ()) does not depend on Y, but keep Y in the notation for consistency with the formulation of the DDS framework. Note that we have two gradient update steps, one for the model parameter 8 in Line 5 and the other for the DDS scorer parameter in Line 8. For the model parameter update, can simply use any of the standard optimization update rule. For the scorer v/s we use the update rule derived in Section 2.3.
scaled
Per- Per-Exa Gradient. As seen from Line 7 of Algorithm 1, well from Eqn. 13, DDS re- quires us to compute Vol(x,, yr, y45 01-1), i.e. the gradient for each example in a batch of training data. This operation is very slow and memory intensive, especially when the batch size is large, egg. our experiments on ImageNet use a batch size of 4096 (see Section 4). Therefore, we propose an efficient approximation of this per- per-example gradient computation via the first-order Taylor expansion of ((v,, yı5 01-1). In particular, for any vector TR! with sufficiently small O, we have: Val(xs, ye; 1) ~ (e(x:, Merr,y:: 0. + ev) E E(s, 0)-
4

---

Under review as a conference paper at ICLR 2020
Eqn 7 can be implemented by keeping shadow version of parameters 0t-1, caching training loss ((2,,, ye: 01-1), and computing the new loss with 0 + ev. Here, is dg as in Line of Algorithm 1.
3.2 FORMULATION FOR MULTILINGUAL NMT
Next we demonstrate application of DDS to multilingual models for NMT, specifically for improving accuracy on low-resource languages (LRL) (Zoph et al., 2016; Neubig & Hu, 2018). In this setting, assume that have particular LRL S that we would like to translate into target language T, and we additionally have multilingual corpus Diin that has parallel data between n source languages (51.S5, .... Sn) and target language T. We would like pick parallel data from any of the source languages to the target language to improve translation of a particular LRL S, so we assume that Day exclusively consists of parallel data between S and T. Thus, DDS will attempt to select data from Irinn that improve accuracy S-to-T translation as represented by D
Algorithm 2: Truining multilinuual NMT with DDS.
Input Duaii K: number of data to train the NMT model before updating vi; E: number of updates
for 01.0g: discount factors the gradient
Output: The converged NMT model 6* Initialize thu, 9,
b Initialize the gradient of each source language grod[S.] 0 for
while not converged do
X.Y +- load data(th, DDuis: K)
Train the NMT model
for 24.M y in X, Y.Y do
8, + GradientUpdate (f1-1. Va, Aaar, yt (c-1))
grad[S.] + on x grad + or x Vo... Mi 0.-)))
end
b Optimize 1/-
for in do
sample B data pairs from Down
n
dg - } E Din1 [grad[S.] grad[S) log s/e + GradientU (ddate (12-- 11
end
end
To make training more efficient and stable in this setting, we make three simple modifications of the main framework in Section 2.3 that take advantage of the problem structure of multilingual NMT. First, instead of directly modeling p(X, Y; v)) we assume a uniform distribution over the target sentence Y, and only parameterize the conditional distribution of which source language sentence to pick given the target sentence: p(X ly; v) This design follows the formulation of Target Conditioned Sampling (TCS; Wang & Neubig (2019)), an existing state-of-the-art data selection method that uses a similar setting but models the distribution p(Xly) using heuristics. Since the scorer only needs to model simple distribution over training languages, network. Second, we only update v after updating the NMT model for fixed number of steps. Third, we sample the data according p(Xly; v to get Monte Carlo estimate of the objective in Eqn. 5. This significantly reduces the training time compared to using all data. The pseudo code of the training process in Algorithm 2.
a fully connected 2-layer perceptron
4 EXPERIMENTS
We now discuss experimental results on both image classification, an instance of the general classifi- cation problem using Algorithm 1, and multilingual NMT using Algorithm 2.
4.1 EXPERIMENTAL SETTINGS
Data. We apply our method on established benchmarks for image classification and multilingual NMT. For image classification, we use CIFAR-IO (Krizhevsky, 2009) and ImageNet (Russakovsky et al., 2015). For each dataset, we consider two settings: reduced setting where only roughly 10% of the training labels are used, and a full setting, where all labels are used. Specifically, the reduced setting for CIFAR-10 uses the first 4000 examples in the training set, and with ImageNet, the reduced setting uses the first 102 TFRecord shards as pre- processed by Kornblith et al. (2019). We use the size of 224 224 for ImageNet.
5

---

Under review as a conference paper at ICLR 2020
Table 1: Results for image classification accuracy (left) and multilingual MT BLEU (right). For MT, the
statistical significance indicated with (p < 0.005) and i (p 0.0001).
CIFAR-10 (WRN-28-K)
ImageNet (ResNet-50)
Methods
Methods
aze
bel
glg
slk
4K. À
Full, à 10
10%
Foll
Uniform
10.31
17.21
26.05
27.44
Uniform
95.55 82.6010.117
56.36/79.45
76.51/93.20
SPCL
9.07
16.99
23.64
21.44
SPCT.
81.09+0.22
93.66-0.11
Related
10.34
15.31
27.41
25.92
Waigh
79.61=0.55
14.11=0.18
TCS
11.18
16.97
27.28
27.72
MentorNet
83.11:0062
94.92=0.34
DDS
10.74
17.24
27.32
28,20"
retrained DDS
DDS
83,63+ 0.29
85.56=+0.20
96.31= 0.13
97.91-0.12
56.81/79.51
77.23/9.577
TCS+DDS
11.84
17.74"
27.78
27.74
For multilingual NMT, we use the 58-language-to-Eng TED dataset (Qi et al., 2018). Following prior work (Qi al., 2018; Neubig & Hu, 2018: Wang al., 2019b), we evaluate translation from four low-resource languages (LRL.) Azerbaijani (a za))), Belarusian (be 1), Galician (glg), and Slovak (slk) to English, where each is paired with similar high-resource language Turkish (tur), Russian (rus), Portugese (por), and Czech (ces) (details in Appendix A.3). We combine data from all languages, and use DDS to optimize data selection for each LRL.
Models and Training Details. For image classification, on CIFAR-10, we use the pre-activation WideResNet-288 (Zagoruyko & Komodakis, 2016), with width factor k = for the reduced setting and k 10 for the normal setting. For ImageNet, we the post-activation ResNet-50 (He et al., 2016). These implementations reproduce the numbers reported the literature (Zagoruyko & Komodakis, 2016; He et al., 2016; Xie et al., 2017), and additional details can be found in Appendix A.4.
For NMT, we use a standard LSTM-based attentional baseline (Bahdanau et al., 2015), which is similar previous models used in low-resource scenarios both this dataset (Neubig & Hu, 2018; Wang et al., 2019b) and others (Sennrich & Zhang, 2019) due to its relative stability compared to other options such the Transformer (Vaswani al., 2017). Accuracy measured using BLEU score (Papineni et al., 2002). More experiment details are noted in Appendix A.2.
Baselines and Our Methods. For both image classification and multi-lingual NMT, we compare the following data selection methods. Uniform where data is selected uniformly from all of the data that we have available, is standard in training models. SPCL (Jiang et al., 2015), a curriculum learning method that dynamically updates the curriculum to focus more on the "easy" training examples based on model loss. DDS, our proposed method.
For image classification, we compare with several additional methods designed for filtering noisy data on CIFAR-10, where we simply consider the dev set as the clean data. Batch Weight (Ren et al., 2018 8), method that scales example training loss in batch with locally optimized weight vector using a small set clean data. MentorNet (Jiang et al., 2018), a curriculum learning method that trains a mentor network to select clean data based on features from both the data and the main model. For machine translation, we also compare with two state-of-thed th-tth-artt heuristic methods for multi-lingual data selection. Related where data is selected uniformly from the target LRL and a linguistically related HRL (Neubig & Hu, 2018). TCS, a recently proposed method of "target conditioned sampling", which uniformly chooses target sentences, then picks which source sentence to use based on heuristics such as word overlap (Wang & Neubig, 2019). Note that both of these methods take advantage of structural properties of the multi-lingual NMT problem, and do not generalize to other problems such as classification.
DDS is flexible framework to incorporate prior knowledge about the data using the scorer network, which can be especially important when the data has certain structural properties such as language or domain. We test such setting of DDS for both tasks. For image classification, we use retrained DDS, where we first train a model and scorer network using the standard DDS till convergence. The trained scorer network can be considered as a good prior over the data, so we use to train the final model from scratch again using DDS. For multilingual NMT, experiment with TCS+DDS, where we initialize the parameters of DDS with the TCS heuristic, then continue training.
4.2
MAIN RESULTS
The results of the baselines and our method are listed Table 1. First, comparing the standard baseline strategy of "Uniform" and the proposed method of "DDS" we can see that in all 8 settings

---

Under review as a conference paper at ICLR 2020
bookshopp
comboy
Dood
Europson qallinge
cuphut crested cockateo
102202
0.03302
blacd
Wwish speniel
3.0G207
2.06816
0407424
0:07756
0.12:30
DIL4020
Figure 3: Example images from the ImageNet and their weights assigned by DDS,
DDS improves over the uniform bascline. This is strong indication of both the consistency of the improvements that DDS can provide, and the generality - it works well in two very different settings. Next, find that DDS outperforms SPCL by large margin for both of the tasks, especially for multilingual NMT. This is probably because SPCL weighs the data only by their easiness, while ignoring their relevance the dev set, which especially important in settings where the data in the training set can have very different properties such as the different languages in multilingual NMT.
DDS also brings improvements over the state-of-the-ar intelligent data utilization methods. For image classification, DDS outperforms MentorNet and Batch Weight on CIFAR-10 in all settings. For NMT, in comparison Related and TCS, vanilla DDS performs favorably with respect to these statc-of-the-art data selection baselines, outperforming each in out of the 4 settings (with exceptions of slightly underperforming Related on glg and TCS on aze). In addition, we see that incorporating prior knowledge into the scorer network leads to further improvements. For image classification, retrained DDS can significantly improve over regular DDS, leading to the new state-of-the- result on the CIFAR-I0 10 dataset. For mulitlingual NMT, TCS+DDS achieves the best performance in three out of four cases (with the exception of s s1k, where vanilla DDS already outperformed TCS).*
DDS does not incur much computational overhead. For image classification and multilingual NMT respectively, the training time is about 1.5x and 2 x the regular training time without DDS
4.3 ANALYSIS
Raw counts
Image Classification. Prior work on heuristic data selection has found that the model performs better if we feed higher quality or more domain-relevant data towards the end of training (van der Wees et al., 2017; Wang et al., 2019a). Here we verify this observation by analyzing the learned importance weight at the end training for image classification. Figure 2 shows that at the end of training. DDS learns to balance the class distribution, which is originally unbalanced due to the dataset creation. Figure 3 shows that at the end of training, DDS assigns higher probabilities to images with clearer class content from ImageNet.
3139
DDS-normalized
3.855
3.000
Figure 2: Class distributions of CIFAR-10 4K.
These results show that DDS learns to focus on higher quality data towards the end of training.
NMT. Next, we focus on multi-lingus NMT, where the choice of data directly corresponds to picking a language, which has an intuitive interpretation. Since DDS adapts the data weights dynamically to the model throughout training, here we analyze how the dynamics of learned weights.
1.00
1.0
1.00
1.00
tur
9.70
0,70
0.50
0.30
(U.00)
1L20
por
9.25
0.30
11:50
0.25/
9.00
0.001
11.00
1.00
Figure 4: Language usage for TCS +DDS by training step. From left to right: aze, bel, glg, alk.
We plot the probability distribution of the four HRLs (because they have more data and thus larger impact training) over the course of training. Figure 4 shows the change of language distribution
4 'For the NMT significance tests (Clark et al., 2011) find significant gains over the haseline for aze, slk, and bel. For glg the gain is not significant, but DDS-uniform without heuristics performs as well as the TCS baseline. 'The code for multilingual NMT is not optimized, its training time could be reduced further
7

---

Under review as a conference paper at ICLR 2020
for TCS+DDS. Since TCS selects the language with the largest vocabulary overlap with the LRL, the distribution is initialized to focus on the most related HRL. For all four LRLs, the percentage of their most related HRL starts to decrease as training continues. For aze, DDS quickly comes back to using its most related HRI. However, for bel, DDS continues the trend of using all four languages. This shows that DDS is able maximize the benefits of the multi-lingual data by having a more balanced usage of all languages.
3.4
0.4
0.4-
ILG
705
Ct5
0.2
0.0
0.0
100
100
Figure 5: Language usage for DDS by training step. From left to right: aze, bel, glg, slk.
Figure 5 shows a more interesting trend of DDS without heuristic initialization. For both aze and bel, DDS focuses on the most related HRL after a certain number of training updates. Interestingly, for bel. DDS learns to focus on both rus, its most related HRL, and ces. Similarly for s s1k, DDS also learns to focus on ces, its most related HRL, and rus, although there is little vocabulary overlap between slk and rus. Also notably, the ratios change significantly over the course of training, indicating that different types of data may be more useful during different learning stages.
5 RELATED WORK
Many machine learning approaches consider how to best present data models. First, difficulty- based curriculum learning estimates the presentation order based on heuristic understanding of the hardness of examples (Bengio al., 2009; Spitkovsky et al., 2010; Tsvetkov al., 2016; Zhang et al., 2016; Graves et al., 2017; Zhang et al., 2018; Platanios et al., 2019). These methods, though effective, often generalize poorly because they require task-specific difficulty measures. On the other hand, self-paced learning (Kumar et al., 2010: Lee & Grauman, 2011) I) defines the hardness of the data based on the loss from the model, but still based on the assumption that the model should learn from easy examples. Our method does not make these assumptions. Closest to the learning to teach framework (Fan et al., 2018) but their formulation involves manual feature design and requires expensive multi-pass optimization. Instead, we formulate which has been successfully applied for a variety of other tasks (Colson et al., 2007; Anandalingam & Friesz, 1992; Liu et al., 2019a; Baydin et al., 2018; Ren et al., 2018).
reward using bi-level optimization,
Data selection for domain adaptation for disparate tasks has also been extensively studied (Moore & Lewis, 2010; Axelrod al., 2011; Ngiam et al., 2018; Jiang & Zhai, 2007; Foster et al., 2010; Wang et al.). These methods generally design heuristics to measure domain similarity. Submodular optimization (Kirchhoff & Bilmes, 2014: Tschiatschek al., 2014) selects training data that are similar to dev set, but the criterion is often based on hand-designed features and the data usage is predefined before training. Besides domain adaptation, selecting also benefits training in the face of noisy or otherwise undesirable data (Vyas et al., 2018; Pham et al., 2018).
Our method is also related to works on training instance weighting (Sivasankaran et al., 2017: Ren et al., 2018; Jiang & Zhai, 2007; Ngiam et al.. 2018). These methods reweigh data based on manually computed weight vector, instead of using a parameterized neural network. Notably, Ren et al. (2018) tackles noisy data filtering for image classification, by using meta-learning to calculate locally optimized weight vector for each batch data. In contrast, our work focuses on the general problem of optimizing data usage. We train parameterized scorer network that optimizes over the entire data space, which can be essential in preventing overfitting mentioned in Sec. 2; empirically method outperform Ren et al. (2018) by large margin in Sec. 4. (Wu et al., 2018: Kumar et al., 2019; Fang et al., 2017) propose RL frameworks for specific natural language processing tasks, but their methods are less generalizable and requires more complicated featurization.
8

---

Under review as a conference paper at ICLR 2020
6 CONCLUSION
We present Differentiable Data Selection, an efficient RL. framework for optimizing training data usage. We parameterize the scorer network differentiable function of the data, and provide an in- tuitive reward function for efficiently training the scorer network. We formulate two algorithms under the DDS framework for realistic and very different tasks, image classification and multilingual NMT, which lead to consistent improvements over strong baselines.
REFERENCES
G. Anandalingam and Terry L. Friesz. Hierarchical optimization: An introduction. Annals OR, 1992.
Amittai Axelrod, Xiaodong He, and Jianfeng Gao. Domain adaptation via pseudo in-domain data selection. EMNLP, 2011.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015.
Atilim Gunes Baydin. Robert Cornish, David Martínez-Rui Rubio, Mark Schmidt, and Frank Wood. Online learning rate adaptation with hypergradient descent. In ICLR, 2018.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In ICML, 2009.
Jonathan H. Clark, Chris Dyer, Alon Lavie, and Noah Smith. Better hypothesis testing for statistical machine translation: Controlling for optimizer instability. In ACL, 2011.
Benoît Colson, Patrice Marcotte, and Gilles Savard. An overview of bilevel optimization. Annals OR, 153(1), 2007.
Yunshu Du, Wojeiech M. Czarnecki, Siddhant M. Jayakumar, Razvan Pascanu, and Balaji Laksh- minarayanan. Adapting auxiliary losses using gradient similarity. CoRR, abs/1812.022224, 2018. URL http:://aaat org/ abs 1812. 02224.
Yang Fan, Fei Tian, Tao Qin, Xiang-Yang Yang Li, and Tie- Yan Liu. Learning to teach. In ICLR, 2018.
Meng Fang, Yuan Li, and Trevor Cohn. Learning how to active learn: A deep reinforcement learning approach. In EMNLP, pp. 595-605, 2017.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML. 2017.
George Foster, Cyril Goutte, and Roland Kuhn. Discriminative instance weighting for domain adaptation in statistical machine translation. EMNLP, 2010.
Alex Graves, Mare G. Bellemare, Jacob Menick, Rémi Munos, and Koray Kavukcuoglu. Automated curriculum learning for neural networks. In ICML, 2017.
Frantisek Grézl, Martin Karafíát, Stanislav Kontár, and Jan Cernocky. Probabilistic and bottle-neck features for lvesr of meetings. In ICASSP, volume 4, pp. IV-757. IEEE, 2007.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CPVR, 2016.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
Jing Jiang and ChengXiang Zhai. Instance weighting for domain adaptation in nlp. In ACL, 2007.
Lu Jiang, Deyu Meng, Qian Zhao, Shiguang Shan, and Alexander G. Hauptmann. Self-paced curriculum learning. AAAI, 2015.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data- driven curriculum for very deep neural networks comrupted labels. In ICML, 2018.

---

Under review as a conference paper at ICLR 2020
Diederik P. Kingma and Jimmy Lei Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Katrin Kirchhoff and Jeff A. Bilmes. Submodularity for data selection in machine translation. In EMNLP, 2014.
Simon Kornblith, Jonathon Shlens, and Quoe V. Le. Do better imagenet models transfer better? In CVPR, 2019.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Gaurav Kumar, George Foster, Colin Cherry, and Maxim Krikun. Reinforcement learning based curriculum optimization for neural machine translation. In NAACL, pp. 2054-2002 2061, 2019.
M. Pawan Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. In NIPS, 2010.
Yong Jae Lee and Kristen Grauman. Learning the easy things first: Self-paced visual category discovery. In CVPR, 2011.
Zachary C Lipton, Yu-Xiang Wang, and Alex Smola. Detecting and correcting for label shift with black box predictors. arXiv preprint arXiv:1802.03916, 2018.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: differentiable architecture search. 2019a.
Shikun Liu, Andrew Davison, and Edward Johns. Self-supervised generalisation with meta auxiliary learning. CoRR, abs/1901 108933, 2019b. URL
/atp//1bs///11/..e//// 08933.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In ICLR, 2017.
Robert C Moore and William Lewis. Intelligent selection of language model training data. In ACL, 2010.
Yurii E. Nesterov. A method for solving the convex programming problem with convergence rate (1/k3)) Soviet Mathematics Doklady, 1983.
Graham Neubig and Junjie Hu. Rapid adaptation neural machine translation new languages. EMNLP, 2018.
Jiquan Ngiam, Daiyi Peng, Vijay Vasudevan, Simon Kornblith, Quoc V. Le, and Ruoming Pang. Domain adaptive transfer learning with specialist models. CVPR, 2018.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In ACI,, 2002.
Minh Quang Pham, Josep Crego, Jean Senellart, and François Yvon. Fixing translation divergences in parallel corpora for neural MT. EMNLP, 2018.
Emmanouil Antonios Platanios, Otilia Stretcu, Graham Neubig, Barnabas Poezos, and Tom Mitchell. Competence-based curriculum learning for neural machine translation. In NAACL, 2019,
Ye Qi, Devendra Singh Sachan, Matthieu Felix, Sarguna Padmanabhan, and Graham Neubig. When and why are pre-trained word embeddings useful for neural machine translation? NAACL, 2018.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In ICML, pp. 4331 4340. 2018.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJCV, 2015.
Rico Sennrich and Biao Zhang. Revisiting low-resource neural machine translation: A study. In ACL, 2019.
Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log- likelihood function. Journal of statistical planning and inference, 90(2):227-244, 2000.
10

---

Under review as a conference paper at ICLR 2020
Sunit Sivasankaran, Emmanuel Vincent, and Irina Illina. Discriminative importance weighting of augmented training data for acoustic model training. In ICASSP, 2017.
Valentin I. Spitkovsky, Hiyan Alshawi, and Daniel Jurafsky. From baby steps to leapfrog: How "less is more" unsupervised dependency parsing. In NAACL. 2010.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. In JMLR, 2014.
Sebastian Tschiatschck, Rishabh K. Iyer, Haochen Wei, and Jeff A. Bilmes. Learning mixtures of submodular functions for image collection summarization. NIPS, 2014.
Yulia Tsvetkov, Manaal Faruqui, Wang Ling, Brian MacWhinney, and Chris Dyer. Learning the curriculum with bayesian optimization for task-specific word representation learning. In ACL, 2016.
Marlies van der Wees, Arianna Bisazza, and Christof Monz. Dynamic data selection for neural machine translation. In EMNLP, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pp. 5998-60990 2017.
Yogarshi Vyas, Xing Niu, and Marine Carpuat. Identifying semantic divergences in parallel text without annotations. In NAACL, 2018.
Rui Wang, Masao Utiyama, Lemao Liu, Kehai Chen, and Eicchiro Sumita. Instance weighting for neural machine translation domain adaptation. In EMNLP.
Wei Wang, Isaac Caswell, and Ciprian Chelba. Dynamically composing domain-data selection with clean-data selection by "co-curricular learning" for neural machine translation. In ACL, 2019a.
Xinyi Wang and Graham Neubig. Target conditioned sampling: Optimizing data selection for multilingual neural machine translation. In ACL, 2019.
Xinyi Wang, Hieu Pham, Philip Arthur, and Graham Neubig. Multilingual neural machine translation with soft decoupled encoding. In ICLR, 2019b.
Ronald Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 1992.
Jiawei Wu, Lei Li, and William Yang Wang. Reinforced co-training. In NAACL, 2018.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In CVPR, 2017.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. ICLR, 2017.
Dakun Zhang, Jungi Kim, Josep Crego, and Jean Senellart. Boosting neural machine translation. Arxiv 16/2.06138, 2016.
Xuan Zhang, Gaurav Kumar, Huda Khayrallah, Kenton Murray, Jeremy Gwinnup, Marianna J Martindale, Paul McNamee, Kevin Duh, and Marine Carpuat. An empirical exploration of curriculum learning for neural machine translation. Arxiy, 1811.00739, 2018.
Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. Transfer learning for low-resource neural machine translation. In EMNLP, 2016.
11

---

Under review as a conference paper at ICLR 2020
A
APPENDIX
A.1
DERIVING GRADIENT OF 1 FOR DIFFERENT OPTIMIZERS
First, we rewrite the update rule of 0 in Eqn. 4 to incorporate the effect of its specific optimization algorithm.
For fixed value of vh, [O, vb) can be optimized using stochastic gradient update. Specifically, time step t, we update
0. 8 o [VVoJ((.-1.)))
(8)
where g(-) is any function that may be applied to the gradient [o v) For instance, in standard gradient descent g(-) is simply a linear scaling of VoJ 1)) by learning rate n), while with the
Adam optimizer (Kingma & Ba, 2015) g also modifies the learning rate on parameter-be by by-parameter basis.
Due to the relationship between 0g and v as in Eqn 8, J(O,, Daev) is differentiable with respect to tb. By the chain rule, we can compute the gradient V [(,, Davv) follows:
V ((e, Dant = V(,, ) T vu)((t)
(chain rule)
V(,, T Vo (0:-1 (Vee(-(
(substitute 0, from Eqn 8)
(9)
== -o, D T (vo(VoJ((
(assume V 0)
Here, we make a Markov assumption that V 1 O, assuming that at step 4, given 0t-1 we do not care about how the values of v/ from previous steps led to 0-- Eqn 9 leads to rule to update vi using gradient descent:
1++1 + 1/ 7o J(,, Dan) T Ve
(10)
Here we first derive V U for the general stochastic gradient descent (SGD) update, then provide examples for two other common optimization algorithms, namely Momentum (Nesterov, 1983) and Adam (Kingma Ba, 2015).
SGD Updates. The SGD update rule for 0 is as follows
0g < 0:-1 (0.(().),
(11)
where 1/ the learning rate. Matching the updates in Eqn 11 with the generic framework in Eqn we can see that in Equ 8 has the form:
o(VoJ(0(-1,))) VoJ
(12)
This reveals linear dependency of g on (e.(()--1,v). allowing the exact differentiation of g with respect to th. From Eqn 10, we have
J(e,, D VV
1p V v. [(00, Dan) Dav) Vol(a,
(13)
= =mp.a.vwp....s ( (J(O,, Dam) (e V logp(a, ys v)
Here, the last equation follows from the log-derivative trick in the REINFORCE algorithm (Williams, 1992).
Momentum Updates. The momentum update rule for is as follows
M (
(14)
8. 0.-1 m.,
where μ is the momentum coefficient and 7/: the learning rate. This means that g has the form:
g pm4-11 + maa
(15)
g (x) tje
Therefore, the computation of the gradient V. for the Momentum update is exactly the same with the standard SGD update rule in Eqn 13.
12

---

Under review as a conference paper at ICLR 2020
Adam Updates. We use slightly modified update rule based on Adam (Kingma & Ba, 2015):
g: Vo 12)
2/ 8201 82)
(16)
8 0 (52)
0: + 01-1 g +
where 82 and I/t are hyper-parameters. This means that g is component-wise operation of the form:
7/ B3 -
g(x) B2 + (1 82) +
(17)
g (x) "y (3 U2- +c)
81
(B2v--1 B2) +c)
M1
32
the last equation holds because we assume Ut-1 is independent of tb. Here the approximation makes sense because empirically observe that the individual values of the gradient vector (o.(6, 1)) i.e. It. are close to O. Furthermore, for Adam, we usually use 3n = 0.999. Thus, the value (1 B2)=² in the denominator of Eqn 17 is negligible. With this approximation, the computation of the gradient V. is almost the same with that for SGD in Eqn 13, with one extra component- wise scaling by the term in Eqn 17.
A.2 HYPERPARAMETERS FOR MULTILINGUAL NMT
In this section, we give detailed description of the hyperparameters used for the multilingual NMT experiments.
We use layer LSTM with hidden size of 512 for both the encoder and decoder, and set the word embedding to size 128.
For multilingual NMT, we only use the scorer to model the distribution over languages. Therefore, we use simple 2-layer perceptron network as the scorer architecture. Suppose the training data is from n different languages. For each target sentence and its corresponding source sentences, the input feature is r--imensional vector of O and 1, where indicates source language exists for the given target sentence.
We simply use the dev set that comes with the dataset as Daev to update the scorer.
The dropout rate is set 0.3.
For the NMT model, we use Adam optimizer with learning rate of 0.001. For the distribution parameter v we use Adam optimizer with learning rate of 0.0001.
We train all models for 20 epochs without any learning rate decay.
We optimize both the NMT and DDS models with Adam, using learning rates of 0.001 and 0.0001 for 0 and 1p respectively.
A.3 DATASET STATISTICS FOR MULTILINGUAL NMT
LRL
Train
Dev
Test
HRL
Train
aze
5,94k
671
903
tur
182k
bel
4.51k
248
664
208k
glg
10.0k
682
1007
por
185k
slk
61.5k
2271
2445
103k
Table Statisties of multilingual NMT datasets.
A.4 HYPERPARAMETERS FOR IMAGE CLASSIFICATION
In this section, we provide additional details for the image classification task:
13

---

Under review as a conference paper at ICLR 2020
We use the cosine learning rate decay schedule (Loshchilov & Hutter, 2017), starting at 0.1 for CIFAR-10 10 and 3.2 for ImageNet, both with 2000 warmup steps.
For image classification, we use an identical network architecture with the main model, but with independent weights and a regressor to predict the score instead of a classifier to predict image classes.
To construct the Davv to update the scorer, we hold out about 10% of the training data. For example, in CIFAR-10 (4,000), Paey is the last 400 images, while in ImageNet-10%% since we use the first 102 TFRecord shards, Daev consists of the last 10 shards. Here, "last" follows the order in which the data is posted on their website for CIFAR-10, and the order in which the TFRecord shards are processed for ImageNet. All data in Deey are excluded from Dain Thus, for example, with CIFAR-I0 10 (4,000), [Diain| 3600, ensuring that in total, we are only using the amount data that we claim to use.
We maintain moving average of all model parameters with the rate 0.999. Follow- ing Korbblith et al. (2019), we treat the moving statistics of batch normalization (loffe & Szegedy, 2015) as untrained parameters and also add them to the moving averages.
For ImageNet,
the post-activation ResNet-50 (He al., 2016). The batch sizes for CIFAR-10 10 and for ImageNet are 128 and 4096, running for 200K steps and 40K steps, respectively.
14


Reviews:
[{"review_id": "rkgz2PwHYB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an iterative method that jointly trains the model and a scorer network that places a non-uniform distribution over data sets.  The paper proposes a gradient method to learn the scorer network based on reinforcement learning, which is novel as to what the reviewer knows.\n\nThere are several concerns/questions:\n\n1) The paper doesn’t define the D_{dev} clearly. How is D_{dev} chosen? Is it a subset of D_{train}? \n\n2) In section 2.1, why “smaller development set D_{dev} is much closer to the P_{test}(X,Y)”? P_{test}(X,Y) is supposed to be not observed during training?\n\n3) In Eq (5), if D_{dev} is s subset of D_{train}, if \\theta* is the minimal of J, it means the gradient \nat  \\theta* is 0. To calculate the gradient of J with respect to \\psi, by chain rule, it need to calculate gradient to \\theta* first then \\theta* to \\psi. If gradient of \\theta* is 0, the product is also 0? So the \\psi will not be updated if D_{dev}  is sufficiently similar to D_{train} ?\n\n4) In Section 2.3, it omits the second order Hessian term. How does that influence the performance? \n\n5) it mentions “without significant computing overhead“ in abstract, which is not demonstrated elsewhere.\n\n6) In the experiments, table 1, it seems the major improvement comes from retrain and TCS rather than DDS? In figure 3, it is better to show the weights of an image without DDS and comparing that with DDS.\n\n7) The paper contains many typos such as Eqn.11 is not defined in the main paper, the “Eqn ??” Appears in the appendix, “tha minimizes” etc.\n\nIn general, the idea of the paper is natural and the results seem promising. I am looking forward to the reply to my questions/concerns. \n\n#############\n\nI have read the author's feedback. I think the clarity of both methodology and experiment does not reach the acceptance level and would maintain my current rating. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "r1lqmp15YS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a reinforcement learning approach towards using data that present best correlation with a validation set’s gradient signal. The broader point of this paper is that there is inevitably some distribution shift going from train to test set - and the validation set can be a small curated set whose distribution is closer to the testing distribution than what the training dataset's distribution is. \n\nThe problem setup bears relationship to several areas including domain adaptation/covariate shift problems, curriculum learning based approaches amongst others. One assumption that I see which needs to be understood more is equation (6) - wherein, somehow, there is a Markov assumption used to zero out the contribution of the scoring network on parameters unto previous time step. Trying to understand the implications of this assumption (how the performance varies with/without this assumption) would be instructive for understanding potential shortcomings of this framework.\n\nI think the paper is well written, handles an important question. That said, I am not too aware of recent work in this area to make a decisive judgement on this paper’s novelty/contributions. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "Skgt9dG-9H", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper introduces a simple idea to optimize the weights of a weighted empirical training distributions. The goal is to optimize the population risk, and the idea is to optimize a distribution over the training examples to maximize the cosine similarity between training set gradients and validation set gradients. The distribution over the training set is parameterized by a neural network taking as arguments the\n\nStrengths:\n- The method is quite simple.\n- The results appear to be strong, although I am less familiar with the NMT baselines. The imagenet results seem quite strong to me.\n\nWeaknesses:\n- I couldn't find a particularly clear description of the scoring networks architecture. Given that it observes the whole dataset, this seems like a critical choice that could have a big impact on the complexity of this approach. At the very least, this should be clearly reported, and I recommend a more thorough investigation of this choice.\n- The authors report that their method takes 1.5x to 2x longer to run than the uniform baseline. Yet, they ran all methods for the same number of steps / epochs. It seems to me that a fairer comparison might be letting all methods enjoy the same total budget measure roughly by wall time.\n\nQuestions:\n- I didn't follow why the computation of the per example gradient grad l(x_i, y_i, theta_t-1) is so onerous. Isn't that computed on line 5 already?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an iterative learning method that jointly trains both a model and a scorer network that places a non-uniform weights on data points, which estimates the importance of each data point for training.  This leads to significant improvement on several benchmarks.  The reviewers mostly agreed that the approach is novel and that the benchmark results were impressive, especially on Imagenet.  There were both clarity issues about methodology and experiments, as well as concerns about several technical issues.  The reviewers felt that the rebuttal resolved the majority of minor technical issues, but did not sufficiently clarify the more significant methodological concerns. Thus, I recommend rejection at this time.
Decision: Reject