MixRL: Data Mixing Augmentation for Regression using Reinforcement Learning
Authors: Seong-Hyeon Hwang
Steven Euijong Whang
Venue: ICLR 2022 Submitted (2022)
Keywords: data augmentation
regression
mixup
reinforcement learning

Abstract: Data augmentation is becoming essential for improving regression accuracy in critical applications including manufacturing and finance. Existing techniques for data augmentation largely focus on classification tasks and do not readily apply to regression tasks. In particular, the recent Mixup techniques for classification rely on the key assumption that linearity holds among training examples, which is reasonable if the label space is discrete, but has limitations when the label space is continuous as in regression. We show that mixing examples that either have a large data or label distance may have an increasingly-negative effect on model performance. Hence, we use the stricter assumption that linearity only holds within certain data or label distances for regression where the degree may vary by each example. We then propose MixRL, a data augmentation meta learning framework for regression that learns for each example how many nearest neighbors it should be mixed with for the best model performance using a small validation set. MixRL achieves these objectives using Monte Carlo policy gradient reinforcement learning. Our experiments conducted both on synthetic and real datasets show that MixRL significantly outperforms state-of-the-art data augmentation baselines. MixRL can also be integrated with other classification Mixup techniques for better results.

Under review as a conference paper at ICLR 2022
MIXRL: DATA MIXING AUGMENTATION FOR REGRES- SION USING REINFORCEMENT LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Data augmentation is becoming essential for improving regression accuracy in critical applications including manufacturing and finance. Existing techniques for data augmentation largely focus on classification tasks and do not readily ap- ply to regression tasks. In particular, the recent Mixup techniques for classifi- cation rely the key assumption that linearity holds among training examples, which is reasonable the label space is discrete, but has limitations when the in regression. We show that mixing examples that
label space is continuous either have a large data or label distance may have an mecrasinglyynneatiin ef- feet model performance. Hence, only holds within certain data or label distances for regression where the degree may vary by each example. We then propose MixRL, a data augmentation meta learning framework for regression that learns for each example how many nearest neighbors should be mixed with for the best model performance using a small validation set. MixRL. achieves these objectives using Monte Carlo policy gradi- ent reinforcement learning. Our experiments conducted both on synthetic and real datasets show that MixRL significantly outperforms state-of-the-ar data augmen- tation baselines. MixRL can also be integrated with other classification Mixup techniques for better results.
the strieter assumption that linearity
INTRODUCTION
As machine learning (MI.) becomes widely used in critical applications including manufacturing and finance, data augmentation for regression becomes essential as provides an opportunity to improve model performance without additional data collection. In comparison to classification tasks like object detection in images, the goal of regression is to predict one or more real numbers.
To emphasize the importance of data augmentation in regression, we provide a case study of semi- conductor manufacturing. Here a common quality check to measure the layer thicknesses of a 3 3-dimensional semiconductor and see if they are even. However, directly measuring each thickness results in destroying the semiconductor itself, so recently-common approach is to take an indirect measurement by applying light waves the semiconductor, measuring the spectrum of wavelengths that bounce back from all the layers, and use MI. to predict the layer thicknesses from the spectrum data (see Fig. 4 in Sec. for for an illustration). With enough spectrum data and thickness information, ML models can be trained to accurately predict thicknesses from a spectrum. The main challenge that there is not enough training data, and the only cost-effective solution is to augment small amounts of data that exist. Even a small improvement in model performance from the data augmen- tation has significant impact in this industry. general, any regression task that predicts real values like emissions, stock prices, or even someone's 's salary can also benefit from data augmentation.
Most data augmentation techniques are designed for image classification. In particular, Mixup (Zhang et al.) 2018; Berthelot et al., 2019) Yun et al., 2019) is popular data augmenta- tion technique that is widely used for classification tasks, but is seldom used for regression because it assumes distinct label space. The idea of Mixup is to mix pairs of examples using the key as- sumption that taking a linear interpolation between examples can be used to estimate the label of any examples in between. Mixup is known to effectively regularize the model. More recently, Manifold Mixup /Verma et al., 2019) has been proposed to improve the hidden representation and decision boundaries where examples mixed in multiple hidden layers of neural networks.

---

Under review as a conference paper at ICLR 2022
f(z) = max + +1,2%
f(z) max + +1,2
2
(b)
Figure 1:
A comparison between the Mixup Zhang et al. 2018) and MixRL on a small regression dataset where each example (circle) is associated with single-dimensional feature (x-axis). (a) Mixup takes a linear interpolation of all possible example pairs. Unfortunately, the mixed examples (stars) do not properly reflect the function (blue plot) where some labels are arbitrarily incorrect. (b) MixRL learns for each example how many nearest neighbors it should be mixed with. As result, each example only mixed with its or 2-nearest neighbors to better reflect
However, the Mixup techniques are not readily applicable to a regression setting because the key linearity assumption does not necessarily hold. Since the label space is continuous, taking a lin- car interpolation of examples that are very different either data-wise or label-wise may result in arbitraril--nnoorrett labels shown in Fig. la As result, the linearity assumption only holds to a certain extent, and the degree may vary for each example. Moreover, other data augmentation techniques for classification including image processing (e.g., flipping rotating) and generative models (c.g., GAN (Goodfellow et al.) 2014) and VAE (Kingma & Welling 2014)) are even less applicable to regression setting (see Sec.
We propose MixRI,, a data mixing augmentation framework that is the first to tailor Mixup for re- gression tasks using reinforcement learning. MixRL uses stricter linearity assumption where it only holds within a certain data or label distance. These distance limits may vary by example, and formulate the problem of learning for each example how many nearest neighbors it should be mixed with. MixRL employs meta learning framework that estimates how valuable mixing an example is for reducing the model loss on small validation set using Monte Carlo policy gradient reinforcement learning. MixRL's 's framework is inspired by the recent Data Valuation using Rein- forcement Learning (DVRL) framework (Yoon et al. 2020), which solves the different problem of measuring how individual examples contribute to model performance without any mixing involved. Fig. shows how limiting the nearest neighbors to mix is better than mixing with all neighbors as in classification. To see if the augmentation is useful, we train simple models the original four examples (i.e., no augmentation), the augmented data in Fig. Ia and the augmented data in Fig. [b] Evaluating the models on 20 random test examples results in Root Mean Square Error (RMSE; Sec. 4 values of 0.2967, 0,5615, and 0. 081434, respectively, where a lower RMSE is better. We thus conclude that carefully mixing examples is important for improving regression performance.
Experiments conducted on real and synthetic datasets show that MixRL. shows better model perfor- mances relative baselines, especially when the linearity is limited, and the mixing must be done selectively. In addition, MixRL only requires small validation sets and scales to large training sets.
2 LIMITED LINEARITY IN DATA AND LABEL SPACE FOR REGRESSION
We explain why the key linearity assumption used for Mixup in classification has limitations in regression setting. In classification, the labels are discrete where many examples may have the same label. The original version of Mixup (Zhang et al. 2018) is to take a linear interpolation between any pair of examples x; and x, with the labels y, and y, to produce the new example Ax, + A)x; with the label Λy; + (1 A)y, where ~ Beta(a, a). The linearity assumption turns out to be reasonable because the label difference between examples is only 0 or I and thus not that sensitive to the data difference. In contrast, the labels in regression are in a continuous space. Although there is still a many-to much smaller than in classification. As a result, when two examples are mixed, the interpolated label can be arbitrarily different than the actual label, e.g., mixing the points and d in Fig. raluuss in
relationship where multiple examples may have the same label, the degree is
2

---

Under review as a conference paper at ICLR 2022
1.6
60
Label Distance
1.4
1.4
40
RMSE
RMSE
1.2
1.2
1.0
3
6
1.0
10
20 Label Distance
15
25
10 Data Distance
Distance
(a)
(b)
(c)
Figure 2: Using the Product dataset described in Sec. (a) as the data distances between mixed examples increase, the RMSE (lower is better) of a model trained on the augmented training set initially decreases, but then gradually increases; (b) similar observations the label distances; and (c) the data and label Euclidean distances do not have a linear relationship.
made when increasing
label nowhere near the actual label. In Sec. 4.1 we also show empirical results where the label error increases for larger data label distances. Furthermore, mixing examples with larger data or label effects on the model trained on the augmented training Figs. 2a and 2b show the model accuracies using RMSE for the Product dataset (described in Sec. 4) when mixing examples with different ranges of distances. Regardless of adjusting the label or data distance, there are diminishing benefits for larger distances.
distances tend to have increasingly-neeet
How do we limit the data and label distances to improve Mixup for regression? One approach is to only limit the data distance, which limits the label distance as well. Suppose that the regression function f is continuous where le f(x) f(c) for any and e within the domain of f. We show that a short-enough data distance sufficiently reduces the label distance as well. Given f's domain D and lim fx) L, the following is known hold: Ve, 38 s.t. Vx E D, if x cl < 6, then [f(x) L < e. We can use this result to prove that Ve, 36 s.t. Yx,, Xj E D, if ax, + (1 (1-60 a)xy = c, 0 S a < 1, and
Xyl 8 then the absolute difference between the mixed example's xy value af(
(x, L. [o(f(xx,) - L.) + a fx, - + (1 fx,) - ae a1- e. We cannot do the converse and limit the data distance by limiting label distance because there is a many-to-one mapping from data to labels, which means that two different examples may have identical labels (c.g.. Fig. lals and d).
and T. is small where [x,((x; +
D]] s
However, limiting the data distance to also limit the label distance may be too restrictive because there is not much correlation between the data and label distances in real data. Fig. [c shows how the data distance relates to the label distance for the Product dataset. Even for small data distances, the label distance has large range, which means that the data distance would have to be extremely limited. Hence, our solution is to limit the data and/or label distance as needed instead of just the data distance. This approach turns out to be more practical as we demonstrate Sec. 4.4
3 MIXRL
The goal of MixRL is to identify which examples to mix with which nearest neighbors. Instead of finding the actual distance limits themselves, we solve the identical problem finding the number of nearest neighbors to mix per example for convenience. We reinforcement learning because there is no training data on how mixing each example affects the model performance, and the reward function is non-differentiable as we explain below. MixRL's framework inspired by DVRL YYon et al. 2020), but we solve the different problem of mixing examples and address new issues.
3.1 POLICY OPTIMIZATION REINFORCEMENT LEARNING
Finding the optimal policy involves taking the gradient of the objective function ((0) E.z, [R]] where π is the policy, f is its parameters, and the reward function. For MixRL, we would like to minimize the regression model loss small validation set when mixing batch of examples. However, the validation loss is computed using the regression model, which does not involve 8. Hence cannot analytically compute the differential of the reward function with respect 0.
3

---

Under review as a conference paper at ICLR 2022
0,.
Moving avg. loss validation set Reinforvement learning signal
Random À nearest neighbor encoding
k.
Select examples and mix with training set D
%
V:
Random batch of examples from training D
Concatenate
0.01
Mixup value network hg
Selection probabilities
Regression model fo
Figure 3: MixRL's 's framework. Given random batch of examples from the training set, each exam- ple is combined with one-hot encoding feature containing a random number of nearest neighbors to mix in terms of data or label distance. The Mixup value network hg is a neural network that returns probabilities in proportion to how helpful mixing an example with its neighbors is. The examples are then randomly selected according to the probabilities and mixed with their nearest neighbors in the training set. The random batch is augmented with the mixed examples and used to train a regression model fa- The validation loss is used to update the Mixup value network. We also use reward scaling and employ a baseline function that takes the moving average of previous losses.
Instead, we use the REINFORCE (Williams) 1992) policy gradient algorithm, which is a Monte Carlo method that is also widely used for data valuation (Yoon et al. (2020) and neural architecture searching (Zoph & Le 2017). is known that (0) can be approximated (see Sec. A.I as:
1
m
T
VoJ(O) ~
R(
Vo log (c(((s
(1)
m
where 7' is the 8th state-action trajectory under policy Tp, m is the number of sample trajectories, T the number of actions taken in a path, si is a state at time i, and aj an action at time
In our setting of minimizing loss, REINFORCE performs gradient descent for each example de- creasing by a (() where a is learning rate. Although the estimated gradient is unbiased, is known to have high variance, which we reduce using baseline techniques (Sutton & Barto 2018).
3.2 FRAMEWORK
We define notations used in MixRL's framework shown in Fig. 3 Let D = (xx,)))) P be the training set where X; E X is a d-dimensional input example, and y, E is e-dimensional label. Let D (x"",""))
pl be the validation set, where p' is the distribution of the test set, which is not necessarily the same as P. Let fa be a regression model, and L the loss function that returns a performance score comparing fo (x:) with the true label y, using Mean Square Error (MSE). We assume a list N of possible data and label nearest neighbors (NNs) that can be mixed with an example. For instance, N could contain the options "1 data NN "2 data NNs", and "2 label NNs". The more fine-grained the NN options are, the more precisely MixRL can determine the optimal number of NNs to mix per example. We do not add "0 NN" option because selecting and not selecting it have identical effects, which makes the policy network training unstable because nothing can be learned. Instead, we support this option separately by excluding examples that are not worth mixing as explain later in this section. The possible NNs can be represented one-hot encoding vector of | values.
We now define the states and actions, which is an important design choice of MixRL. A state 8; is where {()+,, 1 S DD and each k; is an index of an NN option in N that specifies how many data or label NNs should be mixed with. An action o is then choosing Tm C D' where each (x,, in Dm is mixed with its [Kk, NNs in D. A policy To (D" (D*) returns the probability of selecting D at state D'. For each episode, MixRL selects a batch (state) once and chooses subset of the batch to obtain reward (action) once. Since there is only one time step, the transition function does not play a role.
batch of examples D = (x),
A naive implementation of the policy network would have input dimension of B x (d (d+e+t + (NN) and an output dimension of 2 which may be too large to train for large batch sizes. For example, in typical setting of B 1000, d 100, e 1, and |N| 10, the input and output dimensions
4

---

Under review as a conference paper at ICLR 2022
Algorithm/ 1: Pseudo code for Mixup value network training.
Input Training D, validation DD", nearest neighbor options N, learning rate o, reward scaling constant C, moving average window W
Output: Mixup value network hi
Initialize 0, Base = 0;
while until convergence do
Sample D = ((x.)))), k.). ~ P Uniform(N);
D W;
for ((x,,,)) E DD do
I Add Y.: ki) to D with probability ho(xi, Y,, Y,, k:);
Train regression model fo with initialized e on D U Mix(D" D););
Loss y E L(fo(x)),)))
0= a C. (Loss- Base) Vo mm (D",,
Base W-1 Base + Loss;
W
return return ha:
become 111, 000 and 21000 respectively. Instead, we can significantly reduce the network size by assuming independence among examples and decomposing the policy network's prediction into two phases. First, a Mixup value network (x(x.)) y, k) is used to estimate the probability of an example being mixed with k NNs. Next, randomly select which examples mix according to the prob- abilities. The value of To (DD is thus II (x.,,))) he (x. y, k) (x. [1 ho(x,) y, k)]- Also, hg has input and output dimensions of only d + e |N] and 1, respectively, and is practical.
The objective function (@) is the validation set loss to of fa trained on D' U Min(DD N) (either from scratch or from pre-trained state) where Mix(Dm D, N) mixes each (x,. in D with its [[k,] NNs in D:
((0) = 1(0)
E (
(xf(x
Using Eq. I the gradient of Eq. can be approximated using the validation set D9
VaJ(0) vol(O)
((fo("))))) V log(Too DD))
(3)
where Vo log (D" D= V Σ (D log ho y, k) Eing [1 ho kx, We also use reward scaling (Henderson et al. 2018)) to further improve the training.
Algorithm shows the pseudo code for training hg. The computational complexity is not directly related the training set size, but depends the number of iterations needed to train he and how long each iteration takes, Another factor is the number of possible NNs [N] where a higher number may result in slower training. We show in Sec. 4.2 that MixRL scales to large training sets.
Once we train hgg we choose the (x, y, k)'s with the highest ha(x. k) values of at least a threshold T. We optimize T's value using the validation set (see Sec. B.I). We ignore any (x, y, k) whose (x, pair has already been chosen with a different k value for the same data or label distance. We apply in order to exclude (x, y, k)'s 's that are not worth mixing (i.e., make them mix with 0 NNs).
Although actor-critic methods (Mnih et al. 2016 Schulman et al. 2017) improve on REINFORCE by training both value and policy networks, using the two networks may not be practical in our setting due to the high input dimensions. For policy network, we can reduce its dimension using the fact that the log probabilities can be calculated by multiplying selection probabilities. Unfortunately, cannot use the same trick for value network because its output values are not probabilities.
4 EXPERIMENTS
We provide experimental results for MixRL. We evaluate the regression models trained on the augmented training sets separate test sets and repeat all model trainings 5 times. We use Py- Torch (Paszke et al. 2017), and all experiments are performed using Intel Xeon Silver CPUs and NVidia RTX GPUs More experimental setting details are in Sec. B.1
5

---

Under review as a conference paper at ICLR 2022
Predict layer thicknesses
mM m
Figure 4: Spectrum generation
3-d semiconductor for the Product dataset.
Table 1: Settings for the four datasets.
Dataset
Data dim.
Label dim.
Training set size
Val. set size
Test set size
NO2 Product
200
200
100
300
300
200
Synthetic
226
4
100
100
100
Airfoil
1003
300
200
Measures We use accuracy metrics. RMSE =
(u: úz)² is the absolute difference
(w -0) between predicted labels and true labels where lower value is better. R² E.n (9.-)) 3)= relative improvement compared when returning the label average as a prediction. The value is in the range [0, I] where higher value is better. The two measures complement each other.
is the
Datasets We use three real and one synthetic datasets. The NO2 emissions dataset (Aldrin 2004) contains traffic and meteorological information around roads. The features include cars per hour, wind speed, temperature, and others. The label is the NO2 concentration. The Product dataset obtained through a collaboration with company contains spectrum data that generated using the procedure described in Sec. 1 and illustrated in Fig. 4 on 20-layer 3-d semiconductors. The Product dataset proprietary. so for clarity we also experiment on public synthetic dataset (Synthetic) provided by the DACON challenge (DACON Co., Ltd.) 2020) where simulator generates spectrum data using the same procedure as Product assuming 4-layer 3-d semiconductors. Finally, the Airfoil UCI dataset (Dua & Graff 2017) contains aerodynamic and acoustic test results for airfoil blade sections in a wind tunnel where the features include frequency, angle of attack, and chord length. Table compares the four datasets in more detail. Note that Product and Synthetic have high dimensions, while Airfoil has the largest training set size.
multi-layer perceptron (MLP) with 2 hidden layers and set the number of nodes per hidden layer to be [512. 256] for NO2, [2048, 1024, 512, 256] for Product, and [2048, 1024, 512, 256] for Synthetic. We employ layer normalization, early stopping for regularization, and reward scaling and baseline techniques that improve the per- formance and stability of reinforcement learning. We use the Adam optimizer for all MLP trainings. When mixing examples, we set the Mixup ratio À to 0.5 unless specified otherwise. This value em- pirically results the best Mixup performance in a regression setting, and we give an explanation in Sec. B.3 For the possible numbers of NNs for NO2, our default setting is the series of integers (4, 16, 64) for both data and label distances, i.e., [ = 6. For Product, Synthetic, and Airfoil, the default series are {100, 150, 200, 250), (4, 16, 64), and {1, 2). respectively.
MixRL Settings For the regression model,
Baselines We employ four baselines. First, we train a regression model on the labeled data without any data augmentation ("No Augmentation"). Next, we faithfully implement the original Mixup algorithm (Zhang et al., 2018) where all example pairs can be mixed ("Original Mixup"). We also implement Manifold Mixup (Verma et al. 2019). which is state-of-the-an Mixup technique for classification that is also applicable to regression. Finally, we compare with a simplified version of MixRI where we use single NN value to mix examples with their data and label neighbors ("Global kNN")). This k parameter is tuned using Bayesian Optimization (Mockus 1974).
4.1 LIMITED LINEARITY IN REGRESSION
We empirically show that the linearity assumption does not hold from certain data or label distances using the following simple experiment. Given dataset, we first train regression model. We then

---

Under review as a conference paper at ICLR 2022
0.5
0.69
0.99
Data Dist.
La A RMSE
Distarc
Label Dist
Label Die
RMSE
RMSE
0.53
0.53
0.4
0.0
Normalized
Normalized
0.5
1.5
0.53
0.5
Distance
0.30
Label Distance
(a) NO2
(b) Product
(c) NO2
(d) NO2
(c) NO2
Figure 5: (a) For NO2, the label RMSE increases for larger (normalized) data or label distances. (b) Similar label RMSE trends for Product. (c) For NO2, the data and label distances do not have linear relationship. (d) For NO2, the regression model's RMSE increases for larger data distances. (e) For NO2, the model's RMSE decreases, but then increases for larger label distances.
Table 2: Accuracy and runtime results the NO2, Product, and Airfoil datasets. Four baselines are compared with MixRL: (1) No data augmentation; (2) Original Mixup (Zhang et al., (018); (3) Manifold Mixup Verma et al. 2019); and (4) Global kNN. For each global kNN, we show the best k value found with Bayesian optimization. We also integrate MixRL with Manifold Mixup.
Dataset
Method
RMSE
R
Runtime (mins)
No Augmentation
0.5123 0009
0.5128 0000008 0008
0.01
Original Mixup
0.515210.001
0.501811..002
0.01
Manifold Mixup
0.5077 1..0024
0.5182 10.0044
0.02
NO2
Global KNN (k=64)
0.5089 10.009
0.5149:10.0m
0.5
MixRL
1993770000040
0.5445 00.0088
40
MixRL + Manifold Mixup
0..061
0.5221
45
No Augmentation
1.3525
0.7208
2
Original Mixup
1.1617
0. 0.788441000034
Manifold Mixup
1.2553 0086
0. 0.7490 .0300
3
Product
Global KNN (k=267)
1.1375=0.0100
0. 7..7846-000003 0.03
20
MixRL
1.1383
0.7924 0.792240.0020 000000
390
MixRL + Manifold Mixup
1.247510.0124
0.750911.004:
400
No Augmentation
1.2924=0.0277
0.9650+0.0011
3
Original Mixup
2.6394_1..221
0.8665
Manifold Mixup
1.4495 00006
0.9567
11
Airfoil
Global kNN (k=1)
1.5072+0.0200
0.952040.0016
25
MixRL
1.23446
0.9681
410
MixRL Manifold Mixup
1.2030 10.0311
0.9697.0.0015
460
generate mixed examples and compute the RMSE between their interpolated labels and the labels predicted by the regression model. While the regression model is not perfect, the relative trend the error gives sufficient insight. Figs, 5a and 5blindeed show increasing label errors for larger label and data distances for the NO2 and Product datasets, respectively. Figs. 2c and 5c also show how label and data distances do not necessarily have linear relationship for the regression models on top of augmented training sets where we only mix examples within certain distance ranges. As result, mixing examples with larger data or label distances has increasingly-
datasets. We also train
negative effects on the model accuracy as shown in Figs. 2a 2b
5d and 5e
4.2 ACCURACY AND RUNTIME RESULTS
We compare MixRL with the four baselines using the settings in Table Table shows the results for the three real datasets (the Synthetic dataset results are in Sec. B.2 MixRL significantly outper- forms all the baselines in terms of regression model accuracy, which demonstrates that able to pinpoint which examples should be mixed with how many kNNs. Figs. (6] options are frequently used for NO2 and Product (the KNN options for Synthetic and Airfoil are in Sec. B.2). Although the runtime is slower than the basclines due to the reinforeement learning, the gained accuracy can be worth the effort in real applications because data augmentation is one-time cost that run in batch mode. Another observation is that Original Mixup performs relatively bet-
(ndand and 6b show which kNN

---

Under review as a conference paper at ICLR 2022
Frequency
250
Label KNN
Data INN
Frequancy
RMSSE
0.53
RMSE
1.29
Runtime (minss)
200
0.4
150
250
0.47
1.0
150
200
1:40
300
400
603
000 1000
Val.
Training set
(a) NO2
(b) Product
(c) NO2
(d) Product
(e) Airfoil
Figure 6: (a)-(b) KNN option frequency histograms for two real datasets. (c)-(d) Model accuracy vs.
validation set size for two real datasets. (e) MixRL's 's runtime against the training set size for Airfoil.
Table 3: Comparing kNN options for MixRL on the NO2 and Product datasets.
Dataset
Data and Label KNN Options
RMSE
R
{4, 16)
0.499440.008
0.5305
NO2
[4, 16, 64) (Default)
0.4937 10.0046
0.54445.0.0068
(4, 16, 64, 128)
0.4932 10.0023
0.5447 0.544710.0032 10.0032
(100, 150} [100, 150, 200, 250} (Default)
1.148440.000 0040
0.7902=0.0000
Product
1.1383 1.00057
0.7924
(100, 150, 200, 250, 300}
1.1372: 00.004
0.7942::0.0015
ter for Product than other datasets because linearity holds more as evidenced by the large k = 267 value of Global kNN. In general, MixRL outperforms Original Mixup by a larger margin the more limited the linearity. We also integrate MixRL with Manifold Mixup where examples are mixed in multiple layers of the MLP with their NNs (see Sec. [B.4]o for more details). The results are mixed where the integrated method performs the best on the Airfoil dataset, but not on the other datasets.
Validation Set and Scalability Figs. bc and [d] show the validation set size impact on MixRL's accuracy for the NO2 and Product datasets. The Airfoil dataset results are similar and shown in Sec. B.2 We observe that a small validation set is sufficient for MixRL to be effective. Also MixRL scales well against increasing training set sizes as shown in Fig. be
4.3 K NEAREST NEIGHBOR OPTIONS
We investigate how changing the kNN options N affects MixRL's performance in Table 3 using the NO2 and Product datasets. The other dataset results are similar and shown in Sec. B.2 For each dataset, we compare its default series with shorter and longer series. We observe that providing more kNN options improves MixRL's 's performance, but only a certain extent. Hence, MixRL performs well even with a small number of options.
4.4 ABLATION STUDY
We perform an ablation study to investigate the effect of limiting label and data distances in Table using the NO2 and Product datasets. The other dataset results are similar and shown in Sec. B.2 We compare MixRL with three variants: (1) MixRL that does not exploit any distances and selects examples that have the highest he values when fully mixed with all the other examples; (2) MixRL that does not use label distances; and (3) MixRL that does not use data distances. We observe that suggested in Sec. 2
using both distances is necessary for the best accuracy results
5 RELATED WORK
Data Augmentation for Regression There are largely two branches of work for data augmenta- tion regression. One is semi-supervised regression (Kostopoulos et al., 2018 Zhou & Li 2005 Kang et al.) 2016 Kim et al. 2020) where the goal is to utilize unlabeled data for training. In com- parison, MIXLL does not assume the availability of unlabeled data. Another branch is data augmen- tation when there is unlabeled data, which is research focus. To the best of our knowledge, there is no technique tailored to regression, so we instead cover data augmentation techniques for classification that can be extended regression.
8

---

Under review as a conference paper at ICLR 2022
Table 4: Ablation study for MixRL on the NO2 and Product datasets. Three variants are compared: (1) w/o label and data distance limits; (2) w/o label distance limits; and (3) w/o data distance limits.
Dataset
Method
RMSE
R²
W/O Label & Data Dist. Limits
0.5158 0026
0.5019 =00003
NO2
W/O Label Dist. Limits
0.5007 10.0017
0.5298
0.0028
W/O Data Dist. Limits MixRL
0.5026 0044
0.5331
00.0087
0.4937
0.5445
000400000000
W/O Label & Data Dist. Limits
1.1561 0000003
1.7858_000041
Product
W/O Label Dist. Limits
1.16521100077
0.7827
0.0000
W/O Data Dist. Limits
1.1471 1.0.0048
0.7889
0.0022
MixRI.
1.1383.10.00 1000037
0.7924 0.792410.008 10.0020
We explore data augmentation techniques for classification and investigate if they can be used in regression setting. There are largely three approaches: generative models, policies, and Mixup techniques. Generative models including GANs (Goodfellow et al.] 2014) and VAFs (Kingma & Welling, 2014) are popular in classification where the idea is to generate realistic data that cannot be distinguished from the real data by a discriminator. However, a major assumption is that the labels of the same, which does not necessarily hold in regression setting where
the generated examples most examples may have different labels. Another approach to use policies (Cubuk et al. 2019), which specify fixing rules for transforming the data while maintaining the label value. For example, image processing policies may flip, rotate, or adjust the brightness of images to generate new valid images. However, in regression setting, the same transformed examples may now have different unknown labels, making them unsuitable for training. The recently- proposed Mixup (Zhang et al.) recently-proposed
2018 Berthelot et al., 2219)] Yun et al., 2019; Berthelot et al., 2020 Hendrycks et al.) 2020 Verma et al. 2019) takes the alternative approach of generating both data and labels together by mixing ex- isting examples with different labels assuming linearity between training examples (Chapelle al.]
2000 Wu et al. 2020). Although the Mixup paper mentions that its techniques can easily be ex- tended to regression, we make the non-trivial observation that the linearity assumption does not necessarily hold where mixing examples with far-away examples may not be beneficial and even detrimental to model performance. Instead, MixRL shows good performance by carefully choosing which nearest neighbors to mix with, technique that only works in regression setting.
Policy Optimization Reinforcement Learning Within reinforcement learning (RL) (Sutton &
Kaelbling et al. 1996). we are interested in policy searching methods (Neumann & Peters 2015) where our goal is to find the best policy for mixing examples. In particular, MixRL's framework is inspired by the recent Data Valuation using Reinforcement Learning (DVRL) frame- work (Yoon et al. 2020). which uses REINFORCE (Williams) 1992) to estimate the value of data in terms of how much it improves the model performance. In comparison, we solve the com- pletely different problem of data augmentation for regression where extend Mixup using REIN- FORCE. Some data augmentation techniques use actor-critic methods like A3C (Mnih et al.) 2016) and PPO (Schulman et al.) (017) that train value networks along with policy networks. For ample, Autmeng (Cubuk et al. (019)) uses PPO to search for data augmentation policies that dictate how to modify existing examples to generate additional training data for image classification. However, using actor-critic methods is practically challenging in our setting due to the large value networks that need to be trained as we discussed in Sec. 3.2
Barto 2018
CONCLUSION
We proposed MixRL,, which to our knowledge the first Mixup-based data augmentation meta learning framework tailored to regression tasks. We observe that linearity only holds up to a certain data or label distance per example in a regression setting. Hence, MixRL extends Mixup by deter- mining the k nearest neighbors to mix per example using Monte Carlo policy gradient reinforce- ment learning where the objective is to minimize the regression model's loss on a small validation set. Since MixRL only limits the examples to mix, it can be integrated with any existing Mixup technique for classification. In our experiments, we showed that MixRL outperforms various data augmentation baselines for regression by effectively selecting the k nearest neighbor options.

---

Under review as a conference paper at ICLR 2022
Ethics Statement MixRL demonstrates that data augmentation significantly enhance the model accuracy for regression tasks. However, not controlled, data augmentation may also in- troduce new bias in the data that makes the model more discriminative. Thus interesting future work is to augment data while ensuring model fairesss.
Reproducibility Statement We add MixRL's 's source code in the supplementary. We also provide more details on Eq. experimental settings, and experimental results in the appendix.
REFERENCES
Magne Aldrin. Cmu statlib dataset. http://lib.ster htt.://asttac.ccmm.cceatttttttt 2004. Ac- cessed Oct. 5th, 2021.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. abs/1607.0066 2016.
Layer normalization.
CoRR,
David Berthelot, Nicholas Carlini, Ian J. Goodfellow, Nicolas Paperno, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. NeurIPS, pp. 5050-5060, 2019.
David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution matching and augmenta- tion anchoring. ICLR, 2020.
Emma Brunskill. Policy gradient lecture notes. http://web. stanford.edu/clleee.
cs234/s1 htt////liees//ee wtttee/sseettue...pem pdf 2021. Accessed Oct. 5th, 2021.
Olivier Chapelle, Jason Weston, Léon Bottou, and Vladimir Vapnik. Vicinal risk minimization. In NIPS. pp. 416-422 462, 2000.
Ekin D. Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc Le. Autoaugment: Learning augmentation strategies from data. In CVPR, pp. 113-123, 2019.
DACON Co., Ltd. Challenge on semiconductor thin film thickness analysis. https://dacom: 2020. Accessed Oct. 5th, 2021.
Dheeru Dua and Cascy Graff. UCI machine learning repository, 2017. http::///arcch
lcs. www. www...ummmmmmm..
lan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, pp. 2672-2680, 2014.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In AAAI, pp. 3207-3214, 2018.
Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: simple data processing method to improve robustness and uncertainty. In ICLR, 2020.
Leslie Pack Kaelbling, Michael L. Littman, and Andrew W. Moore. Reinforcement learning: A survey, J. Artif. Int. Res., 4(1):237-285, May 1996.
Pilsung Kang, Dongil Kim, and Sungzoon Cho. Semi-supervised support vector regression based on self-training with label uncertainty: An application to virtual metrology in semiconductor manufacturing. Expert Syst. Appl., 51:85-106, 2016.
Sung Wook Kim, Young Gon Lee, Bayu Adhi Tama, and Seungchul Lee. Reliability camera lens module classification using semi-supervised regression method. Applied Sciences, 10(11), 2020.
Diederik P. Kingma and Max Welling, Auto-encoding variational bayes. In ICLR, 2014.
10

---

Under review as a conference paper at ICLR 2022
Georgios Kostopoulos, Stamatis Karlos, Sotiris Kotsiantis, and Omiros Ragos. Semi- regression: A recent review. J. Intell. Fuzzy Syst. 3582):143--1100, 2018.
Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In /CML, pp. 1928-1937, 2016.
Jonas Mockus, On bayesian methods for seeking the extremum. In Proceedings of the IFIP Techni- cal Conference, pp. 400-404, Berlin, Heidelberg, 1974. Springer-Verie Verlag. ISBN 354007 16502.
Gerhard Neumann and Jan Peters. Policy search: Methods and applications. In JCML, 2015.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop, 2017.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. optimization algorithms. CoRR, 0on//107.06647, 2017.
Proximal policy
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. Bradford Book, 2018. ISBN 0262039249.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez- Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In ICML, 2019.
Ronald Williams. Simple statistical gradient-following algorithms for connectionissrreifforeeeee learning. Mach. Learn., 8:229 256, 1992.
Sen Wu, Hongyang R. Zhang, Gregory Valiant, and Christopher Ré. On the generalization effeets of linear transformations data augmentation. JCML, pp. 10410-10420, 2020.
Jinsung Yoon, Sercan Ömer Arik, and Tomas Pfister. Data valuation using reinforcement learning. In ICML, pp. 10842-10851, 2020.
Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Seong Joon Oh, Youngjoon Yoo, and Junsuk Choe. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, pp- 6022-6031, 2019.
Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empiri- cal risk minimization. In ICLR, 2018.
Zhi-Hua Zhou and Ming Li. Semi-supervised regression with co-training. In IJCAI, pp. 908-913, 2005.
Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In ICLR, 2017.

---

Under review as a conference paper at ICLR 2022
A
APPENDIX - FRAMEWORK
A.1
GRADIENT OF (0)
We continue from Sec. 3.1 and derive Eq.
Brunskill 2021):
1 R(T")
(O) R
Vo log To To(a(s)))
(I)
==
where is state-action trajectory under policy no. is the number of sample trajectories, T is the number of actions taken in a path, St is a state at time t, and az is an action at time t. Suppose that P(T; 0) is the probability of a trajectory T when using the policy me and R(F) the reward of r. Then [0) can be expressed asE, P(T; P(T:O)(((T). In addition, let us specify so as the initial state, p(ss) as the initial state's distribution, and P(s++1 St, ai) as the dynamics model that specifies
which predicts the next state S(+1 when taking action from state st. We can then take the gradient of J [0) as follows:
VoJ(O) -oo Σ P(T:0)R(T)
Σ (OP(T:O)R(T)
P(T:0) (oP(T:))R(T) P(T;0)
E P(T:O)R((T ()P(T; P(T,0)
Σ P(T:0))(T)V log P(;;
R( log P)
R( log
m(s0)
II
(a(a))s))
T-1
R(T
V
log ((s))
log Ne (ailss) + log P(s)41ls(,at)
R(T*)
V log (ailst)
B APPENDIX EXPERIMENTS
B. MORE EXPERIMENTAL SETTINGS
We continue describing our experimental settings from Sec. 4 For the regression model, we use layer normalization (Ba al. 2016) for the Product, Synthetic, and Airfoil datasets. We also use early stopping to prevent overfitting on the validation set for all the datasets. For the mixup value network hes we use reward scaling and baseline techniques that improve the performance and stability of reinforcement learning. For the mixup value network hag we use multi-layer perceptron with hidden layers and set the number of nodes per hidden layer to be [100, 100, 100, 100] for the four datasets. For hig we use learning rate of a 0.001 for all datasets. For the regression model fa. we use learning rate of 0.0001 for Product and 0.001 for the other datasets. When training he, set the batch size B to be the training data's 's size. When training for
batch size of 32 for NO2, 64 for Product, 32 for Synthetic, and 64 for Airfoil. We use a reward scaling (Henderson
al. 2018) constant of C 5 for NO2, C for Product, C = 0.1 for Synthetic, and C. = for
12

---

Under review as a conference paper at ICLR 2022
Airfoil. We use moving average window of size W 20 for the three datasets. When setting the threshold for limiting examples to mix, we choose a value that results in a high model accuracy (low RMSE) on the validation set.
When constructing the Synthetic dataset, we take a subset of the entire DACON challenge dataset (DACON Co., Ltd. 2020). While the the range of thicknesses for the full datasets is [10, 300] for all four layers, we only use examples with thicknesses within the range [10, 50]. The pur- pose reduce the training time while still making clear comparison between MixRL and the other methods.
To improve the runtime of MixRI, we pre-trained regression models for the Product, Synthetic, and Airfoil datasets. First, we train the base regression model on the training data. Then, we fine-tune the pre-trained model obtain the validation losses.
B.2 MORE Synthetic AND Airfoil DATASET EXPERIMENTS
We continue from Sec. 4.2 and show the accuracy and runtime results for the Synthetic dataset 3 The results are similar to the Product dataset in Table
Accuracy and Runtime Results
Table 5:
Accuracy and runtime performances for the Synthetic dataset. The other settings
identical to those of Table
Dataset
Method
RMSE
Ra
Runtime (mins)
No Augmentation
9.2624
0.5742+0 00.0010
0.1
Original Mixup
8.4932 05052
0.6357 +0.0030
0.1
Manifold Mixup
8.9210 0687
0.5973 20.066
0.1
Synthetic
Global KNN (k=43)
8.4113
0.6400 +0.0030
11
MixRL
8.3567 +0.0339
0.6463 10.0043
260
MixRI. + Manifold Mixup
8.5958
0.6297
370
kNN Options Frequency Histogram We continue from Sec. 4.2 and show which kNN options
are frequently used in our default settings for the Synthetic and Airfoil datasets Figs. 7a and 7b respectively. Compared to the Product dataset, Synthetic has cases where A = 0 because mixing more beneficial to the model's accuracy according to the validation set.
1100
Frequency
150
KNN
Frequency
900
Label
100
700
60
20
4
16
64
0
1
2
4
k
k
(a) Synthetic
(b) Airfoil
Figure 7: kNN option frequency histograms for the (a) Synthetic dataset and (b) Airfoil dataset.
Validation Set Size and Accuracy We continue from Sec. and show how the validation set size impacts MixRL's accuracy for the Airfoil dataset in Fig. g The observations are similar those of the other datasets where small validation set size is sufficient for MixRL to perform well.
kNN Options Comparison We continue from Sec. 4.3 and provide the kNN options experiments for the Synthetic and Airfoil datasets in Table 6 The observations are similar to those Table where small number of options is sufficient for good model accuracy.
13

---

Under review as a conference paper at ICLR 2022
1.4
RMS
1.2
1.0
100
200 Val. set size
Figure Model accuracy vs. validation set size for the Airfoil dataset.
Table 6: Comparing kNN options for MixRL the Synthetic and Airfoil datasets.
Dataset
Data and Label KNN Options
RMSE
R
{4, 16]
8.9018 04490
0.5954+0.0034
Synthetic
[4. 16, 64) (Default)
8.3567 +0
0.646340.0043
{4, 16, 64, 100}
3.5211+0.0214
0.6291
[1.2]
1.282440.0200
0.9660=0..010
Airfoil
[1, 2, 4) (Default) 1. 2, 4, 8]
1.2346 03325
00.0325
0.9681
1.25-46+0.0088 0096
0.9678
We continue from Sec. 4.4 and provide the ablation study for the Synthetic and Airfoil datasets in Table 7 The results are similar to those in Table 4 where using both label and data distances results in the best model accuracies.
Ablation Study
Table 7: Ablation study for MixRL on the Synthetic and Airfoil datasets. Three variants compared: (1) w/o label and data distance limits; (2) w/o label distance limits; and (3) w/o data distance limits.
Dataset
Method
RMSE
R
W/O Label Data Dist. Limits
8.4392 10.0658
0.6370 19970-0.0006
W/O Label Dist. Limits
9.2794 1.00554
0.5708 10.003
Synthetic
W/O Data Dist. Limits
8.3720
0.6424_10.0033
MixRL
8.3567 +0.339
0.6463
W/O Label Data Dist, Limits
1.2852.10.0144
0.9659.2..0009
W/O Label Dist. Limits
1.2447
0.9678
Airfoil
W/O Data Dist. Limits
1.2552.10.0118
0.9682_1.00006
MixRL
1.2346_10.0222
0.9681 00.0015
B.3 JUSTIFICATION FOR USING À 0.5
We continue from Sec. and provide more details on why setting À 0.5 empirically results in the best Mixup performance in a regression setting. Tigure[sshor MixRL's 's accuracy when we vary using the same setting in Table for the NO2 and Product datasets. As a result, the accuracy peaks when À = 0.5. We suspect that generating examples in the sparsest areas helps the regularization the most. In classification setting, however, this strategy may not be ideal especially if the generated examples are too close to decision boundary that is in the middle of the mixed examples.
B.4 INTEGRATING MiXRL WITH MANIFOLD MIXUP
We continue from Sec. 4.2 and provide more details on integrating MixRL with Manifold Mixup (Verma al. 2019). We first MixRL and determine which examples to mix. Next, instead of mixing examples in the training data before model training, we mix examples on multiple layers in the MI.P during model training. For the second mixing step, Manifold Mixup as is
14

---

Under review as a conference paper at ICLR 2022
NO2
Product
1.25
0.51
RMSE
0.50
RMSH
1.20
0.49
1.15
0.48
1.10
0.25
0.50
0.75
0.25
0.50
0.75
À
A
Figure 9: MixRL's accuracy for different À values the NO2 and Product datasets.
where we start with set of eligible layers for mixing. (In our experiments, always set the first three layers as eligible.) For each training batch, we randomly select mixing layer and mix the features and labels of the examples that MixRL decided to mix.
15


Reviews:
[{"review_id": "w1aSa2xAjy9", "reviewer": "Reviewer_3Wyg", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "MixRL is inspired by the problem of measuring how individual examples contribute to model performance. The idea is reasonable for regression tasks.\nThe experiments verify its effectiveness by improving regression performance by carefully mixing examples.\nThe paper is well written and organized.\n\nI notice MixRL cannot improve previous Mixup methods by a clear margin. Since I am hesitant about the importance of the improvement, I want to hear about other reviewers’ opinions to make the decision.", "summary_of_the_paper": "To apply Mixup for regression tasks, the paper first utilizes the stricter assumption that linearity only holds within specific data or label distances for regression.\nThen this paper proposes a data mixing augmentation method called MixRL. The goal of MixRL is to identify which examples to mix with which nearest neighbors. MixRL employs a meta-learning framework that estimates how important mixing a sample is to minimize the model loss on a validation set using policy gradient reinforcement learning.", "summary_of_the_review": "I'd like to see the other reviewers' opinions about the empirical results.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "vPCJ6hYy_G5", "reviewer": "Reviewer_aYea", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n-Data-dependent input/output proximity constraints on mixup are learned.\n\n-Consistent but small gains over mixup and manifold mixup are realized on several regression datasets.\n\nLimitations:\n\n-The set of nearest neighbors considered for use in mixup, while dynamic, are highly restricted (e.g. {4, 16, 64, 128} for the NO2 dataset in table 3.\n\n-More generally, the lack of a local input/output kernel to prioritize more local data in a continuous manner feels like a significant limitation.\n\n-Optimized input/output kernels that restrict and more generally re-weight mixup pairs or construct locally non-constant regressions to sample from feel like important, missing baselines. The lack of a locally non-constant regression model in MixRL (beyond mixing data pairs) is likely limiting performance significantly.", "summary_of_the_paper": "The authors propose MixRL to improve upon mixup in regression settings. MixRL is used to impose a proximity constraint on the input/output pairs that are mixed during mixup-based data augmentation, by predicting how many nearest neighbors to utilize, from a small set of pre-specified options, based on feedback from evaluating the validation set. Consistent but small gains over mixup and manifold mixup are realized on several datasets.", "summary_of_the_review": "The technique while sound is currently quite limited in scope in that important baselines and algorithmic elements (e.g. locally non-constant models and local kernels) that need to be considered in the context of doing high fidelity data augmentation for regression problems have not been adequately investigated or discussed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ZJKnbVvmBZd", "reviewer": "Reviewer_XF4Y", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe idea is simple and appears to be implemented correctly, and the experiments appear to be justifiable given mixup is a useful regularization technique for supervised learning, and has been shown to be very useful in classification tasks. The application of REINFORCE as a stochastic gradient approximator is well known in other context that involve NNs with discrete processes (see DLGMs, e.g., Rezende 2014). The experiments are straightforward and support the value of their work.\n\nWeaknesses:\nI have deep concerns about this paper being marketed as \"RL for mixup\". RL comes packaged with a great deal more than just the REINFORCE algorithm, e.g., the policy gradient, and rephrasing a lot of what is a gradient estimator for a discrete process as \"RL\" not only may potentially confuses the reader on what RL is, but also doesn't connect with an active area of research on doing backprop through discrete processes. Yes, I understand that the cited Yoon 2020 paper did this as well, but I don't think they should have either. One of the most famous examples of using REINFORCE is with discrete VAEs or DLGMs (Rezende 2014), and they are careful not to call it \"RL\"\n\nBut there are a number of other works or baselines that this paper should have compared to, and I think in part this is because the related works talk about other RL algorithms: for instance A3C or even PPO aren't really relatable to the problem being addressed here as straight-through estimators (Bengio 2013) or Gumbel Softmax (Jang 2017) (Yoon does a better job at connecting to these types of works). \n\nFor instance, it seems possible to backprop through to the selection probabilities using Gumbel Softmax. Variance is usually much better than REINFORCE, even with the baseline removal.\n\nThese are important baselines and it is important this paper places itself correctly among related works.\n\nNext, there are many claims about why classification doesn't need this sort of mixup, but I didn't follow the arguments entirely (P1-P2 in Section 2). I don't see any classification experiments to back up this claim, at least in the main text.  For instance \"Linear assumption (in classification) turns out to be \"reasonable\" because the label difference...\" Why is it reasonable to leave the label space in classification mixup: with regression I have a better chance of hitting a \"label\" in the distribution.\n\nOther notes / concerns:\nSo the motivation was on some physical systems, and we're using REINFORCE to learn how to mix data. But if we're in the physical systems such as you describe, usually we have some sort of (closed-form solution) model that describes the physical process (maybe poorly). Even if this model has error, wouldn't this be a good place to design a prior that tells you how to sample? e.g., if the masses are some set distance apart, then use, else don't.\n\nI see meta-learning as a description of the model, but not in the algorithm 1.\n\nHow do you sample k when you decide how many neighbors to mix? Is this sampled from a prior?\n\nRezende 2014: Stochastic Backpropagation and Approximate Inference in Deep Generative Models\nBengio 2013: Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation\nJang 2017: Categorical Reparameterization with Gumbel-Softmax", "summary_of_the_paper": "This paper improves on the idea of mixup by selecting samples for mixup via a model that selects suitable pairs found using knn in a batch. In order to provide a learning signal to this discrete process, the authors apply REINFORCE, using the loss of the downstream regressor (not applied to classification tasks). They show promising results on a number of regression tasks.", "summary_of_the_review": "I recommend reject, as the paper's core story isn't well placed compared to related works. The paper is placed as \"RL\" so we're missing important baselines and connections to other gradient estimators through discrete processes.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper generalize the idea of Mixup-based data augmentation for regression. Compared to classification for which Mixup was used, the paper argues that in regression the linearity assumption only holds within specific data or label distances. The paper thus proposes MixRL to select suitable pairs using k-nearest neighbor in a batch for mixup. The selection policy is trained with meta-learning by minimizing the validation-set loss. The approach provides consistent but small improvement over mixup on several datasets. Reviewers have also suggested discussion and comparison with more baselines, such as respective method using other (lower-variant) gradient estimators (e.g., gumbel-softmax), and using local input/output kernels for data selection, etc.
Decision: Reject