Year: 2021

Abstract:
Data augmentation is becoming essential for improving regression accuracy in critical applications including manufacturing and finance. Existing techniques for data augmentation largely focus on classification tasks and do not readily apply to regression tasks. In particular, the recent Mixup techniques for classification rely on the key assumption that linearity holds among training examples, which is reasonable if the label space is discrete, but has limitations when the label space is continuous as in regression. We show that mixing examples that either have a large data or label distance may have an increasingly-negative effect on model performance. Hence, we use the stricter assumption that linearity only holds within certain data or label distances for regression where the degree may vary by each example. We then propose MixRL, a data augmentation meta learning framework for regression that learns for each example how many nearest neighbors it should be mixed with for the best model performance using a small validation set. MixRL achieves these objectives using Monte Carlo policy gradient reinforcement learning. Our experiments conducted both on synthetic and real datasets show that MixRL significantly outperforms state-of-the-art data augmentation baselines. MixRL can also be integrated with other classification Mixup techniques for better results.

heading: 1 INTRODUCTION
text: As machine learning (ML) becomes widely used in critical applications including manufacturing and finance, data augmentation for regression becomes essential as it provides an opportunity to improve model performance without additional data collection. In comparison to classification tasks like object detection in images, the goal of regression is to predict one or more real numbers.
To emphasize the importance of data augmentation in regression, we provide a case study of semiconductor manufacturing. Here a common quality check is to measure the layer thicknesses of a 3-dimensional semiconductor and see if they are even. However, directly measuring each thickness results in destroying the semiconductor itself, so a recently-common approach is to take an indirect measurement by applying light waves on the semiconductor, measuring the spectrum of wavelengths that bounce back from all the layers, and use ML to predict the layer thicknesses from the spectrum data (see Fig. 4 in Sec. 4 for an illustration). With enough spectrum data and thickness information, ML models can be trained to accurately predict thicknesses from a spectrum. The main challenge is that there is not enough training data, and the only cost-effective solution is to augment small amounts of data that exist. Even a small improvement in model performance from the data augmentation has significant impact in this industry. In general, any regression task that predicts real values like emissions, stock prices, or even someone’s salary can also benefit from data augmentation.
Most data augmentation techniques are designed for image classification. In particular, Mixup (Zhang et al., 2018; Berthelot et al., 2019; Yun et al., 2019) is a popular data augmentation technique that is widely used for classification tasks, but is seldom used for regression because it assumes a distinct label space. The idea of Mixup is to mix pairs of examples using the key assumption that taking a linear interpolation between examples can be used to estimate the label of any examples in between. Mixup is known to effectively regularize the model. More recently, Manifold Mixup (Verma et al., 2019) has been proposed to improve the hidden representation and decision boundaries where two examples are mixed in multiple hidden layers of neural networks.
However, the Mixup techniques are not readily applicable to a regression setting because the key linearity assumption does not necessarily hold. Since the label space is continuous, taking a linear interpolation of examples that are very different either data-wise or label-wise may result in arbitrarily-incorrect labels as shown in Fig. 1a. As a result, the linearity assumption only holds to a certain extent, and the degree may vary for each example. Moreover, other data augmentation techniques for classification including image processing (e.g., flipping or rotating) and generative models (e.g., GAN (Goodfellow et al., 2014) and VAE (Kingma & Welling, 2014)) are even less applicable to a regression setting (see Sec. 5).
We propose MixRL, a data mixing augmentation framework that is the first to tailor Mixup for regression tasks using reinforcement learning. MixRL uses a stricter linearity assumption where it only holds within a certain data or label distance. These distance limits may vary by example, and we formulate the problem of learning for each example how many nearest neighbors it should be mixed with. MixRL employs a meta learning framework that estimates how valuable mixing an example is for reducing the model loss on a small validation set using Monte Carlo policy gradient reinforcement learning. MixRL’s framework is inspired by the recent Data Valuation using Reinforcement Learning (DVRL) framework (Yoon et al., 2020), which solves the different problem of measuring how individual examples contribute to model performance without any mixing involved. Fig. 1b shows how limiting the nearest neighbors to mix is better than mixing with all neighbors as in classification. To see if the augmentation is useful, we train simple models on the original four examples (i.e., no augmentation), the augmented data in Fig. 1a, and the augmented data in Fig. 1b. Evaluating the models on 20 random test examples results in Root Mean Square Error (RMSE; see Sec. 4) values of 0.2967, 0.5615, and 0.1834, respectively, where a lower RMSE is better. We thus conclude that carefully mixing examples is important for improving regression performance.
Experiments conducted on real and synthetic datasets show that MixRL shows better model performances relative to baselines, especially when the linearity is limited, and the mixing must be done selectively. In addition, MixRL only requires small validation sets and scales to large training sets.

heading: 2 LIMITED LINEARITY IN DATA AND LABEL SPACE FOR REGRESSION
text: We explain why the key linearity assumption used for Mixup in classification has limitations in a regression setting. In classification, the labels are discrete where many examples may have the same label. The original version of Mixup (Zhang et al., 2018) is to take a linear interpolation between any pair of examples xi and xj with the labels yi and yj to produce the new example λxi+(1−λ)xj with the label λyi + (1 − λ)yj where λ ∼ Beta(α, α). The linearity assumption turns out to be reasonable because the label difference between examples is only 0 or 1 and thus not that sensitive to the data difference. In contrast, the labels in regression are in a continuous space. Although there is still a many-to-one relationship where multiple examples may have the same label, the degree is much smaller than in classification. As a result, when two examples are mixed, the interpolated label can be arbitrarily different than the actual label, e.g., mixing the points a and d in Fig. 1a results in a
label nowhere near the actual label. In Sec. 4.1, we also show empirical results where the label error increases for larger data or label distances. Furthermore, mixing examples with larger data or label distances tend to have increasingly-negative effects on the model trained on the augmented training set. Figs. 2a and 2b show the model accuracies using RMSE for the Product dataset (described in Sec. 4) when mixing examples with different ranges of distances. Regardless of adjusting the label or data distance, there are diminishing benefits for larger distances.
How do we limit the data and label distances to improve Mixup for regression? One approach is to only limit the data distance, which limits the label distance as well. Suppose that the regression function f is continuous where limx→c f(x) = f(c) for any x and c within the domain of f . We show that a short-enough data distance sufficiently reduces the label distance as well. Given f ’s domain D and limx→c f(x) = L, the following is known to hold: ∀ , ∃δ s.t. ∀x ∈ D, if |x − c| < δ, then |f(x)−L| < . We can use this result to prove that ∀ , ∃δ s.t. ∀xi, xj ∈ D, if αxi + (1− α)xj = c, 0 ≤ α ≤ 1, and |xi − xj | < δ then the absolute difference between the mixed example’s xj value and L is small where |αf(xi) + (1 − α)f(xj) − L| = |α(f(xi) − L) + (1 − α)(f(xj) − L)| ≤ α|f(xi)− L|+ (1− α)|f(xj)− L| < α + (1− α) = . We cannot do the converse and limit the data distance by limiting label distance because there is a many-to-one mapping from data to labels, which means that two different examples may have identical labels (e.g., Fig. 1a’s a and d).
However, limiting the data distance to also limit the label distance may be too restrictive because there is not much correlation between the data and label distances in real data. Fig. 2c shows how the data distance relates to the label distance for the Product dataset. Even for small data distances, the label distance has a large range, which means that the data distance would have to be extremely limited. Hence, our solution is to limit the data and/or label distance as needed instead of just the data distance. This approach turns out to be more practical as we demonstrate in Sec. 4.4.

heading: 3 MIXRL
text: The goal of MixRL is to identify which examples to mix with which nearest neighbors. Instead of finding the actual distance limits themselves, we solve the identical problem of finding the number of nearest neighbors to mix per example for convenience. We use reinforcement learning because there is no training data on how mixing each example affects the model performance, and the reward function is non-differentiable as we explain below. MixRL’s framework is inspired by DVRL (Yoon et al., 2020), but we solve the different problem of mixing examples and address new issues.

heading: 3.1 POLICY OPTIMIZATION REINFORCEMENT LEARNING
text: Finding the optimal policy involves taking the gradient of the objective function J(θ) = Eπθ [R] where π is the policy, θ is its parameters, and R is the reward function. For MixRL, we would like to minimize the regression model loss on a small validation set when mixing a batch of examples. However, the validation loss is computed using the regression model, which does not involve θ. Hence we cannot analytically compute the differential of the reward function with respect to θ.
Instead, we use the REINFORCE (Williams, 1992) policy gradient algorithm, which is a Monte Carlo method that is also widely used for data valuation (Yoon et al., 2020) and neural architecture searching (Zoph & Le, 2017). It is known that∇θJ(θ) can be approximated (see Sec. A.1) as:
∇θJ(θ) ≈ 1
m m∑ i=1 R(τ i) T−1∑ t=0 ∇θ log πθ(ait|sit) (1)
where τ i is the ith state-action trajectory under policy πθ, m is the number of sample trajectories, T is the number of actions taken in a path, sit is a state at time t, and a i t is an action at time t.
In our setting of minimizing loss, REINFORCE performs gradient descent for each example decreasing θ by α∇θJ(θ) where α is a learning rate. Although the estimated gradient is unbiased, it is known to have a high variance, which we reduce using baseline techniques (Sutton & Barto, 2018).

heading: 3.2 FRAMEWORK
text: We define notations used in MixRL’s framework shown in Fig. 3. Let D = {(xi, yi)}Si=1 ∼ P be the training set where xi ∈ X is a d-dimensional input example, and yi ∈ Y is an e-dimensional label. Let Dv = {(xvi , yvi )}Vi=1 ∼ P t be the validation set, where P t is the distribution of the test set, which is not necessarily the same as P . Let fφ be a regression model, and L the loss function that returns a performance score comparing fφ(xi) with the true label yi using Mean Square Error (MSE). We assume a list N of possible data and label nearest neighbors (NNs) that can be mixed with an example. For instance, N could contain the options “1 data NN”, “2 data NNs”, and “2 label NNs”. The more fine-grained the NN options are, the more precisely MixRL can determine the optimal number of NNs to mix per example. We do not add a “0 NN” option because selecting and not selecting it have identical effects, which makes the policy network training unstable because nothing can be learned. Instead, we support this option separately by excluding examples that are not worth mixing as we explain later in this section. The possible NNs can be represented as a one-hot encoding vector of |N | values. We now define the states and actions, which is an important design choice of MixRL. A state st is a batch of examples Db = {((xi, yi), ki)}Bi=1 where {(xi, yi)}Bi=1 ⊆ D and each ki is an index of an NN option in N that specifies how many data or label NNs xi should be mixed with. An action at is then choosing Dm ⊆ Db where each (xi, yi) in Dm is mixed with its N [ki] NNs in D. A policy πθ(Dm|Db) returns the probability of selecting Dm at state Db. For each episode, MixRL selects a batch (state) once and chooses a subset of the batch to obtain a reward (action) once. Since there is only one time step, the transition function does not play a role.
A naı̈ve implementation of the policy network would have an input dimension of B× (d+ e+ |N |) and an output dimension of 2B , which may be too large to train for large batch sizes. For example, in a typical setting of B = 1000, d = 100, e = 1, and |N | = 10, the input and output dimensions
Algorithm 1: Pseudo code for Mixup value network training. Input : Training set D, validation set Dv , nearest neighbor options N , learning rate α, reward scaling constant C, moving average window W Output: Mixup value network hθ Initialize θ, Base = 0; while until convergence do
Sample Db = {(xi, yi), ki} B 1 ∼ P × Uniform(N); Dm = ∅; for ((xi, yi), ki) ∈ D
b do Add (xi, yi, ki) to D
m with probability hθ(xi, yi, ki); Train regression model fφ with initialized φ on Db ∪Mix(Dm,D, N); Loss = 1
V ∑V i=1 L(fφ(x v i ), y v i );
θ = θ − α · C · (Loss−Base) · ∇θ log(πθ(Dm,Db)); Base = W−1
W Base+ 1 W Loss;
return return hθ;
become 111, 000 and 21000, respectively. Instead, we can significantly reduce the network size by assuming independence among examples and decomposing the policy network’s prediction into two phases. First, a Mixup value network hθ(x, y, k) is used to estimate the probability of an example being mixed with k NNs. Next, we randomly select which examples to mix according to the probabilities. The value of πθ(Dm|Db) is thus ∏ (x,y,k)∈Dm hθ(x, y, k) ∏ (x,y,k)∈Db\Dm [1− hθ(x, y, k)]. Also, hθ has input and output dimensions of only d+ e+ |N | and 1, respectively, and is practical. The objective function J(θ) is the validation set loss lθ of fφ trained on Db ∪ Mix(Dm,D, N) (either from scratch or from a pre-trained state) where Mix(Dm,D, N) mixes each (xi, yi) in Dm with its N [ki] NNs in D:
J(θ) = l(φ) = E (xvi ,y v i )∼P t L(fφ(xvi ), y v i ) (2)
Using Eq. 1, the gradient of Eq. 2 can be approximated using the validation set Dvas:
∇θJ(θ) = ∇θl(θ) ≈ 1
V V∑ i=1 L(fφ(xvi ), y v i ) · ∇θ log(πθ(Dm,Db)) (3)
where∇θ log πθ(Dm,Db) =∇φ ∑ (x,y,k)∈Dm log hθ(x, y, k)+ ∑
(x,y,k)∈Db\Dm [1− hθ(x, y, k)]. We also use reward scaling (Henderson et al., 2018) to further improve the training.
Algorithm 1 shows the pseudo code for training hθ. The computational complexity is not directly related to the training set size, but depends on the number of iterations needed to train hθ and how long each iteration takes. Another factor is the number of possible NNs |N | where a higher number may result in slower training. We show in Sec. 4.2 that MixRL scales to large training sets.
Once we train hθ, we choose the (x, y, k)’s with the highest hθ(x, y, k) values of at least a threshold T . We optimize T ’s value using the validation set (see Sec. B.1). We ignore any (x, y, k) whose (x, y) pair has already been chosen with a different k value for the same data or label distance. We apply T in order to exclude (x, y, k)’s that are not worth mixing (i.e., make them mix with 0 NNs).
Although actor-critic methods (Mnih et al., 2016; Schulman et al., 2017) improve on REINFORCE by training both value and policy networks, using the two networks may not be practical in our setting due to the high input dimensions. For a policy network, we can reduce its dimension using the fact that the log probabilities can be calculated by multiplying selection probabilities. Unfortunately, we cannot use the same trick for a value network because its output values are not probabilities.

heading: 4 EXPERIMENTS
text: We provide experimental results for MixRL. We evaluate the regression models trained on the augmented training sets on separate test sets and repeat all model trainings 5 times. We use PyTorch (Paszke et al., 2017), and all experiments are performed using Intel Xeon Silver CPUs and NVidia RTX GPUs. More experimental setting details are in Sec. B.1.
Table 1: Settings for the four datasets.
Dataset Data dim. Label dim. Training set size Val. set size Test set size
NO2 7 1 200 200 100 Product 580 20 300 300 200
Synthetic 226 4 100 100 100 Airfoil 5 1 1003 300 200
Measures We use two accuracy metrics. RMSE = √
1 n ∑n i=1(yi − ŷi)2 is the absolute difference
between predicted labels and true labels where a lower value is better. R2 = 1− ∑n i=1(yi−ŷi)
2∑n i=1(yi−ȳ)2
is the relative improvement compared to when returning the label average as a prediction. The value is in the range [0, 1] where a higher value is better. The two measures complement each other.
Datasets We use three real and one synthetic datasets. The NO2 emissions dataset (Aldrin, 2004) contains traffic and meteorological information around roads. The features include cars per hour, wind speed, temperature, and others. The label is the NO2 concentration. The Product dataset obtained through a collaboration with a company contains spectrum data that is generated using the procedure described in Sec. 1 and illustrated in Fig. 4 on 20-layer 3-d semiconductors. The Product dataset is proprietary, so for clarity we also experiment on a public synthetic dataset (Synthetic) provided by the DACON challenge (DACON Co., Ltd., 2020) where a simulator generates spectrum data using the same procedure as Product assuming 4-layer 3-d semiconductors. Finally, the Airfoil UCI dataset (Dua & Graff, 2017) contains aerodynamic and acoustic test results for airfoil blade sections in a wind tunnel where the features include frequency, angle of attack, and chord length. Table 1 compares the four datasets in more detail. Note that Product and Synthetic have high dimensions, while Airfoil has the largest training set size.
MixRL Settings For the regression model, we use a multi-layer perceptron (MLP) with 2 to 4 hidden layers and set the number of nodes per hidden layer to be [512, 256] for NO2, [2048, 1024, 512, 256] for Product, and [2048, 1024, 512, 256] for Synthetic. We employ layer normalization, early stopping for regularization, and reward scaling and baseline techniques that improve the performance and stability of reinforcement learning. We use the Adam optimizer for all MLP trainings. When mixing examples, we set the Mixup ratio λ to 0.5 unless specified otherwise. This value empirically results in the best Mixup performance in a regression setting, and we give an explanation in Sec. B.3. For the possible numbers of NNs for NO2, our default setting is the series of integers {4, 16, 64} for both data and label distances, i.e., |N | = 6. For Product, Synthetic, and Airfoil, the default series are {100, 150, 200, 250}, {4, 16, 64}, and {1, 2, 4}, respectively.
Baselines We employ four baselines. First, we train a regression model on the labeled data without any data augmentation (“No Augmentation”). Next, we faithfully implement the original Mixup algorithm (Zhang et al., 2018) where all example pairs can be mixed (“Original Mixup”). We also implement Manifold Mixup (Verma et al., 2019), which is a state-of-the-art Mixup technique for classification that is also applicable to regression. Finally, we compare with a simplified version of MixRL where we use a single k NN value to mix all examples with their data and label neighbors (“Global kNN”). This k parameter is tuned using Bayesian Optimization (Mockus, 1974).

heading: 4.1 LIMITED LINEARITY IN REGRESSION
text: We empirically show that the linearity assumption does not hold from certain data or label distances using the following simple experiment. Given a dataset, we first train a regression model. We then
generate mixed examples and compute the RMSE between their interpolated labels and the labels predicted by the regression model. While the regression model is not perfect, the relative trend of the error gives sufficient insight. Figs. 5a and 5b indeed show increasing label errors for larger label and data distances for the NO2 and Product datasets, respectively. Figs. 2c and 5c also show how label and data distances do not necessarily have a linear relationship for the two datasets. We also train regression models on top of augmented training sets where we only mix examples within certain distance ranges. As a result, mixing examples with larger data or label distances has increasinglynegative effects on the model accuracy as shown in Figs. 2a, 2b, 5d, and 5e.

heading: 4.2 ACCURACY AND RUNTIME RESULTS
text: We compare MixRL with the four baselines using the settings in Table 1. Table 2 shows the results for the three real datasets (the Synthetic dataset results are in Sec. B.2). MixRL significantly outperforms all the baselines in terms of regression model accuracy, which demonstrates that it is able to pinpoint which examples should be mixed with how many kNNs. Figs. 6a and 6b show which kNN options are frequently used for NO2 and Product (the kNN options for Synthetic and Airfoil are in Sec. B.2). Although the runtime is slower than the baselines due to the reinforcement learning, the gained accuracy can be worth the effort in real applications because data augmentation is a one-time cost that can run in batch mode. Another observation is that Original Mixup performs relatively bet-
ter for Product than other datasets because linearity holds more as evidenced by the large k = 267 value of Global kNN. In general, MixRL outperforms Original Mixup by a larger margin the more limited the linearity. We also integrate MixRL with Manifold Mixup where examples are mixed in multiple layers of the MLP with their NNs (see Sec. B.4 for more details). The results are mixed where the integrated method performs the best on the Airfoil dataset, but not on the other datasets.
Validation Set and Scalability Figs. 6c and 6d show the validation set size impact on MixRL’s accuracy for the NO2 and Product datasets. The Airfoil dataset results are similar and shown in Sec. B.2. We observe that a small validation set is sufficient for MixRL to be effective. Also MixRL scales well against increasing training set sizes as shown in Fig. 6e.

heading: 4.3 K NEAREST NEIGHBOR OPTIONS
text: We investigate how changing the kNN options N affects MixRL’s performance in Table 3 using the NO2 and Product datasets. The other dataset results are similar and shown in Sec. B.2. For each dataset, we compare its default series with shorter and longer series. We observe that providing more kNN options improves MixRL’s performance, but only to a certain extent. Hence, MixRL performs well even with a small number of options.

heading: 4.4 ABLATION STUDY
text: We perform an ablation study to investigate the effect of limiting label and data distances in Table 4 using the NO2 and Product datasets. The other dataset results are similar and shown in Sec. B.2. We compare MixRL with three variants: (1) MixRL that does not exploit any distances and selects examples that have the highest hθ values when fully mixed with all the other examples; (2) MixRL that does not use label distances; and (3) MixRL that does not use data distances. We observe that using both distances is necessary for the best accuracy results as we suggested in Sec. 2.

heading: 5 RELATED WORK
text: Data Augmentation for Regression There are largely two branches of work for data augmentation in regression. One is semi-supervised regression (Kostopoulos et al., 2018; Zhou & Li, 2005; Kang et al., 2016; Kim et al., 2020) where the goal is to utilize unlabeled data for training. In comparison, MixRL does not assume the availability of unlabeled data. Another branch is data augmentation when there is no unlabeled data, which is our research focus. To the best of our knowledge, there is no technique tailored to regression, so we instead cover data augmentation techniques for classification that can be extended to regression.
We explore data augmentation techniques for classification and investigate if they can be used in a regression setting. There are largely three approaches: generative models, policies, and Mixup techniques. Generative models including GANs (Goodfellow et al., 2014) and VAEs (Kingma & Welling, 2014) are popular in classification where the idea is to generate realistic data that cannot be distinguished from the real data by a discriminator. However, a major assumption is that the labels of the generated examples are the same, which does not necessarily hold in a regression setting where most examples may have different labels. Another approach is to use policies (Cubuk et al., 2019), which specify fixing rules for transforming the data while maintaining the label value. For example, image processing policies may flip, rotate, or adjust the brightness of images to generate new valid images. However, in a regression setting, the same transformed examples may now have different unknown labels, making them unsuitable for training. The recently-proposed Mixup (Zhang et al., 2018; Berthelot et al., 2019; Yun et al., 2019; Berthelot et al., 2020; Hendrycks et al., 2020; Verma et al., 2019) takes the alternative approach of generating both data and labels together by mixing existing examples with different labels assuming linearity between training examples (Chapelle et al., 2000; Wu et al., 2020). Although the Mixup paper mentions that its techniques can easily be extended to regression, we make the non-trivial observation that the linearity assumption does not necessarily hold where mixing examples with far-away examples may not be beneficial and even detrimental to model performance. Instead, MixRL shows good performance by carefully choosing which nearest neighbors to mix with, a technique that only works in a regression setting.
Policy Optimization Reinforcement Learning Within reinforcement learning (RL) (Sutton & Barto, 2018; Kaelbling et al., 1996), we are interested in policy searching methods (Neumann & Peters, 2015) where our goal is to find the best policy for mixing examples. In particular, MixRL’s framework is inspired by the recent Data Valuation using Reinforcement Learning (DVRL) framework (Yoon et al., 2020), which uses REINFORCE (Williams, 1992) to estimate the value of data in terms of how much it improves the model performance. In comparison, we solve the completely different problem of data augmentation for regression where we extend Mixup using REINFORCE. Some data augmentation techniques use actor-critic methods like A3C (Mnih et al., 2016) and PPO (Schulman et al., 2017) that train value networks along with policy networks. For example, AutoAugment (Cubuk et al., 2019) uses PPO to search for data augmentation policies that dictate how to modify existing examples to generate additional training data for image classification. However, using actor-critic methods is practically challenging in our setting due to the large value networks that need to be trained as we discussed in Sec. 3.2.

heading: 6 CONCLUSION
text: We proposed MixRL, which is to our knowledge the first Mixup-based data augmentation meta learning framework tailored to regression tasks. We observe that linearity only holds up to a certain data or label distance per example in a regression setting. Hence, MixRL extends Mixup by determining the k nearest neighbors to mix per example using Monte Carlo policy gradient reinforcement learning where the objective is to minimize the regression model’s loss on a small validation set. Since MixRL only limits the examples to mix, it can be integrated with any existing Mixup technique for classification. In our experiments, we showed that MixRL outperforms various data augmentation baselines for regression by effectively selecting the k nearest neighbor options.
Ethics Statement MixRL demonstrates that data augmentation can significantly enhance the model accuracy for regression tasks. However, if not controlled, data augmentation may also introduce new bias in the data that makes the model more discriminative. Thus an interesting future work is to augment data while ensuring model fairness.
Reproducibility Statement We add MixRL’s source code in the supplementary. We also provide more details on Eq. 1, experimental settings, and experimental results in the appendix.

heading: A APPENDIX – FRAMEWORK
text: A.1 GRADIENT OF J(θ)
We continue from Sec. 3.1 and derive Eq. 1 (Brunskill, 2021):
∇θJ(θ) ≈ 1
m m∑ i=1 R(τ i) T−1∑ t=0 ∇θ log πθ(ait|sit) (1)
where τi is a state-action trajectory under policy πθ, m is the number of sample trajectories, T is the number of actions taken in a path, st is a state at time t, and at is an action at time t. Suppose that P (τ ; θ) is the probability of a trajectory τ when using the policy πθ, and R(τ) is the reward of τ . Then J(θ) can be expressed as ∑ τ P (τ ; θ)R(τ). In addition, let us specify s0 as the initial state, µ(s0) as the initial state’s distribution, and P (st+1|st, at) as the dynamics model that specifies which predicts the next state st+1 when taking action at from state st. We can then take the gradient of J(θ) as follows:
∇θJ(θ) =∇θ ∑ τ P (τ ; θ)R(τ)
= ∑ τ ∇θP (τ ; θ)R(τ)
= ∑ τ P (τ ; θ) P (τ ; θ) ∇θP (τ ; θ)R(τ)
= ∑ τ P (τ ; θ)R(τ) ∇θP (τ ; θ) P (τ, θ)
= ∑ τ P (τ ; θ)R(τ)∇θ logP (τ ; θ)
≈ 1 m m∑ i=1 R(τ i)∇θ logP (τ i; θ)
= 1
m m∑ i=1 R(τ i)∇θ log
[ µ(s0)
T−1∏ t=0 πθ(a i t|sit)P (sit+1|sit, ait)
]
= 1
m m∑ i=1 R(τ i)∇θ [ logµ(si0) T−1∑ t=0 log πθ(a i t|sit) + logP (st+1|sit, ait) ]
= 1
m m∑ i=1 R(τ i) T−1∑ t=0 ∇θ log πθ(ait|sit)

heading: B APPENDIX – EXPERIMENTS
text: B.1 MORE EXPERIMENTAL SETTINGS
We continue describing our experimental settings from Sec. 4. For the regression model, we use layer normalization (Ba et al., 2016) for the Product, Synthetic, and Airfoil datasets. We also use early stopping to prevent overfitting on the validation set for all the datasets. For the mixup value network hθ, we use reward scaling and baseline techniques that improve the performance and stability of reinforcement learning. For the mixup value network hθ, we use a multi-layer perceptron with 4 hidden layers and set the number of nodes per hidden layer to be [100, 100, 100, 100] for the four datasets. For hθ, we use a learning rate of α = 0.001 for all datasets. For the regression model fφ, we use a learning rate of 0.0001 for Product and 0.001 for the other datasets. When training hθ, we set the batch size B to be the training data’s size. When training fφ, we use a batch size of 32 for NO2, 64 for Product, 32 for Synthetic, and 64 for Airfoil. We use a reward scaling (Henderson et al., 2018) constant of C = 5 for NO2, C = 1 for Product, C = 0.1 for Synthetic, and C = 1 for
Airfoil. We use a moving average window of size W = 20 for the three datasets. When setting the threshold T for limiting examples to mix, we choose a value that results in a high model accuracy (low RMSE) on the validation set.
When constructing the Synthetic dataset, we take a subset of the entire DACON challenge dataset (DACON Co., Ltd., 2020). While the the range of thicknesses for the full datasets is [10, 300] for all four layers, we only use examples with thicknesses within the range [10, 50]. The purpose is to reduce the training time while still making a clear comparison between MixRL and the other methods.
To improve the runtime of MixRL, we use pre-trained regression models for the Product, Synthetic, and Airfoil datasets. First, we train the base regression model on the training data. Then, we fine-tune the pre-trained model to obtain the validation losses.
B.2 MORE Synthetic AND Airfoil DATASET EXPERIMENTS
Accuracy and Runtime Results We continue from Sec. 4.2 and show the accuracy and runtime results for the Synthetic dataset in Table 5. The results are similar to the Product dataset in Table 2.
kNN Options Frequency Histogram We continue from Sec. 4.2 and show which kNN options are frequently used in our default settings for the Synthetic and Airfoil datasets in Figs. 7a and 7b, respectively. Compared to the Product dataset, Synthetic has no cases where k = 0 because mixing is more beneficial to the model’s accuracy according to the validation set.
Validation Set Size and Accuracy We continue from Sec. 4.2 and show how the validation set size impacts MixRL’s accuracy for the Airfoil dataset in Fig. 8. The observations are similar to those of the other datasets where a small validation set size is sufficient for MixRL to perform well.
kNN Options Comparison We continue from Sec. 4.3 and provide the kNN options experiments for the Synthetic and Airfoil datasets in Table 6. The observations are similar to those of Table 3 where a small number of options is sufficient for good model accuracy.
Ablation Study We continue from Sec. 4.4 and provide the ablation study for the Synthetic and Airfoil datasets in Table 7. The results are similar to those in Table 4 where using both label and data distances results in the best model accuracies.
B.3 JUSTIFICATION FOR USING λ = 0.5
We continue from Sec. 4 and provide more details on why setting λ = 0.5 empirically results in the best Mixup performance in a regression setting. Figure 9 shows MixRL’s accuracy when we vary λ using the same setting in Table 1 for the NO2 and Product datasets. As a result, the accuracy peaks when λ = 0.5. We suspect that generating examples in the sparsest areas helps the regularization the most. In a classification setting, however, this strategy may not be ideal especially if the generated examples are too close to a decision boundary that is in the middle of the mixed examples.
B.4 INTEGRATING MIXRL WITH MANIFOLD MIXUP
We continue from Sec. 4.2 and provide more details on integrating MixRL with Manifold Mixup (Verma et al., 2019). We first run MixRL and determine which examples to mix. Next, instead of mixing examples in the training data before model training, we mix examples on multiple layers in the MLP during model training. For the second mixing step, we use Manifold Mixup as is
where we start with a set of eligible layers for mixing. (In our experiments, we always set the first three layers as eligible.) For each training batch, we randomly select a mixing layer and mix the features and labels of the examples that MixRL decided to mix.

Review summaries:
This paper improves on the idea of mixup by selecting samples for mixup via a model that selects suitable pairs found using knn in a batch. In order to provide a learning signal to this discrete process, the authors apply REINFORCE, using the loss of the downstream regressor (not applied to classification tasks). They show promising results on a number of regression tasks.

The authors propose MixRL to improve upon mixup in regression settings. MixRL is used to impose a proximity constraint on the input/output pairs that are mixed during mixup-based data augmentation, by predicting how many nearest neighbors to utilize, from a small set of pre-specified options, based on feedback from evaluating the validation set. Consistent but small gains over mixup and manifold mixup are realized on several datasets.

To apply Mixup for regression tasks, the paper first utilizes the stricter assumption that linearity only holds within specific data or label distances for regression. Then this paper proposes a data mixing augmentation method called MixRL. The goal of MixRL is to identify which examples to mix with which nearest neighbors. MixRL employs a meta-learning framework that estimates how important mixing a sample is to minimize the model loss on a validation set using policy gradient reinforcement learning.