Improving the Accuracy of Learning Example Weights for Imbalance Classification
Authors: Yuqi Liu
Bin Cao
Jing Fan
Venue: ICLR 2022 Poster (2022)
Keywords: Imbalance classification
Meta learning
Data weighting.

Abstract: To solve the imbalance classification, methods of weighting examples have been proposed. Recent work has studied to assign adaptive weights to training examples through learning mechanisms, that is, the weights, similar to classification models, are regarded as parameters that need to be learned. However, the algorithms in recent work use local information to approximately optimize the weights, which may lead to inaccurate learning of the weights. In this work, we first propose a novel mechanism of learning with a constraint, which can accurately train the weights and model. Then, we propose a combined method of our learning mechanism and the work by Hu et al., which can promote each other to perform better. Our proposed method can be applied to any type of deep network model. Experiments show that compared with the state-of-the-art algorithms, our method has significant improvement in varieties of settings, including text and image classification over different imbalance ratios, binary and multi-class classification.

Published as a conference paper at ICLR 2022
IMPROVING THE ACCURACY OF LEARNING EXAMPLE WEIGHTS FOR IMBALANCED CLASSIFICATION
Yuqi Liu & Bin Cao'& Jing Fan
College of Computer Science, Zhejiang University of Technology. Hangzhou, China (1 (uiuyuqi, bincao, fan jin) ez jut. cn
ABSTRACT
To solve the imbalanced classification, methods of weighting examples have been proposed. Recent work has studied to assign adaptive weights to training exam- ples through learning mechanisms. Specifically, similar to classification models, the weights regarded parameters that need to be learned. However, the algorithms in recent work use local information to approximately optimize the weights, which may lead inaccurate learning of the weights. In this work, we first propose a novel mechanism of learning with a constraint, which can accu- rately train the weights and model. Then, we propose combined method of our learning mechanism and the existing work, which can promote each other to perform better. Our method can be applied to any type of deep network model. Experiments show that compared with state-of-the-art algorithms, our method has significant improvement in varicties settings, including text and image classifi- cation over different imbalance ratios, binary and multi-class classification.
INTRODUCTION
Classification is fundamental task in machine learning, but in practical classification applications, the number of examples among classes may differ greatly, even by several orders of magnitude. Standard learning methods train the classification model on such an imbalanced data set, which makes the trained model biased. This bias is that the model will prefer the majority class and easily misclassify the minority class examples. This class-imbalance problem exists in many domains, such as Twitter spam detection (Li & Liu 2018), named entity recognition (Grancharova et al., 2020) in text classification, and object detection (Oksuz et al. 2020), video surveillance (Wu & Chang 2003) in image classification.
There are very rich research lines on using the methods of weighting examples to solve the class imbalance problem. In general, the weight of the minority class is higher than that of the majority class, so that the bias towards the majority class is alleviated. Typically, the example weight value of each class often set inverse class frequency (Wang et al. 2017 or inverse square root of class frequency (Mahajan et al.) 2018). However, the example weights in these methods are designed empirically, hence they can not be adapted to different datasets and may perform poorly.
Recent work has studied the methods of using learning mechanisms to adaptively calculate the ex- ample weights. Ren et al. (2018) propose use meta-learning paradigm (Hospedales et al. 2020) to learn the weights. In this method, the example weights can be regarded the classification model is a learner. The meta-learner guides the learner to learn by weighting the example loss in the model optimization objective. More specifically, the model objective is to get the optimal model that minimizes the example-weighted loss of the imbalanced training set. Ob- viously, different weights will affect the performance of the optimal model. Which weight values make the corresponding optimal model the best? This method collects a small balanced validation set and evaluates the weight values through the validation performance of the model. Therefore, the meta-learner objective, namely meta-objective, gives the best weights that make the optimal model minimize the loss of the balanced validation set. This optimization problem challenging, The key is that, in the meta-objectie the weights indirectly affect the loss through the optimal model, so it
meta-learner and
Conresponding author

---

Published as a conference paper at ICLR 2022
is necessary to clearly define the dependence of the weights and the optimal model in the model ob- jective for optimizing the weights. However, it is expensive to get this dependence through multiple gradient descent steps in the model objective. Ren et al. (2018) propose an online approximation method to estimate this dependence, that is, the method trains the model using a gradient descent step in the model objective and then can determine the relationship between the weights and the trained model in this step. Hu et al. (2019) propose to update the example weights iteratively to replace the re-estimation proposed by Ren et (2018), but also adopt the local approximation to optimize the weights. However, this approximation only considers the influence of the weights on the trained model in a short term (in descent step), resulting inaccurate learning of the weights.
In this paper, we firstly propose novel learning mechanism that can obtain the precise relationship between the weights and the trained model in the model objective, so that the weights and model can be optimized more accurately. In this mechanism, we convert the model objective into an equation of the current model and weights. Then, we derive their relationship from this equation, and then we use this relationship to optimize the weights in the meta-objective and update the corresponding model. Since this optimization process always satisfies this equation, we call it learning with a constraint. However, the mechanism only uses the model objective to calculate the relationship but does not optimize the model for the model objective. To solve this problem, we integrate the method proposed by Hu al. (2019) into our learning mechanism and propose combined algorithm. In this algorithm, the method of Hu et al. can help to further optimize the model in the model objective, and our learning mechanism can make the weights and model learn more accurately. Finally, we conduct a lot of experiments to validate the effectiveness of this algorithm. The experimental settings include (I) different domains, namely text and image classification; (2) different scenarios, namely binary and multi-class classification, (3) different imbalance ratios. The results show that our algorithm not only outperforms the state-of-the-ar art (SOTA) method in data weighting but also performs best among other comparison methods in varieties of settings.
The remainder of this paper is organized as follows. Section 2 introduces preliminaries of the two objectives and the main idea Hu et al.] (2019). Section Section[Jpresent presents our mechanism of learning with a constraint and the combined algorithm. SectionA4reeeett the experimental settings and evaluation results. Section 5 summarizes the related work and Section 6 concludes this paper.
2 PRELIMINARIES AND NOTATIONS
Let (xr,y) v) be the input and target pair. For example, in image classification, or is the image and y is the image label. Let Drrain denote the train set, and DD Dtrain = {(1,, y1), 1 < i & N). Let Drat be a small balanced validation set, and Dvel {(1, 31), 1 E i E M) where M < N. We denote neural network model as 4 (x, 0), where 0 E RK is the model parameter. The predicted value ÿ (zz, 0). We use loss function f(. y) to measure the difference between predicted value ý and target value y, and the loss function of data is defined as f:( for clarity. Standard training method is to minimize the expected loss on the training set: Σ (:),), and each example has same weight. However, for an imbalanced data set, the model obtained by this method will be biased towards the majority class. Here, we aim to learn model parameter @ that is fair to the minority class and the majority class by minimizing the weighted loss of training examples:
0* (w) = arg ming waJc(()
(1)
where w = (wi, .... WN)" is the weights of all training examples. We use Cirain to represent the weighted loss the training set Dtrain- For a given w. can obtain the corresponding optimal d* from [ Thus, there is a dependence between 0 and wr and we write it as 0* = 8* (w).
Learning to Weight Examples The recent work (Ren et al. 2018) proposed a method of learning the weights of training examples. In this method, the optimal w is to make the model parameter O" obtained from Eq minimize the loss on a balanced validation set. It means that this model performs well on a balanced validation set, and it can fairly distinguish examples from different classes. Formally, the optimal w is given as
M
w = arg min
1
Σ
(w))
M
2

---

Published as a conference paper at ICLR 2022
where the superscript stands for validation set. Let Leat be the loss the validation set vaal-
Learning the Parameters The recent work (Hu et al., 2019) introduced an algorithm of solving the model parameter 0 and weight w". The algorithm optimizes 0 and alternately until convergence. In each iteration, the algorithm utilizes a gradient descent step in Eq approximate the relation- ship between A and w, and then calculates the gradient V Wealal and VoLtrain to update and respectively.
More specifically, at the t-th iteration, the algorithm first caleulates the approximate relationship between 0 and wo through the 1-th gradient descent step in Eq We define a matrix F(0) (VSi (0), V (0)), whose column vector represents the derivative of [=(0) with respect 0, so we calculate the derivative of VeCtrain with respect to f as VøLtrain = F(f)w: w. Then, the t-th gradient descent step of is given as
= -
(3)
where is the descent step size on O. In order to avoid very expensive calculations, the algorithm ignores the influence of w on O,. Therefore, in the single gradient descent step, 0,-1 linearly depends on W.
Then, based on this linear dependence, the algorithm can calculate the gradient V V.rcoa and uses gradient descent to update w, and then updates 9 again to make it perform better on validation set. We can observe that the w acts on d(+1 and then affects Lvai. Thus, combining Eq we can calculate the gradient 6..1
Substituting the updated 02+1 into Eq 2 we have Leat = *े Σm ft
L (0
V Voffval (V.odet1) V Voey Lout -10 F( V Loaa, so the update
t step is given as
W++1 Wt Te (0.)" f
where Te the descent step size on w. According gradient descent theory, when Ye appropri- ately small, Leat(W) (W++1) & Lival (wee. This means using W+++1 to update 0 performs better than uy- Therefore, the algorithm substitutes the updated W[+1 into Ea obtain the new update on
0.41 0, Fo F(,)
(5)
where 8.41 satisfies Leal (0(+1) S Leal (0.4)) that is, 0.41 have better validation performance than 0411- Finally, the algorithm repeatedly calculates Eq 3 and 5 and alternately optimizes 0 and until convergence.
NEW METHOD OF LEARNING THE PARAMETERS
In this section, we introduce a new method to learn the model parameter and weight w" in Eq and 2q First, in Section 3.1 we propose to learn 0 and w with constraint, which can accurately optimize and w. Then, in Section 3.2 we propose a combined method to train and w to make the model parameter have better performance.
3.1 LEARNING WITH CONSTRAINT
In the section, we first analyze the difficulty of solving 0* and w". Gradient-based optimization is commonly used method in machine learning. Thus, we first need calculate the gradient VaLtrain and V.r Based on 2 we have V.,fral = (V.. V Loat- However, difficult to explicitly give the form of function 0* (w), resulting in Cw cannot be calculated directly. The previous work obtained the relationship between 0 and w through the gradient descent process of 8, and only considered the influence of on in single gradient descent step. Based on this relationship. calculating the gradient and updating the parameter is not precise.
Here, obtain the relationship between 0 and from new perspective. First, observe the gradient VoLtrains that is,
V VoLirain F(0)x=
3

---

Published as a conference paper at ICLR 2022
Algorithm 1: Learning to Weight Examples Using Combination Method
Input
The network model parameter 0 The weight of training examples w Training set D Prain:: Validation set Pal
The number of iterations of the combination method T
The number of iterations of our method T"
Initialize model parameter 0 and weight w for t= 0...
do Calculate the relationship between 0 and w on Divain through Eq Optimize w on Daal through E
Update 0 through 5 for t' = 0 T do
Calculate the derivative v on Derain through 7
Optimize on Deal through Eq 8
Update through Eq 9
Output: Trained model parameter 0* and weight w
where e the gradient value. We can see that changing the value of w can find corresponding 0 to satisfy 6q It means that there is a functional relationship between 0 and w in Eq Because all 0 and satisfying this equation have the same value of VoLtrain. we also call Eq constraint of i and w. In particular, the optimal model parameter 0 and w satisfy the constraint: F(f")w 0. Then, we can make use of the constraint to derive precise relationship between and w. Our network model may be very complex, and cannot explicitly give the functional form of 0 and according to the constraint. However, by applying the implicit function theorem, the derivative of with respect to in 6 can be obtained as follow
V = - - (oF(())))) = F
where H E Rk x RK is the Hessian matrix, namely, the second derivative of Ltrain with respect to 0. However, calculating an exact Hessian matrix very expensive. Especially nowadays network models have huge amount of parameters. In addition, in this case, we require the inverse of H, rather than H itself. Therefore, we adopt diagonal approximation to evaluate In other words, we only need to calculate the diagonal elements of H. Furthermore, is trivial to calculate the inverse by taking the reciprocal of the diagonal elements. Let E R& be the reciprocal of the diagonal elements of H. Then, the derivative is evaluated as
(Bishop) (0066)
V = ddaggh)
Next, we can make use of the derivative calculate the gradient V., Co.lea and then update and 0. The update process always satisfies the constraint of Eq so we call learning with a constraint. Combining 7 we have V. FF()) diag(h) V Thus, the update of is
w' = - n/, F(0)" diag(h)Vooog
where V/a is the step size. Then, we use the updated w' to calculate the corresponding 0' in the constraint. Since we do not know the explicit functional form and w in Eq we use the first order derivative to approximate 6'. Combing 7 0' is evaluated as
0' ~ 0 V.w w)
(())(' -w)
Finally, under the condition of satisfying the constraint, we repeatedly optimize w and 6, corre- sponding to Eq & and Eq until convergence. The detailed proof of this convergence can be found
Theorem in Appendix A.2
3.2 LEARNING A COMBINED WAY AND IMPLEMENTATION
The method we proposed in Sectior still has shorteoming. In the method, we make use of the gradient Vo VaCtrain of To obtain a constraint VoLtrain = c, and then calculate the solution of w and 0 under the constraint. However, this method only ensures that the solution is optimal in Eq
4

---

Published as a conference paper at ICLR 2022
under the constraint, and cannot guarantee that the solution of 0 is optimal in Eq Because method only use [q obtain the constraint, but not to optimize for Eq and when c # 0, the solution of 0 is not optimal in [q
In order to calculate the better solution, our method needs to be combined with another algorithm that can make f reach the optimal in Eq The method of Hu al. in Section 2 choice, rather than directly updating using the gradient VyLirain- Because will first adjust w and then update 0 based on the new w. It is explained in Sectior[] 0 obtained in this way has better validation performance than 0 directly using gradient descent.
is a more appropriate
Therefore, we propose way to learn 0 and by combining our method in Section with the method (Hu et al.) in Section In this way. we alternately use these two methods to learn I and w. In each iteration, we first update 9 and to using the method (Hu et al.). It can make 9 reduce the value of Lirain and approach the optimal in Eq I while 0 also reduces Leal and perform well on validation set. Then, we optimize I and w using our method until convergence, so that further reduces Leal and has the best verification performance among all 0 with the same gradient VeLtrain.
This combined way can overcome the shortcomings of each method. On the one hand,the method (Hu et al.) can use the gradient VaLtrain to update 0 and make 0 close to the optimal in [ It makes up for the shortcoming that our method cannot optimize [ On the other hand, the method (Hu et al.) only considers the influence of on 0 in single gradient descent step, and then uses this approximation to optimize Eq Hence, obtained by the method (Hu et al.) may not be optimal for Our method make use of constraint to derive accurate functional relationship between 0 and W. Thus, by optimizing 0 through our method, a better solution can be obtained.
2q
This combination algorithm is listed in Algorithm [ It takes T iterations to alternatey use two methods to optimize 0 and w. In (-th iteration, it first adopts the method (Hu et al.) to update w and method to optimize w and 0 repeatedly T times (lines 6-9), making 0 converges under the current constraint. Finally, it outputs the trained model parameter 0 and weight wt, The proof of convergence of the Algorithm [ can be found in Theorem in Appendix
0 (lines 3-5), and then it uses
A.2 In addition, we discuss the convergence rate of Algorithm I According to the conclusion in the paper (Ren al., 2018). when we take T steps to update the parameter 0 through the method (Hu et al.), it can achieve VoLval || I O( 1
/T where || VeLeal is the update precision of parameter B.
For the method in Section 3.1 achieving the same precision requires O(vT) steps. More detailed proofs are Theorem 3 in Appendix A.3 Therefore, as the combined method, Algorithm I needs
T x ((VT)= = ) to converge.
EXPERIMENTS
In this section, we perform extensive experiments to validate the effectiveness of our method. First, we describe the experimental setup in detail. Second, compare different methods in two domains: text and image classification and in two situations: binary classification and multi-class classifica- tion. Third, design experiments study the performance of our method in different imbalance ratios. Moreover, we evaluate the performance of our methods with different metrics on a large-scale data in Appendix A.I
4.1 EXPERIMENTAL SETUP
Models. We choose two network models for text and image classification. Specifically, in text classification, we use the BERT (base, uncased) model (Devlin et al.] 2018) to extract the 768- dimensional representation of text data (Xiao) 2018) and then use a simple 4-layer fully connected network (FCN) for classification. The FCN model is given as Table D The pair of numbers in brackets respectively indicate the sizes of input and output of the linear layer. In addition, the first two layers apply rectified linear unit activation (ReLU) function to avoid the vanishing gradient problem during training, and the third layer uses nonlinear activation function (Tanh) to enhance the model learning ability. The last layer is the classification layer, in which the size of output depends the number of classes in a classification task. In image classification, CNN-C network model that is a sequence of 9 convolution layers. Noting that our method does not rely on the classification model, and can also be applied to other models.
use the ALL-
5

---

Published as a conference paper at ICLR 2022
Table 1: The network model for text classification
Table 2: Description of four data sets
Input 768-dimensional text representation
Data sets
Classes
Fine-tume/ / Pretrain
(768, 768) linear layer, ReLU
SST-2
2
2 x 500
(768, 768) linear layer. ReLU
SST-5
x 500
(768, 10) linear layer. Tanh
CIFARIO
10
10 X 4000
(10, size of labels) linear layer
CIFARI00
2
0
Data Sets and Model Preparation. We choose data sets for text and image classification, and we use part of training examples in the data sets to prepare for the subsequent training of the models. The information of the four data sets we used is shown in Table 2 In text classification, we use two popular benchmark datasets. We use the SST-2 sentiment analysis benchmark (Socher et al., 2013) for binary classification, and use the SST-5 sentence sentiment (Socher et al. 2013)) with 5 categories for multi-class classification. In image classification, we adopt the commonly-used CIFARIO (Schneider et al. (2019) for multi-class classification experiment and select the examples of class 0 and I from CIFARIO0 (Schneider et al. 2019) to form a data set for binary classification.
To make subsequent experiments on strong models, we use part of training examples to fine-tune the BERT and pre-train the ALL-CNN-C model respectively. In text classification, we use the text data in a specific domain to fine-tune BERT, so that we can extract the better text representations from the fine-tuned BERT and improve the performance of the FCN model. In image classification, we first pre-train the ALL-CNN-C model using image data, and then the pre-trained model can be helpful to improve downstream tasks for subsequent experiments. these examples. On SST-2 and SST-5 data sets, we take out 500 training examples of each class to fine-tune the BERT model. For the experiments on CIFARIO, we use 4000 training examples per class to pre-train the ALL-CNN-C model. In the experiments CIFAR1O0, we do not pre-train the model, because can perform well on this binary classification task without pre-training. Noting that the training examples used to improve the models will not be used in subsequent experiments.
T we list the number of
Comparison Methods. We compare our method with five approaches: (1) Baseline, a method without any processing. In other words, the classification model is directly trained on an imbalanced training set. (2) Proportion, a commonly used method that weights examples by inverse class fre- quency. (3) Hu et al.'s, is the SOTA approach (Hu al. 2019) for data weighting, which is described in Section 2 and implemented using the cod provided by the authors. In addition, since we set a small validation set our experiments, the methods that need to be learned on the validation set are easy to over-fit. Therefore, in the following methods, we add regularization for the model param- eters in Eq 2 (4) Hu et al. Rs,, method that adds regularization to the validation learning in the method (Hu et al.). (5) Two-phase (Wahab et al.) 2017). learning method divided into two phases. first trains the model to learn a good classification representation on an imbalanced training set and then adjusts the imbalance bias of the model by learning on balanced validation set. When the model is trained on the validation set, we also add regularization.
Training and Evaluation. In experiments, the ALL-CNN-C model. In the following training, the text data is first converted into vector rep- resentations by BERT and then used to train the FCN model, and when training CIFARIO, the ALL-CNN-C model is initialized by the pre-trained model. Next, we apply different methods to train the models. We divide this training process into 2 stages, and we take an imbalanced training set and a small balanced validation set from the remaining training examples (not including the ex- amples used for model preparation). In Stage 1, we only use the training set to train the models, and the trained models can be regarded as the model initialization for subsequent training. For the method (Hu et al.), the trained model has basic classification capabilities, so that it can use stable gradient information to optimize the weights during weighting the examples (Ren et al. 2018). For the two-phhs phase method, Stage corresponds to its first learning phase. In Stage 2, we train the models according to their respective methods. Our method and the method (Hu et al.) will learn the model and example weight using the training set and validation set together. For the two-phase method, we only train the model on the balanced validation set, corresponding to its second phase. For Bascline and Proportion, the models still learn on the training set.
first fine-tune the BERT model and pre-train
'Code available http:///hhiiiccccaaaaaaaaaiaaiaa

---

Published as a conference paper at ICLR 2022
Table 3: Settings of the training process data sets
Data sets
Fine-tune / Pretrain
Stage 1
Stage 2
Adam(5e-6) epochs:5 batch size:8
Adam(le-2)
Adam( le-2)
SST-2 SST-5
epochs:15
epochs:10
batch size:50
batch size:50
Adam(1e-6)
Adam(le-3
follow the training (Springenberg et al. 2014
CIFARIO
epochs:200
epochs:10
batch size:
batch size:128
Adam(1e-4)
follow the training (pringeebbrg et al. 2014)
C0F
epochs:
batch size:128
Table 4: Results of six methods on four data sets
SST-2 100:1000
SST-5 50:500
CIFARIO 50:500
CIFARI00
Methods
40:400
Baseline
75.52
2.99
40.24 0.99
69.95 + 3.35
85.00 + 1.10
Proportion
79,59 + 3.35
42.59 + 1.12
79.58 + 0.34
85.20 + 1.21
Two-phase
81.99 0.80
42,60 + 1.44
79.63 + 0.44
86.00 + 1.61
Hu et al.'s
81.57 0.74
39.82 1.07
79.36 + 0.51
85.40 + 1.07
Hu 's+R
82.25 1.16
40.14 0.39
79.55
+ 0.21
86.50 +
Ours.
82.58 0.98
44.62 + 1.08
79.71 1 0.37
87.40 + 1.66
The settings of the training process on the four data sets are listed Table 3 Each cell in the table indicates the settings in the current stage, including the optimizer used, learning rate, number of epochs, and batch size, where the value in brackets is the learning rate. In text classification, we use Adam optimization. In image classification, al. to use SGD optimization, and then we use Adam to optimize in subsequent training.
first follow the implementation of Springenberg
Finally, we indicate the evaluation criteria and hyperparameters tuning. We the accuracy the full test set of each data set to evaluate the performance of models. During the training process, the final result may be over-fitting, so record the best step corresponding to the highest accuracy on the validation set. In addition, we also tune series of hyperparameters for different methods and report the best in the test set. For the method (Hu et al.), we follow (Hu al., 2019) and set the decay of weight to avoid exploding value. The decay value is selected from (1. 5, 10}. For Hu-regularization, we the learning rate for weight update, which is taken from {1, le-1, 1e-2, le-3). For our method, we set the learning rate and epochs for updating the weights during learning with constraint, and they are taken from { (le-2, 1e-3, le-4, le-5} and 11 5, 10, 15, 30) respectively. We adopt general regularization, namely Lp -n-norm (Bohra & Unser 2020). for the methods that need to trained on the validation set. The value of is selected from f2. 4. 6, 8). The log value of regularization coefficient is selected from {-4, .... 4] for text data sets and {-4, .... 9} for image data sets. All experiments were implemented with Python 3.8 and PyTorch 1.8 and were evaluated on Linux server with RTX 3080 GPU and 128GB RAM. All results averaged over 5 runs one standard deviation.
4.2 RESULTS ON DIFFERENT DATA SETS
We compare the performance of different methods on the four data sets. The four data sets involve text and image domains, as well as binary classification and multi-class classification scenarios. They can more comprehensively reflect the performance of our method. In this experiment, we set an imbalance ratio of 1:10, which is the ratio of the example size of the minority class to the majority class.
data sets, we set class 0 as the minority class, and the rest as the majority class. On the four data sets, the size of training examples is different. We set the number of training examples for each majority class of the data set SST-2, SST-5, CIFARIO. CIFARI00 to 1000, 500, 500, 400 respectively. In addition, for all data sets, the number of examples in the validation set is 10 for each class. The training set and validation set are randomly selected from the remaining training examples in each data set.

---

Published as a conference paper at ICLR 2022
Table 5: Results of different imbalance ratios SST-2 data
Methods
10:1000
20:1000
100:1000
Bascline
49.92 0.00
49,92 0.00
75.52 + 2.99
Proportion
60.63 + 13.13
78.76 + 2.40
79.59 = 3,35
Two-phase
75.35 + 8,90
80,52 + 1.96
81.99 + 0,80
Hu et al.'s
55.84 + 11.84
73.61 + 11.86
81.57 + 0.74
Hu al.'s+R
66.68 + 13.99
79.53 + 1.64
82.25
+ 1.16
Ours.
80.62 0.93
81.14 + 1.25
82.58 + 0.98
Table 6: Results of different imbalance ratios on CIFAR100 data set
Methods
4:400
8:400
40:400
Baseline
64.40 1 11.98
77.40 1: 12.23
85.00 1: 1.10
Proportion
60.50 1 8.40
69.60 1 8.56
85.20 1 1.21
Two-phase
66.00 + 13.95
79.50 3.75
86.00 + 1.61
Hu et al.'s
60.20 8.19
69.10 + 8.08
85.40 = 1.07
Hu et al. 's+RR
71.80 + 11.73
82,50 4.27
86,50 = 2.41
Ours.
77.20 + 3.75
82.60 + 3.87
87.40 = 1.66
Results The results on the four data sets are shown in Table We can see that our method has the best performance in these 4 data sets. Especially on the SST-5 data set, our method exceeds the second-best method by more than 2 accuracy points. demonstrates that our method can perform well in multiple domains and different classification scenarios. Hu et al.+ and Two-phase are competitive methods. On the SST-2 and CIFAR 100, Hu al.' is the second-best, and on the SST-5 and CIFARIO, Two-phase also reaches the second-best. It shows that using a balanced data set to simply adjust a biased model can also achieve good results. In addition, Hu et al." +R performs better than Hu al.'s. all data sets, and the CIFAR100, it surpasses the latter by accuracy point. It indicates that adding regularization to the validation learning can effectively improve the method (Hu et al.). However, on the SST-5, the method (Hu et al.) performs worse than the baseline, which may be due to the ineffectiveness of the approximation on SST-5. The accuracy of the proportion method lower than that of our method by more than 2 accuracy points on the SST-2, SST-5, and CIFARI00. It shows that the method of learning weight has more advantages than weighting empirically. The baseline method performs the worst on three data sets due the lack of measures to solve the imbalance.
than
4.3 RESULTS OF DIFFERENT IMBALANCE RATIOS
We study the performance our method with different imbalance ratios. In this experiment, we the SST-2 and CIFAR 100 data sets, and we vary the imbalance ratio from (1:10, 1:50, 1:100). The example size of majority classes in the training set and the validation set are consistent with the setting of Section
4.2 In addition, the training set and validation set also randomly constructed.
Results Table 5 and Table respectively shows the results of different imbalance ratios on SST-2 and C1FAR100 data set. The results are listed in the order of imbalance ratios of 1:10, 1:50, 1:100. There are three main observations. First, our method has achieved the highest accuracy rates in all imbalance ratio settings. It further demonstrates that our method different situations, such as slight imbalance, extreme imbalance, etc. Second, as the training data becomes more imbalanced, the performance of our method is more dominant than other methods. On the SST-2, the accuracy of our method exceeds the second-best method by about 0.3 at 100:1000 and more than accuracy points 10:1000. Similarly, on the CIFARI00, the accuracy of our method improves the second-best method over 0.9 at 100:1000 and more than 6 accuracy points at 10:1000. shows that our method more advantageous the imbalance ratios are 1:50 and 1:100, the accuracy rates of the proportion method are almost lower than other imbalance classification methods. On the CIFAR100, the proportion method even performs worse than the baseline. It indicates that the data imbalance becomes serious, the proportion method may not be effective. On the contrary, the advantage the methods of learning weights is obvious.
have excellent performance in
extreme imbalance. Third, when
8

---

Published as a conference paper at ICLR 2022
5 RELATED WORK
There have been very rich studies on weighting examples for imbalance classification, and these studies can be grouped into two categories, namely empirical weighting and automatic weighting.
Empirical Weighting. The empirical weighting methods assign the manual weight values to the ex- amples. Generally, the minority class example will be assigned larger weight value than that of the majority class, so as to relieve the bias of the model trained on the imbalanced data set. The methods of weighting by class are first proposed (King & Zeng 2001). In these methods, the examples of each class are manually set to the same value, such as inverse class frequency ( (Wang et al. 2017 Huang et al.) 2016) or inverse square root of class frequency (Mikolov et al. 2013] Mahajan et al., 2018). [Cu et al. (2019) proposed to calculate the effective number of examples as class frequency and then also use its inverse to weight examples and achieved significant improvements on long- tailed training data. In addition, many methods of weighting by example have also been proposed. Hard example mining (Shrivastava et al. 2016) thought focusing on the hard examples can improve the model on the imbalanced data. Dong et al. (017);; Malisiewicz et al. (2011) proposed to utilize the example loss to find hard examples and assign them higher weights. Lin et al. (2017) proposed to use the predicted probability to calculate higher weights for the hard examples and dynamically adjust the weight values during training. Empirical weighting is convenient implement and can achieve excellent performance, though it cannot adapt to different data sets and may cause poor performance. In addition, manually setting weights will also increase the engincering burden.
Automatic Weighting, The automatic weighting methods assign adaptive weights to the examples through learning mechanisms. Curriculum learning can provide an example weighting strategy for neural network models to learn on corrupted labels (Jiang et al. 2018 Wei et al. (02T), but the method focuses on examples that are easy to learn (Zhang et al. (020). On the contrary, imbalance learning prefers hard examples, so it is different from the methods of data weighting in imbalance classification. Ren et al. (2018) proposed to learn the example weights by a meta-learning paradigm (hang et al., 2021). l). This algorithm treats the example weights as meta- meta--ccarer and guides the learner to learn on the imbalanced training set. The loss on the balanced validation set is used as the meta-objective optimize the example weights (Bai al., (02T). In each iteration of updating the weights, this algorithm uses a gradient descent step to approximate the relationship between the weights and the learner. Hu al. (2019) improved the algorithm by iteratively optimizing weights instead of re-estimation at each iteration. Our work is based on the work of Ren and Hu et al. and make further research. There is a key difference between our work and theirs. We use the model optimization objective to derive the precise relationship between the model and weights, instead of the local approximation strategy they used. Therefore, our algorithm can accurately optimize the example weights and get better model for imbalance classification. The massive experimental results show that our algorithm makes significant improvements.
6 CONCLUSION
In this paper, based on the work of Ren and Hu et al., we further propose an improved algorithm to learn the example weights for imbalance classification. In this algorithm, we propose a learning mechanism that can accurately update the weights and the classification model under a constraint and improves the validation performance of the model. This is a key improvement compared to the method proposed by Ren et al. that uses the local approximation addition, the algorithm we proposed is a combination of our learning mechanism and the method proposed by Hu et al., which can promote each other and make the model perform better. Finally, the experimental evaluation shows that our algorithm can achieve significant improvement compared with the SOTA method in data weighting and other imbalance methods. In our future work, we plan to extend our algorithm and explore the performance of our algorithm in data augmentation.
optimize the weights. In
Acknowledgements. This research was partially sponsored by the following funds: National Key R&D Program of China (2018YFB 1402800), Key Research Project of Zhejiang Province 2022C00145)) Fundamental Research Funds for the Provincial Universities of Zhejiang (RF- A2020007) and Zhejiang Lab A3AB0.)

---

Published as a conference paper at ICLR 2022
REFERENCES
Yu Bai, Minshuo Chen, Pan Zhou, Tuo Zhao, Jason Lee, Sham Kakade, Huan Wang, and Caiming Xiong. How important is the train-validation split in meta-learning? In International Conference on Machine Learning, pp. 543-553. PMLR, 2021.
CM Bishop. Pattern recognition and machine learning, m. jordan, j. kleinberg, and schölkopf, eds, 2006.
Pakshal Bohra and Michael Unser. Continuous-domain signal reconstruction using {p} -oomm reg- ularization. JEEE Transactions on Signal Processing, 68:4543-44 -45444 2020.
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 111999669999 9977, 2019.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: bidirectional transformers for language understanding. arXiv preprint /rXiv:/8/// 1810004805 2018.
Pre-training of deep
Qi Dong, Shaogang Gong, and Xiatian Zhu. Class rectification hard mining for imbalanced deep learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1851- 1860, 2017.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL http://///archi
ics. edd/mm
Mila Grancharova, Hanna Berg, and Hercules Dalianis. Improving named entity recognition and classification in class imbalanced swedish electronic patient records through resampling. In Eighth Swedish Language Technology Conference (SLTC). Förlag Göteborgs Universitet, 2020.
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: survey. arXiv preprint arXiv:2004. 05439, 2020.
Zhiting Hu, Bowen Tan, Ruslan Salakhutdinov, Tom Mitchell, and Eric P. Xing. Learning Data Manipulation for Augmentation and Weighting. Curran Associates Inc., Red Hook, NY, USA, 2019.
Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang. Learning deep representation for imbalanced classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5375-5384, 2016.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Fei-Fei. Mentoret: Learning data- driven curriculum for very deep neural networks corrupted labels. In International Conference on Machine Learning, pp. 2304-2313. PMLR, 2018.
Gary King and Langche Zeng. Logistic regression in rare events data. Political analysis, 9(2): 137-163, 2001.
Chaoliang Li and Shigang Liu. A comparative study of the class imbalance problem in twitter spam detection. Concurrency and Computation: Pructice and Experience, 30(5)e 2018.
Ysung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 2980-2988, 2017.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. In Proceedings of the European conference on computer vision (ECCV), pp. 181- 196. 2018
Tomasz Malisiewicz, Abhinav Gupta, and Alexei A Efros. Ensemble of exemplar-svms for object detection and beyond. In 201 International conference on computer vision, pp. 89-96. IEEE, 2011.
10

---

Published as a conference paper at ICLR 2022
Tomas Mikolov, llya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed represen- tations words and phrases and their compositionality. In Advances neural information pro- cessing systems, pp. 3111-3119, 2013.
Kemal Oksuz, Baris Can Cam, Sinan Kalkan, and Emre Akbas. Imbalance problems in object detection: A review. IEEE transactions on pattern analysis and machine intelligence, 2020.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In International Conference on Machine Learning, pp. 4334-4343, PMLR, 2018.
Frank Schneider, Lukas Balles, and Philipp Hennig. Deepobs: A deep learning optimizer benchmark suite. arXiv preprint orXiv:/903. (90.0099999 2019.
Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick. Training region- based object detectors with online hard example mining. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 761-769, 2016.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language pro- cessing, pp. 1631-1642, 2013.
2013.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv: 1412.64412.6806, 2014.
Noorul Wahab, Asifullah Khan, and Yeon Soo Lee. Two-phase deep convolutional neural network for reducing class skewness in histopathological images based breast cancer detection. Computers in biology and medicine, 85:86-91 97, 2017.
Yu-X Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. Learning to model the tail. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 7032-7042, 2017.
Jerry Wei, Arief Suriawinata, Bing Ren, Xiaoying Liu, Mikhail Lisovsky, Louis Vaickus, Charles Brown, Michael Baker, Mustafa Nasir-Moin, Naofumi Tomita, et al. Learn like a pathologist: curriculum learning by annotator agreement for histopathology image classification. In Proceed- ings of the IEFF/CVF Winter Conference on Applications of Computer Vision, pp. 2473-2483, 2021.
Gang Wu and Edward Y Chang. Class-boundanl alignment for imbalanced dataset learning. In ICML. 2003 workshop on learning from imbalanced data sets II, Washington, DCC, pp. 49-56. Citeseer, 2003.
Han Xiao. bert-as-service. httppp///gtttt.....///aaaa/////a
2018.
Lei Zhang. Yuehuan Wang, and Yang Huo. Object detection in high-resolution remote sensing images based on a tett--exmmllmmiinin network. IEEE Transactions on Geoscience and Remote Sensing, 2020.
Shen Zhang, Fei Ye, Bingnan Wang, and Thomas Habetler. Few-shot bearing fault diagnosis based on model-agnostic meta-learning. IEEE Transactions on Industry Applications, 2021.
A APPENDIX
A.1 EXPERIMENTAL RESULTS WITH DIFFERENT METRICS
In this section, we evaluate the performance of our methods with different metrics a large-scale data set. In addition to the accuracy, we also adopt the Marco-Fl score and G-means that are often used in imbalanced classification. In this experiment, data set that the operational data for Air Pressure System (APS) Failure at Scania trucks. The settings and results of this experiment are as follows.
use additional large-scale unbalanced

---

Published as a conference paper at ICLR 2022
Table 7: The network model for APS Failure dataset
Table 8: Description of APS Failure dataset
Input 171-dimensional examples
Classes
Train Set
Validation
Test
Set
Set
(171, 100) linear layer, ReLU (100, 100) linear layer, Tanh
Negative
10000
1000
(100, 2) linear layer
Positive
10
1000
Table 9: Results with different metrics on APS Failure dataset
Methods
Accuracy
Marco-F1 score
G-means
Baseline
50.00 + 0.00
33.33 + 0.00
0.00 + 0.00
Proportion
93.29 1 2.59
93.27 + 2.63
93.17 2.78
Two-phase
93.19 1 0.87
93.19 0.87
93.15 0.89
Hu al.'s
94.09 1 0.94
93.35 1 1.73
93.28 1.79
Hu et R
93.98 + 1.78
94.65 = 0.76
94.65 = 0.76
Ours.
94.49 = 0.70
95.10 = 0.66
94.86 0.58
Dataset and Models. APS Failure dataset is from UCI Machine Learning Repository (Dua & Graff 2017). This dataset contains 76000 examples. We randomly construct the train, validation and test data sets from the original data set. These sub-data sets are described in Table 8] The train set consists of 10000 negative examples and 10 positive examples, which is extremely imbalanced, and its imbalance rate reaches 1:1000. The validation and test sets are balanced data sets, and their example sizes for each class are 5 and 1000 respectively. In addition, APS Failure dataset dataset has 171 features, and replace the missing feature values with the average of the examples in the same class. Finally, we scale the all feature values to [0, 1]. The model for APS Failure dataset is shown in Table 7 and simple 3-layer FCN.
Other Settings. During training, we use Adam optimization with an initial learning rate of 1e-2 and set the batch size to 128. Other hyperparameter settings are the same as the text classification in Section 4.1
Results. The results of different metrics on APS Failure dataset are shown in Table g There are three main observations. First, method achieves the best score in all metrics. It indicates that our method has comprehensive advantages compared with other methods. Second, the score rankings of the six methods are almost consistent among these metrics. It shows that the evaluation of these metrics on a balanced test set similar. Third, our method performs well on a large-scale data set. It demonstrates that our method is also effective large-scale data.
A.2 CONVERGENCE PROOF OF OUR METHOD
This section firstly provides a proof of the convergence of the learning method with constraint in Section 3.1 and then we prove the convergence of the combination method in Section 3.2 Definition 1. function (x) Rd
said be Lipschitz- with constant I. if
VV(x)) - vS(v) | L x - ll. Vx, E Rd
Definition 2. A function f(x) has a-bounded gradients if
ff(x) | o, Vx E Rd
Theorem 1. Suppose the validation loss function Lwat is Lipschitz-smonn with constant L, and the training loss function fi corresponding the example x; has oboundedd gradients and the Hessian matrix H, namely, the second derivative of Lirašn with respect to 0, is bounded by Let the learning rate The satisfies The S LN Then, after each iteration of the model parameter 0, the validation loss always decreases. More specifically.
Loat(0tt1) S (0,)
(10)
12

---

Published as a conference paper at ICLR 2022
Proof.
The validation loss function Leat is Lipschitz-smooth, we have
I. 2
(0al(0(+1) (O.) + (VoLvat) 0
A0
(11)
Let (VoLca)) + 211 A |]. We can see that only v S 0, there is (oai(0(+1) I (O).
Then, substituting the 0 update formula: AO vn. (We+1 wx) and the w update formula:
W++1 W/ = (V. 0) VaLval from Section 3.1 into v, have
= (VoLcal) V (Wi+1 wx) + 2 Then Vu (w++1 - wx)
(12)
Li
2
( V (V VoCral
Vud(V v')) VoLval
(13)
2
= (VeCval) VV L
2
v(V Tho VoLral
(14)
2
where / is the identity matrix.
Lod - Vul(())) T/n,, so (VoLvat) T VoLval. Next, prove Let S = Vwo(V 1)
that S is a semi-negative definite matrix such that S 0.
We observe that in S, the term V (V))' is a symmetric and positive semi-definite matrix. We use A denote this term and define its eigendecomposition as (rdiag(A)P-) where À is
vector composed of eigenvalues and A 2 0 for all i. Substituting this eigendecomposition into S,
we have
S Pdiag (A) P)P-))PPP----) Liker Pdiag(A)PP11 - Nno,
(15)
Pdiag(
x A Tiu A)P-1
(16)
2
where represents the hadamard product.
En 2
Therefore, in order make S semi-negative definite matrix, Tho must satisfies 2
X Thu, A S
0 for all i, namely, S Ve. I Z2, Let Amax = max;(A)). so finally The, must satisfies
2
0 0 v/u Ldmar
(17)
Further, we estimate the boundary of the scalar value Amez- In Section 2
we define matrix
= fi,.., VfN), where fi is the training loss function and N the number of the training examples. Since fi has o-bounded gradients, 7 into A, we have
obtain F I No. Substituting v in Eq.
A - V (V |||| diag(h)F diag(h) di)ag())))
(18)
S| diag(h) IIII F III| F" IIIl diag(h) ||= N2021²
(19)
Since Amaz is the eigenvalue of A, we can obtain
Amaz <| A A 2
(20)
Therefore, combining Eq. 17 and Eq. 20 the satisfying range of v/a is
2
0 E vla,
(21)
This finishes proof for Theorem
13

---

Published as a conference paper at ICLR 2022
Theorem 2. Suppose the validation loss function Lvai, the training loss function f. and the learning rate The satisfies Theorem ! the algorithm successively uses Hu et al. and our methods to update 8 and let t' represent the time step inside time-step t and the algorithm only apply our method to update The range of t' is [0, 10,...., T2 11. Then the validation loss always decreases after the t-th iteration, namely,
conditions. Same as Algorithum [ is denoted as the time step where
(0a1()++1) S (0.)
(22)
Proof. The Or the updated parameter through the method (Hu et al.). According to convergence theorem in the paper (Ren et al. 2018 Hu et al. 2019), can obtain
Lmaa(O)) (0,) 2 (0v=(Or=o)
(23)
After that, we use our method to update the parameter T times. According to the Theorem have
I we
(vr=() 2 (vai(Or=1).. 2 (eat(Ov=T(-1)
(24)
Here, the algorithm completes the update of the parameter in time-step i, namely, 0 0s+1- Combining Eq. 23 and Eq. 24] we can obtain
Levat S (0.)
(25)
This finishes our proof of Theorem
A.3 CONVERGENCE RATE OF OUR METHOD
This section provides a proof of the convergence rate of the learning method with constraint in Section 3.1
Theorem 3. Suppose the validation loss function fovat, the training loss function Ji and the learn-
ing rate n'u, satisfies Theorem 7 conditions. Then the learning method in Section 3.1 achieves || Vo, Loat ||S e in O(2) steps, namely,
C
min oct-
Vo, Lvat 1|S
(26)
where C' is some constant
Proof. According to Eq.
and Eq. 14 in Theorem we can obtain
Leat (6 - Lvat(f.) S (V,, Gval) S. V.Vo Lval
(27)
where S, is the matrix S at time-step and is defined in Theorem |
Then have
(Vo, Cual) V,V Leat((rrt) - (ve)
t=0
Loai Lovi(0o)
(28)
where (fa)(0") is the minimum of function (f). Then, observe that there exist time- step 0 < r < such that,
T(V Lrat)" "'
2 Loat(O") Loat S, Vo, Loat
(ov)(
We have proved that S, is semi-negative definite matrix. According to Eq. 16 we have
S, Pdi PdiaggX)) (X) pT
(30)
14

---

Published as a conference paper at ICLR 2022
where X' is vector composed of eigenvalues and X I 0 for all i. Pis the matrix composed of eigenvectors of A in Theorem and P-1 pf because A is symmetric matrix. Substituting Eq.
30 into Eq. 29 we have
Lual (0") Loo) S T( T(Vo, T Pdiag(X) p2 Vor Leal
(31)
= =((P" Vo, Coal) diag(X) ()) "'''P''VVo Vo,
(32)
- T.P"VVV. Ve. Xmax IPT V Lva
(33)
= =TXoax || Vo, fivol
(34)
We can regard Eq. 32 as the quadratic form the diagonal matrix diag(X), and we scale all eigenvalues X to Xmax where Xmax max; X, so that we obtain the inequality in Eq. 33
So we have
Lvat Lvai(Oo) C
Vo, Lvat I|
(35)
T
Xmaz
T
where (oui(0") (ov is a constant independent of T.
X
Therefore, we can obtain min 0<T<
Vo,Cval ||S|| Ve.L ||s G. It means that our method can
achieve min || Vo, Leat || ( in T steps. This finishes our proof of Theorem O<t<T
15


Reviews:
[{"review_id": "pma1_TOL0Bc", "reviewer": "Reviewer_fpgG", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong points:\nThe problem is very important. The solution is novel from theoretical perspective, and it is evident in the experiments as well. \nThe competing methods does not only contain previous published work but also include its variants to improve the performance, which make the proposed method further attractive.\n\nThe paper is well written. Both the proposed method and experiments are clearly explained. \n\nWeak points:\nAccording to Algorithm 1, the proposed method needs O(T^2) to converge, where T is the number of iterations. However, if I understand correctly, the previous method requires O(T) for convergence. Will the proposed method need much more time to converge in practice?\n\nIn Eq.(7), the authors uses diagonal approximation to replace the inverse of Hessian. Would other methods, e.g., Quasi-Newton’s method, give a more accurate approximation without too much cost? Please comment on this. \n\nIt might be good to include the cost of such approximation in the future work when using a small model. The proposed method may also beneficial other shallow models as well.\n", "summary_of_the_paper": "This paper proposes a new approach to learn sample weights aiming to solve imbalance classification problem. The problem is challenging because the learnable model parameter and sample weights are coupled and cannot be directly optimized together. The previous method learns the parameter and weights in an alternative fashion, which solves the problem approximately. The authors argues that there exists a constraint and allow both parameters to be learned together thus this paper proposed a new algorithm combined with previous method to learn the sample weights. Experimental results shows that the proposed method outperforms other competing method especially for the extreme imbalance cases (1:100).", "summary_of_the_review": "As many real-world datasets naturally forms in an imbalanced form, the solution provided in this paper motivated from theoretical side and is strongly proved empirically by comparing with other methods on different datasets. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "gJX_bv6_tpF", "reviewer": "Reviewer_XCGn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n-Good review of existing work on the topic\n-Formulation of the proposed approach is well motivated\n-Extensive experimental analysis\nWeaknesses:\n-Some concepts in section 2 are not well defined and are used in non-standard ways: for example, what does it mean for the parameters of a model to be biased? What does it mean for a validation dataset to be unbiased? \n-The novelty and significance are limited, since the proposed method is similar to the one proposed by Hu et al and the experimental results in most cases show no statistically significant difference in the results (although for one case the difference is very large).\n-In the title and abstract they refer to \"Imbalance classification\" but it should be \"Imbalanced classification\"\n \n", "summary_of_the_paper": "The paper presents a new method for learning example weights together with the parameters of a deep neural network. The difference between the proposed method and previous work is that they use a constraint to tie together the values of the parameters and the weights as they do the joint optimization through gradient descent. They do extensive experiments using both text and image datasets, with different imbalance ratios and show that the proposed method outperforms the state-of-the-art in terms of accuracy on a (balanced) test set.", "summary_of_the_review": "The proposed method is well-motivated: it is a small modification to recent related work that seems to be justified mathematically by the constraint argument they give for tying together the optimization of the weights and parameters of the network. The paper is generally well-written but I missed a better setting of the problem in section 2 -- which I find confusing -- especially in the use of the \"biased\"/\"unbiased\" terms when referring to parameters and datasets. The experimental section describes the experiments with enough details that they can be reproduced, the experiments are well structured and the accuracy results show a small but consistent improvement in most experiments over the state-of-the-art. Therefore, I believe the contribution of this paper is just enough to warrant acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RqAEXE_TgAq", "reviewer": "Reviewer_yKjB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed method provides some useful results, but there are some questions needed to be answered.\n1.  For the learned θ in Eq. (1), why it is an unbiased θ, and you need to give more details.\n2.  Eq. (6) is used as a functional relationship between θ and w, but you tell us little about how to perform this constraint in the final objective function and optimized process because this is important for your model.\n3.  In Section 3.2, you combine your model with another algorithm in section 2, which can guarantee that θ is optimal for Eq.1 and Eq.2. Intuitively, this model has significant improvement in varieties of settings, but whether you give a simple mathematical proof. \n4. In your experimental settings, note that you use a small balanced validation set, why? Since your setting is designed for the imbalanced classification, and whether you need to utilize an unbalanced validation set.\n5.  Since the four datasets used in this paper essentially include two kinds of datasets, thus you need to provide more results for different types of datasets.\n6.  There are various metrics (e.g., Acc, Recall, F_score, G-mean) for evaluating the imbalanced classification, however, you only use Acc in your paper. This is not comprehensive to show the advantages of the proposed method in the imbalanced classification. \n7.\tSeveral mistakes need to be corrected. For example, the model parameters θ obtained by this method is biased. “parameters” should be “parameter”. Therefore, I suggest that you should check your paper carefully to avoid similar mistakes. \n", "summary_of_the_paper": "They propose a novel mechanism of learning with a constraint, which can accurately train the weights and model. Then, they propose a combined method of our learning mechanism and the work by Hu et al., which can promote each other to perform better. ", "summary_of_the_review": "In a word, the idea of the proposed model is useful to obtain a better performance, however, several detail issues are not clear in the proposed model, and the experimental results are not sufficient to verify the effectiveness of the proposed method.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "93p6WxwXEb", "reviewer": "Reviewer_auNr", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors start by explaining the process of learning from a class imbalance dataset. In such a scenario, different weights are given to the different training examples. The model objective is to identify the optimal model that minimizes the example-weighted loss of the training set. The meta objective is to give the best weight to the training example so that the model minimizes the loss of the balanced validation set.\n\nThe authors first propose a method to learn theta and w with a constraint, which can accurately optimize both parameters. Next they combine this with a method previously proposed to boost the performance further. The authors start by noting that the optimal model parameter theta* and w satisfy the constraint F(theta*)w = 0. This constraint gives a precise relationship between these two parameters. From this equation they identify the gradient to be related to the inverse of the Hessian of F(theta). This helps them derive the iterative update rules for both theta and w which satisfy the constraint. \n\nHowever, this solution only satisfies the meta objective as described before. This is not optimal for the model objective. As such, they combine their approach with the one proposed by Hu et al., to get the best results.\n\nThe authors then show the efficacy of the proposed method under different experimental setups. This shows that the proposed combined approach can be very effective under different imbalance ratios.\n\n", "summary_of_the_paper": "The authors propose an approach to tackle the class imbalance problem present widely in the machine learning domain. For this purpose, they first propose a mechanism to precisely learn the relationship between the weights and the trained model in the model objective. This allows the weights and models to be optimized more accurately. They then combine this process with the mechanism proposed by Hu et al., which helps the model learn the model objective better. Finally, they show the efficacy of their proposed method through experiments.", "summary_of_the_review": "The paper is relatively smooth to read. However, the descriptions can sometimes be very verbose (e.g., introduction) and thus there is some room for improvement there. Overall the approach seems justified and the results are promising. However, the improvements can be minimal in some cases from the baselines as shown by the experimental results. Another potential issue can be the low volume of training data. I would have liked to see more larger datasets being used here as in practice much bigger datasets are the use cases where the class imbalance problem pops up.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: To solve imbalance classification problem, this paper proposes a method to learn example weights together with the parameters of a neural network. The authors proposed a novel mechanism of learning with a constraint, which allows accurate training of the weights and model at the same time. Then they combined this new learning mechanism and the method by Hu et al. (2019), and demonstrated its usefulness in extensive experiments.

I would like to thank the authors for their detailed feedback to the initial reviews, which clarified most of the unclear points in the manuscript. Overall the paper is well written and the effectiveness was demonstrated in experiments. Since the contribution is valuable to ICLR2022, I suggest its acceptance.
Decision: Accept (Poster)