Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a new method for learning example weights together with the parameters of a deep neural network. The difference between the proposed method and previous work is that they use a constraint to tie together the values of the parameters and the weights as they do the joint optimization through gradient descent. They do extensive experiments using both text and image datasets, with different imbalance ratios and show that the proposed method outperforms the state-of-the-art in terms of accuracy on a (balanced) test set.
main_review: Strengths:
-Good review of existing work on the topic
-Formulation of the proposed approach is well motivated
-Extensive experimental analysis
Weaknesses:
-Some concepts in section 2 are not well defined and are used in non-standard ways: for example, what does it mean for the parameters of a model to be biased? What does it mean for a validation dataset to be unbiased? 
-The novelty and significance are limited, since the proposed method is similar to the one proposed by Hu et al and the experimental results in most cases show no statistically significant difference in the results (although for one case the difference is very large).
-In the title and abstract they refer to "Imbalance classification" but it should be "Imbalanced classification"
 

summary_of_the_review: The proposed method is well-motivated: it is a small modification to recent related work that seems to be justified mathematically by the constraint argument they give for tying together the optimization of the weights and parameters of the network. The paper is generally well-written but I missed a better setting of the problem in section 2 -- which I find confusing -- especially in the use of the "biased"/"unbiased" terms when referring to parameters and datasets. The experimental section describes the experiments with enough details that they can be reproduced, the experiments are well structured and the accuracy results show a small but consistent improvement in most experiments over the state-of-the-art. Therefore, I believe the contribution of this paper is just enough to warrant acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: They propose a novel mechanism of learning with a constraint, which can accurately train the weights and model. Then, they propose a combined method of our learning mechanism and the work by Hu et al., which can promote each other to perform better. 
main_review: The proposed method provides some useful results, but there are some questions needed to be answered.
1.  For the learned θ in Eq. (1), why it is an unbiased θ, and you need to give more details.
2.  Eq. (6) is used as a functional relationship between θ and w, but you tell us little about how to perform this constraint in the final objective function and optimized process because this is important for your model.
3.  In Section 3.2, you combine your model with another algorithm in section 2, which can guarantee that θ is optimal for Eq.1 and Eq.2. Intuitively, this model has significant improvement in varieties of settings, but whether you give a simple mathematical proof. 
4. In your experimental settings, note that you use a small balanced validation set, why? Since your setting is designed for the imbalanced classification, and whether you need to utilize an unbalanced validation set.
5.  Since the four datasets used in this paper essentially include two kinds of datasets, thus you need to provide more results for different types of datasets.
6.  There are various metrics (e.g., Acc, Recall, F_score, G-mean) for evaluating the imbalanced classification, however, you only use Acc in your paper. This is not comprehensive to show the advantages of the proposed method in the imbalanced classification. 
7.	Several mistakes need to be corrected. For example, the model parameters θ obtained by this method is biased. “parameters” should be “parameter”. Therefore, I suggest that you should check your paper carefully to avoid similar mistakes. 

summary_of_the_review: In a word, the idea of the proposed model is useful to obtain a better performance, however, several detail issues are not clear in the proposed model, and the experimental results are not sufficient to verify the effectiveness of the proposed method.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new approach to learn sample weights aiming to solve imbalance classification problem. The problem is challenging because the learnable model parameter and sample weights are coupled and cannot be directly optimized together. The previous method learns the parameter and weights in an alternative fashion, which solves the problem approximately. The authors argues that there exists a constraint and allow both parameters to be learned together thus this paper proposed a new algorithm combined with previous method to learn the sample weights. Experimental results shows that the proposed method outperforms other competing method especially for the extreme imbalance cases (1:100).
main_review: Strong points:
The problem is very important. The solution is novel from theoretical perspective, and it is evident in the experiments as well. 
The competing methods does not only contain previous published work but also include its variants to improve the performance, which make the proposed method further attractive.

The paper is well written. Both the proposed method and experiments are clearly explained. 

Weak points:
According to Algorithm 1, the proposed method needs O(T^2) to converge, where T is the number of iterations. However, if I understand correctly, the previous method requires O(T) for convergence. Will the proposed method need much more time to converge in practice?

In Eq.(7), the authors uses diagonal approximation to replace the inverse of Hessian. Would other methods, e.g., Quasi-Newton’s method, give a more accurate approximation without too much cost? Please comment on this. 

It might be good to include the cost of such approximation in the future work when using a small model. The proposed method may also beneficial other shallow models as well.

summary_of_the_review: As many real-world datasets naturally forms in an imbalanced form, the solution provided in this paper motivated from theoretical side and is strongly proved empirically by comparing with other methods on different datasets.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose an approach to tackle the class imbalance problem present widely in the machine learning domain. For this purpose, they first propose a mechanism to precisely learn the relationship between the weights and the trained model in the model objective. This allows the weights and models to be optimized more accurately. They then combine this process with the mechanism proposed by Hu et al., which helps the model learn the model objective better. Finally, they show the efficacy of their proposed method through experiments.
main_review: The authors start by explaining the process of learning from a class imbalance dataset. In such a scenario, different weights are given to the different training examples. The model objective is to identify the optimal model that minimizes the example-weighted loss of the training set. The meta objective is to give the best weight to the training example so that the model minimizes the loss of the balanced validation set.

The authors first propose a method to learn theta and w with a constraint, which can accurately optimize both parameters. Next they combine this with a method previously proposed to boost the performance further. The authors start by noting that the optimal model parameter theta* and w satisfy the constraint F(theta*)w = 0. This constraint gives a precise relationship between these two parameters. From this equation they identify the gradient to be related to the inverse of the Hessian of F(theta). This helps them derive the iterative update rules for both theta and w which satisfy the constraint. 

However, this solution only satisfies the meta objective as described before. This is not optimal for the model objective. As such, they combine their approach with the one proposed by Hu et al., to get the best results.

The authors then show the efficacy of the proposed method under different experimental setups. This shows that the proposed combined approach can be very effective under different imbalance ratios.


summary_of_the_review: The paper is relatively smooth to read. However, the descriptions can sometimes be very verbose (e.g., introduction) and thus there is some room for improvement there. Overall the approach seems justified and the results are promising. However, the improvements can be minimal in some cases from the baselines as shown by the experimental results. Another potential issue can be the low volume of training data. I would have liked to see more larger datasets being used here as in practice much bigger datasets are the use cases where the class imbalance problem pops up.
review_title: null

Meta-review: To solve imbalance classification problem, this paper proposes a method to learn example weights together with the parameters of a neural network. The authors proposed a novel mechanism of learning with a constraint, which allows accurate training of the weights and model at the same time. Then they combined this new learning mechanism and the method by Hu et al. (2019), and demonstrated its usefulness in extensive experiments.

I would like to thank the authors for their detailed feedback to the initial reviews, which clarified most of the unclear points in the manuscript. Overall the paper is well written and the effectiveness was demonstrated in experiments. Since the contribution is valuable to ICLR2022, I suggest its acceptance.
Decision: Accept (Poster)