Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes and empirically validates a way to use - and tune - high-dimensional hyperarameters affecting training loss, which are used for fairness-seeking test objectives.
main_review: To the best of my knowledge, the proposed method is novel. My experience is primarily in high-dimensional bilevel optimization, and the related work looks adequately cited there.  I am less able to comment on related work long-tailed learning or group-sensitive and fair learning.

The paper's bilevel optimization framework seems technically sound.  The gradient estimates seem reasonable, and I’m particularly impressed by the authors results with the final re-training phase with fixed optimal hyperparameters.  

A contribution the authors did not emphasize, but I think is present is the following. It is common to neglect this training phase, and many models fail to re-train to reasonable performance with the final hyperparameters due to hysteric optimization effects.  The hysteric effects are particularly strong when tuning the loss.  However, the authors circumvent this with their loss parameterization. It would be interesting to try to generalize these techniques to more general setups than fairness, or understand this phenomena more abstractly. For example, by building on Lemma 2.

I think the paper suffers from various issues with clarity.

The text from 207-219 is unclear and the notation seems inconsistent.  Is \theta^* a function or a point?  Where are derivatives evaluated at?  What do you mean by “Also, observe …. Computed by taking the gradient”?

You discuss the importance of a train-val split in 220-233, but I do not see a discussion of how you tuned the split for this problem?  You also discuss the effects of overparameterization, but its not clear to me what kind of sizes of alpha, theta, train size, val size, etc. are actually used in practice.

I find it difficult to read objects like table 1, table 4 due to the dense notations.  I would appreciate a notation table summarizing it. 

The paragraph from line 39-58 is long and dense.  Perhaps the ideas should be modularized.  The results and discussion from 273-299 is similar.

Many components of line 252-267 could probably be moved to the appendix.

I believe the results are reasonably significant because they propose a useful way to introduce high-dimensional hyperparameters controlling the loss that still work even after retraining.  The results seem useful for fairness seeking objectives too. Practitioners are likely to use the method, because it is simple to understand, does not look difficult to implement, does not introduce excessive computational burden, and seems to work reasonably well. The paper also claims to advance the state-of-the-art, which seems supported by the experiments to the best of my knowledge.

My score is 7, because I think the paper makes good contributions to learning losses with fairness seeking objectives. My confidence is a 3, because I am not an expert on fairness seeking objectives. The score would be improved to an 8 if the authors largely improved the clarity.

limitations_and_societal_impact: The authors have adequately addressed the potential negative societal impacts of their work.

The authors present reasonable limitations.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper proposes a method for long-tailed classification. The key idea is to use a generalized loss function parameterized by hyper-parameters \alpha and to (automatically) tune \alpha on a validation set. Results are presented on CIFAR-10-LT, CIFAR-100-LT, ImageNet-LT and iNaturalist datasets.
main_review: Strengths
-------------
(1) Experimental results are good.
(2) The paper is written well and the key technical details are sufficiently described.

Weaknesses
-----------------
(1) Unfortunately, there is limited scientific novelty in this work in my opinion. The main loss function is a straightforward generalization of the loss functions developed in previous works such as [8], [50] and others. The authors then optimize the hyper-parameters of this function on a validation dataset using the method proposed in [43]. Therefore, the overall method proposed in this paper is a highly-engineered solution with limited scientific novelty/interest.

(2) The idea of fitting hyper-parameters to a validation set depends on the premise that (as stated in line 62): "unlike training data, the validation data is difficult to fit and will provide a consistent estimator of the test objective". However, this assumption is flawed. With millions of parameters in the actual model and a handful of hyper-parameters, it is easy to overfit to the validation set as well, specially when the validation set is small. For example, in CIFAR-100-LT, the validation set authors use has only 1 example for the tail-most class (= 500 / 100 * 20%). It is not reasonable to assume that it is difficult to overfit to such a small dataset when automatically optimizing over hyper-parameters using a gradient descent procedure.

(3) Since the loss function itself (Eq. 4.2) is not novel, one could argue that the key contribution of this paper is automatically tuning hyper-parameters on the validation set. But then the method should be compared to other automatic hyper-parameter tuning methods such as Bayesian Optimization methods.
limitations_and_societal_impact: Authors have adequately addressed the limitations and potential negative societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper demonstrates how to jointly optimize parametrized cross entropy and model parameters and shows improvement on datasets with class imbalance.
main_review: The central theme of paper is to optimize the model parameters on training data and parametrized cross entropy on validation set. The model itself can be optimized using simple gradient based approaches, more sophisticated approaches from literature are used for computing parametrized cross entropy on the validation set. Additionally data augmentation for minority samples is proposed to increase classifier margin from low population samples. Authors perform extensive experimentation on the cifar, imagenet and inaturalist imbalanced datasets.
limitations_and_societal_impact: 1. Paper contributions state that it "automatically designs a training loss function". However the actual training loss is cross entropy only with multiplicative and additive parameters. Since the actual loss is a variation of bias+weighted cross entropy this needs to be mentioned in both abstract and contributions in clear terms. There are other works out there which use different kind of loss functions to optimize for class imbalance [1]- they should be cited.
2. The papers is written in a complicated way and is difficult to read and understand. Simple statements have been made into complex math jargon, and a lot of places terms are first used in equation and then defined later.
3. Imbalanced data of only images is considered - other modalities should be considered
4. There are many small and large datasets which are not modified to create LT datasets - for example cubs and flowers [1]. The paper would greatly benefit by analyzing performance on such datasets. 
5. "Hyper" gradient computation on L_up is critical to implementation of paper - this needs to appear in main text not just supplementary. Its also not completely clear how to implement it.
6. Paper uses data augmentation (previously proposed epsilon ball perturbations - applied to low population classes). Upon first inspection the data augmentation seems to create a bigger impact than the primary algorithm. This needs to be highlighted properly.


[1] Dubey, A., Gupta, O., Guo, P., Raskar, R., Farrell, R., & Naik, N. (2018). Pairwise confusion for fine-grained visual classification. In Proceedings of the European conference on computer vision (ECCV) (pp. 70-86).
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work introduces a novel algorithm (AutoBalance) to address the imbalance and group sensitive classification problems and maximise the fairness-seeking objectives. The algorithm uses a bi-level optimization framework to optimize the model weights over the training data while automatically fine-tuning the loss function by monitoring the validation data. In addition, AutoBalance uses data augmentation policies personalised to subpopulations in the data.
main_review: \+ means strength, \- means weakness

**Originality**

\+ *The paper presents a novel algorithm.* The algorithms applies ideas from bilevel optimization [43, 46, 55, 26, 45] to fairness and class imbalance learning. Unlike previous work [41, 33, 16, 32, 8, 13, 50, 63], which employs fixed class balancing loss functions based on some training data statistics, the proposed method automatically guides the loss function design and learns a more optimal set of parameters for the balanced loss.  

**Quality**

\+ *Claims are well supported through experimental evidence.* 

\+ *The authors are careful and honest about reporting the paper's limitations.*

\+ *The paper compares its method to recent state-of-the-art [8,50] and beats them.* 

\- *Lack of more baselines.* The paper compares performance to only two baselines [8,50], and it is not clear to me why more baselines are not evaluated (e.g. [41, 33, 16, 32, 13, 63]).

\- *No error bars in the results.* There are no error bars in the result tables, making it more challenging to assess the significance of the improvements.

**Clarity**

\+ *The paper is written very well, and it is easy to follow.* On the most part, the paper adequately informs the reader. 

\- *Some clarifications about the experiments are needed.* Table 1 does not have any $PDA$ component in Algo. 1, does this mean that no $PDA$ was applied in those experiments? (if $PDA$ was used, did the authors also apply it to the baselines?) Similarly, did experiments in Table 2 apply $LA\~\text{init}$? And lastly, does Table 3 show results for "Algo 1. $ \alpha \leftarrow \Delta \\& l, LA~\text{init}, PDA$"? 

\- *A possible inconsistency in the results* but I may also be misunderstanding something (see above for my confusion). Specifically, if I am correct in understanding, the diagonal results in Table 3 show the performance for "Algo 1. $ \alpha \leftarrow \Delta \\& l, LA\~\text{init}, PDA$" which have a higher error rate than both "Algo 1. $ \alpha \leftarrow PDA, \Delta \\& l $" in Table 2 and "Algo 1. $ \alpha \leftarrow \Delta \\& l, LA\~\text{init}$ " in Table 1. This suggests that, in fact, $PDA$ and $LA\~\text{init}$ degrade performance when applied together. If my understanding is correct, this undermines the second contribution point (lines 73-77) and should be explicitly clarified in the paper. If my understanding is wrong, why are the results in the diagonal in Table 3 different from either the bottom row of Table 1 or Table 2? 

**Significance**

\+ *This work aims to address a timely problem of fairness-seeking optimization, and the paper will be relevant to the community.*

___
### Post rebuttal update
___
I have read other reviewers' comments and the authors' rebuttals, and I am inclined to keep my original score of 7 with increased confidence of 4, condioned on the additional baselines being included in Table 1 in the main paper. The rebuttal addresses the clarity issues, and the additional results show that the method is at least as good as other baselines [28][63][41] in the Cifar-10-LT setting. The performance advantage over [28][63] in the ImageNet-LT dataset is much more obvious. Although results for [41] in the ImageNet-LT setting were not provided, likely the method would also be beaten like in the Cifar-10-LT setting. I think this work is good, addressing a timely problem and it will be relevant for the NeurIPS conference. 
limitations_and_societal_impact: The paper adequately and fairly outlines its limitations in a dedicated section.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies learning in settings with imbalanced data or under-represented subgroups in the data, where the target objective is fairness-based. E.g. in the setting of imbalanced data, the target objective is to maximize the balanced test accuracy, and in the setting with minority subgroups, target objectives studied are balanced (w.r.t. subgroups) test accuracy, and Difference of Equal Opportunity (DEO) risk. 

The paper is motivated by the fact that many existing methods for imbalanced datasets rely on training theoretically-motivated modifications to the cross-entropy loss, where there are multiplicative and additive adjustments applied to the predicted logits for each class. This paper proposes a bi-level optimization procedure to directly learn these modifications, by directly learning the parameters governing the multiplicative and additive adjustments to the class logits. The bi-level optimization optimizes a classifier on a loss defined with specific instantiations of the adjustment parameters in an inner loop, and the outer loop optimizes some desired test objective (e.g., class balanced loss) on a separate holdout validation set. The paper also proposes learning class-specific data augmentation, with the motivation that minority classes are regularized more strongly than majority classes. 

The empirical results evaluate the algorithm on imbalanced class and imbalanced subgroup settings.n They show that their algorithms can outperform existing methods in several evaluation metrics.
main_review: In summary, this is a strong paper with clear writing, a clean and well-motivated algorithm, and impressive empirical performance. More details below:

Originality: The methods in this paper are novel and related work is well-cited.

Quality: Claims are well-supported by strong empirical results. 

Clarity: The paper is generally clearly written and easy to follow. 

Significance: The results in this paper serve as a nice way to encompass some of the more theoretically-motivated prior work which needed to mathematically derive and justify different modifications to the cross entropy loss. In contrast, this paper observes that directly learning these modifications is possible, given the low dimensionality of the parameter space for cross entropy loss adjustments. This is a nice idea and a neat application of AutoML to the imbalanced learning setting, and could hopefully motivate to other applications of AutoML in related scenarios. 

Some questions/concerns for the authors:

-- The main concern is that the experiments for the imbalanced group setting are only performed for a single dataset -- it would be nice to see experimental results for additional datasets. 
-- What is the comparison of runtime for the proposed algorithm v.s. prior work? Also, what is the schedule for retraining the parameters after the search phase has concluded (e.g, learning rate, # of epochs, etc.)
-- How sensitive are the results to the exact choice of parameters $\alpha$ found by the search phase?
-- For the imbalanced group setting, is there any extension of DEO to the case with multiple groups in a class? 
limitations_and_societal_impact: Yes. Authors address limitations and societal impact in the conclusion section of their paper.
review_title: null

Meta-review: The paper proposes a bi-level optimization framework to design a training loss for long-tail and group-fair learning. The reviews attracted a a lot of back-and-forth discussion with the authors, and I appreciate the authors for providing very detailed responses and additional experimental results.

While there were concerns raised about the proposed approach being prone to overfitting to the validation sample, I think the authors have satisfactorily explained how they take precautions to avoid it. I think Reviewer mSPX's concern about there being limited novelty does have some merit, but given that this is one of the first few works to successfully engineer an elegant loss-tuning procedure for long-tail learning and that the experimental results are significant, I would recommend an accept. 

Having said this, I strongly urge the authors to use the feedback provided to improve their manuscript, and in particular include the following promised additions to the final version:
- a detailed discussion on the risk of over-fitting and the precautions taken to avoid it (do report the validation errors, perhaps in the appendix)
- comparison to other Bayesian optimization approaches
- comparison to the additional baselines mentioned by Reviewer GFgA

Finally, here are some additional references on prior work on learning loss functions for specialized tasks, which may be of some relevance:

https://arxiv.org/pdf/1905.10108.pdf

https://arxiv.org/pdf/1803.09050.pdf
Decision: Accept (Poster)