Venue: AutoML 2023 Conference

Reviews:

final_score: rating: 6: Borderline Leaning Accept: Technically sound paper where reasons to accept outweigh reasons to reject. Please use sparingly.
review_content: summary_of_contributions: This paper proposes a meta-learning method to choose different strategies to deal with imbalance for different classes in problems with many classes.

overall_review: The problem is important and the approach is very interesting and obtains good results.

However, the paper is hard to read, which reduces it's usefulness and potential impact.

review_summary: In spite of the potential value, the effort required to improve clarity is significant.

actions_required_to_increase_overall_recommendation: 1. Improve clarity.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak impact, and/or weak evaluation.
review_content: summary_of_contributions: Real-world datasets may have imbalanced classes (e.g., certain classes may have many examples, while others have few, leading to a long-tail in the class distribution) or may have noisy labels. These data imbalance issues pose challenges for both standard classification accuracy and fairness objectives. To address such imbalances, one may wish to treat each class differently, for example upweighting the contribution of examples from rare (e.g., under-represented) classes.

This paper proposes an approach called Class-Attribute Priors (CAP) for meta-learning a mapping from class attributes (such as the frequency of a class in the dataset, the level of label noise, similarity to other classes, and training difficulty) to an embedding vector that can be passed to a downstream algorithm to modulate the training algorithm on a per-class basis.

The authors use CAP to meta-learn the hyperparameters of the following loss function (which was used in prior work, such as AutoBalance),
$$
\ell(y, f(x)) = \omega_k \log \left( 1 + \sum_{k \neq y} e^{\ell_k - \ell_y} \cdot e^{\Delta_k f_k(x) - \Delta_y f_y(x)} \right)
$$
where $\{ \omega_k, \ell_k, \Delta_k\}$ are class-specific hyperparameters representing the overall loss weighting factor $\omega_k$, the additive logit adjustment $\ell_k$, and the multiplicative adjustment $\Delta_k$.

The $n$ attributes for class $k$, denoted $\mathcal{A}_k$, are mapped to an embedding space by a fixed, nonlinear mapping $\mathcal{F}(\mathcal{A}_k) : \mathbb{R}^n \to \mathbb{R}^M$ , and then this embedding is mapped to a vector of hyperparameters $\mathcal{S}_k = W \mathcal{F}(\mathcal{A}_k)$ via a weight matrix $W$ that is learned. These hyperparameters are the $(\omega_k, \ell_k, \Delta_k)$ that define the loss function for the given class $k$. Conceptually, the embedding serves as a bottleneck that can map classes with similar attributes close to each other, such that the resulting class-specific losses are similar. The idea is that the relatively small number of class attributes can reduce the number of meta-parameters that need to be learned, e.g., $W$ has dimension $s \times M$ rather than $s \times K$ as would be the case if learning separate hyperparameters for each of the $K$ classes directly.

The authors propose two variants of the approach: in the first, they formulate the learning of $W$ as a bilevel optimization problem, where the inner problem optimizes model parameters using the current parametric loss function, and the outer problem evaluates the trained parameters on a validation set, targeting a fairness objective. The second approach performs post-hoc optimization of a function $g$ that modifies the logits output by $f(x)$ to minimize the loss on a test dataset (without performing bilevel optimization).

The paper evaluates CAP on long-tail variants of CIFAR and Imagenet. They use CAP to optimize for different fairness objectives, including quantile class performance and conditional value at risk (CVaR), and they compare to a few prior approaches for dealing with class imbalance. They also measure balanced error, and check which attributes are most useful on different tasks in an ablation study.
overall_review: **Pros**

* The CAP method is simple, and yields few hyperparameters.

* The empirical results are good; CAP outperforms logit adjustment (LA), CDT, and AutoBalance on balanced error in Table 2, and outperforms LA when optimizing for two different fairness objectives in Figure 2.

* The authors implement two instantiations of CAP: 1) using CAP to learn a parametric loss function via bilevel optimization; and 2) using CAP for post-hoc logit adjustment.


**Cons**

* Overall, the paper is not well-written, and this is a major weakness. The method is very simple, but it is not clearly explained; the writing is very wordy, and obfuscates simple ideas using overly abstract and complex terminology. I believe that the paper would benefit from a significant rewrite.

* The paper does not provide an algorithm box detailing CAP.

* I think that the method name "class-attribute priors" is not well-explained; in what way is this a prior? It does not seem to have a Bayesian interpretation. A clearer method name could be "class-attribute embeddings." Also, with regard to method naming, both A2H and CAP are used at different points in the paper, while it would be clearer to use one consistently.

* The introduction---especially the last paragraph from L64-L67---overstates the contribution of this work. To a large extent, this paper can be seen as a minor extension of AutoBalance leveraging a mapping from class attributes to embedding vectors. I believe that the authors should make this clear, rather than stating that "This work makes key contributions to fairness and heterogeneous learning problems in terms of methodology, as well as practical impact."

* The paper downplays its similarities to AutoBalance [1]. Section 3.2 of this paper ("CAP for loss function design") is nearly identical to AutoBalance, the only difference being the use of learned class embeddings. The loss function that is learned is identical to that in [1], and the bilevel optimization setup is the same. Also, part of this paper implies that the hyperparameter dimensionality for AutoBalance scales as $O(K)$ where $K$ is the number of classes; however, this scaling issue was already addressed in Section 2.2 in [1], where they use $K' < K$ dimensional hyperparameters, clustering classes together when they occur with similar frequencies in the dataset. While this is briefly mentioned in Lines 171-177 of this paper, the majority of the paper is still written in a way that suggests that reducing the hyperparameter dimensionality is a novel contribution, which is not the case.

* Overall, the novelty of the paper is limited (and overstated), as it can be interpreted as a simple extension of AutoBalance that learns embedding vectors representing classes, based on attributes of those classes.

* What meta-parameters were used to train the AutoBalance baseline? In particular, what was the dimensionality of the hyperparameter space (how many groups $g$ were used), and how does it compare with CAP?

* L150: Why is this parameterization used for $\Delta$? $\Delta = \text{sigmoid}\left( \sqrt{K} \frac{\mathcal{D} w_{\Delta}}{\| \mathcal{D} w_{\Delta} \|} \right)$

* Because the loss function parameterization is the same as AutoBalance (among other related works), did the authors of CAP try adding the personalized data augmentation (PDA) from Eq. 2.2 in [1]? This would be interesting to see with CAP, as PDA was justified theoretically in [1]. It would be useful to evaluate the impact of using class attributes for data augmentation compared to learning separate parameters for each class.

* No experimental results are provided for the iNaturalist-2018 dataset, which was used in [1].

* None of the components of Figure 1 are described in the main text or figure caption. Figure 1 is not even referenced in the main text at all. The caption should give the necessary context to parse the figure; however, the caption is vague and does not ground any of its statements in the figure. Just some of the questions that arise: What does $S$ correspond to? What does the "Function" box around the attributes box mean? Why is CAP written on the arrow connecting the function to $S$? What does the dictionary inside the attributes box refer to? What does the dashed line from the "full dataset" to the "sample size and distribution shift" plot denote? Why is "heterogeneous" color-coded the same green as the dashed arrow? How should the green and blue points be interpreted (are they two clusters of examples with labels shown by the colors, to illustrate label noise)?

  - As with the rest of the paper, the caption of Figure 1 contains overly abstract and vague language. What is the "global dataset"? What is a "composition of heterogeneous sub-datasets induced by classes"? Why is the term "strategy" used interchangeably with "hyperparameters"?

* The right-hand side of Figure 1 does not seem to be related to the LHS. The caption does not draw any connection between the RHS and LHS; I think that the information in the RHS would be better suited for a table, or simply discussed in the main text without a figure. Otherwise, the reader is left wondering which parts of the LHS are connected to each statement in the RHS.

* The term "meta-strategy" is not clearly defined.

* I think that it would be good to include more ablation studies over CAP. For example, what if only a subset of the parameters of the loss function are tuned (similarly to ablations in [1])? What happens as more features (class attributes) are passed into the embedding layer, increasing the hyperparameter dimensionality of CAP? What if the number of attributes is artificially inflated, either by adding new, random attributes, or by replicating existing attributes several times? Such investigations may be useful to check whether performance improvements are due to implicit regularization from a small number of attributes (and therefore few hyperparameters). If there are many hyperparameters, does CAP overfit to the validation set?

* In some experiments, CAP is only slightly better than AutoBalance, and it is not clear where the improvement comes from (thus the need for more ablations).

* The appendix section dedicated to experimental details is sparse.


**Minor**

* In the abstract, L12-L13, it should be clarified what type of improvement this refers to, e.g., compute/memory requirements, or performance on accuracy or fairness?

* L30: "that tailored" --> "that are tailored"

* L40: "treating hyperparameters as free variables" --> This terminology is unnecessarily confusing; "free variables" are not defined anywhere.

* L47: "as it requires $\mathcal{O}(1)$ --> Shouldn't this be $\mathcal{O}(M)$ where $M < K$?

* L259: "stand deviation" --> "standard deviation"

* L295: What does "in the context of dataset heterogeneity adoption" mean?

review_summary: While the problem is worth studying, and the results are decent, this paper needs a significant re-write to improve clarity. The clarity is a major issue.
actions_required_to_increase_overall_recommendation: In order for this paper to meet the bar for acceptance, I believe it would have to be rewritten to improve clarity, and remove obfuscation. In addition, please see the comments regarding ablations and the comparison to AutoBalance listed in the Overall Review box.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak impact, and/or weak evaluation.
review_content: summary_of_contributions: This paper proposes a novel method for adjusting the hyperparameters of a loss function to account for heterogeneity across classes in (multi-class) classification tasks. The method works by defining a set of per-class attributes, applying a set of transformations to the attributes and then learning a linear mapping from the transformed attributes to the hyperparameters. They consider two strategies: one that involves a bilevel optimization and one that performs post-hoc optimization using a second function, g, applied to the trained model, f. The authors compare their method against previously proposed methods for addressing class heterogeneity and find that their method performs favorably in terms of minimizing balanced error (as well as and other metrics) using common multi-class image classification benchmark datasets. 
overall_review: Positives
- Motivation: this work addresses an interesting and relevant problem. I think with some (substantial) improvement, a version of this work would be well-received by the community. 
- Unifying framework: I enjoyed the fact that the authors demonstrated how existing methods can be viewed as instantiations of their more general method. In this regard, I think this work presents a valuable lens with which we can understand how existing methods relate to one another and how we can continue to iterate on them.  

Negatives
- Writing: as noted previously, I found this work to be very poorly written and would encourage the authors to carefully check their grammar/spelling in future iterations of this work. 
- Limited exploration of attribute space: throughout the work, the authors list multiple kinds of class-level attributes that could be incorporated into their method. As such it was very disappointing to see that their experiments only considered three relatively simple attributes.
- Poorly justified design choices: related to my point above, the choice of attributes considered is not well-motivated. In addition, other aspects of their proposed method are not well justified/explored e.g., the nonlinear feature map,
- Empirical evaluation: as previously noted, it is unclear which, if any, of the empirical results are statistically significant; Appendix F seems to indicate that the majority of experiments are only repeated 3 times, which is quite low in my opinion. 
review_summary: My recommendation is to reject this work in its current form due to the weak/unclear empirical evaluation, insufficient detail in the description of the method, and poor writing.
actions_required_to_increase_overall_recommendation: I am willing to increase my score, potentially to an accept-level score, if the clarity issues are addressed and the questions raised in the "Technical Quality and Correctness" section are answered.
review_title: null

final_score: rating: 9: Strong Accept: Technically flawless paper with major impact and strong evaluation, with no obvious flaws. Should be highlighted in the program.
review_content: summary_of_contributions: This paper focuses on the supervised classification problem where the number of classes are relatively high, and the classes are significantly imbalanced, and the goal is to enable learning of a fair classifier with explicit focus on the performance of the classifier on the smaller "tail classes". To this end, the paper focuses on the use of class-specific hyperparameters that allow for fair classification, and proposes the use of "class attributes" -- class-specific data meta-features -- to learn a mapping from the class-attributes to class-specific hyperparameters. This setup is well motivated since the number of hyperparameters to be optimized in the vanilla case can be high for large number of classes, and the use of class-attributes and a corresponding mapping significantly reduces the number of parameters to be estimated. This is especially useful when considering classes with very few examples. The paper demonstrates how some of the existing schemes to address this problem can be obtained as special cases of the proposed class-attribute based scheme. The empirical evaluation demonstrates that the proposed use of class-attributes leads to significant performance improvement on tail classes across multiple datasets, fairness objectives, and class heterogeneity, relative to existing schemes.

overall_review: To the best of my understanding, this paper studies an important problem of fair classification, and presents a fundamental framework for class-fair learning with significant practical impact on tail classes. The paper is well written, with the contributions well articulated, the proposed approach well positioned against existing work. The empirical evaluation clearly motivates each experiment, highlighting different properties of the proposed scheme, and explicitly discusses the conclusion from each of them. It is also relevant to AutoML in the sense that it is trying to configure hyperparameters (such as the hyperparameters of the loss function) and making use of data meta-features. 


I am unable to identify any significant negative aspect of this paper. There are some minor clarification issues (as I have discussed in the **Clarity** section, but nothing significant that cannot be easily fixed.

review_summary: I think this paper studies an important problem and presents a well-motivated intuitive approach with strong empirical performance against existing baselines. I do not recognize any significant negative aspect of this paper. Hence, I am recommending this paper be accepted.

actions_required_to_increase_overall_recommendation: I am already scoring this paper fairly high. I will increase my score if the authors can improve my understanding of this paper and highlight the challenges, if any, in learning the **A2H** mapping and how they mitigate them.
review_title: null

final_score: rating: 8: Accept: Technically sound paper with major impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: The paper proposed new meta-strategy class-attribute priors (CAP) that support improving fairness objectives while adapting to the different heterogeneities of the individual classes. Results show that using CAP can lead to an improvement in fairness objectives performance. 
overall_review: The paper presents a compelling argument to use the CAP meta-strategy to address differences in heterogeneities across classes when optimizing for fairness objectives. 

There are minor gaps that can help strengthen the paper further. For example, In section 1.1, the authors discuss how the bilevel problem is time-consuming. However, it needs to be clarified if the proposed method is time-consuming. If yes, by what scale? It is vital to connect the benefits of CAP back to the problem with prior techniques.

review_summary: Class label attributes can influence the model performance and fairness objective when working with real-world data. While there are several methods out there in the field, there is no one method that solves them all. Different techniques can help in different instances. So I believe this CAP strategy can bolster this filed of research.
actions_required_to_increase_overall_recommendation: A couple of statements attempt to connect the benefits of CAP back to the problem with prior techniques.
review_title: null

Decision: Accept(Workshop)