Understanding and Reducing the Class-Dependent Effects of Data Augmentation with A Two-Player Game Approach
Authors: Yunpeng Jiang
Yutong Ban
Paul Weng
Venue: Accepted by TMLR ()

Abstract: Data augmentation is widely applied and has shown its benefits in different machine learning tasks. However, as recently observed, it may have an unfair effect in multi-class classification. While data augmentation generally improves the overall performance (and therefore is beneficial for many classes), it can actually be detrimental for other classes, which can be problematic in some application domains. In this paper, to counteract this phenomenon, we propose CLAM, a CLAss-dependent Multiplicative-weights method. To derive it, we first formulate the training of a classifier as a non-linear optimization problem that aims at simultaneously maximizing the individual class performances and balancing them. By rewriting this optimization problem as an adversarial two-player game, we propose a novel multiplicative weight algorithm, for which we prove the convergence. Interestingly, our formulation also reveals that the class-dependent effects of data augmentation is not due to data augmentation only, but is in fact a general phenomenon. Our empirical results over five datasets demonstrate that the performance of learned classifiers is indeed more fairly distributed over classes, with only limited impact on the average accuracy.

Published in Transactions on Machine Learning Research (06/2025)
Understanding and Reducing the Class-Dependent Effects of Data Augmentation with A Two-Player Game Approach
Yumpeng Jiang
jup9961 1 @dsjt..ed.cor
Shanghai Jiao Tong University
Yutong Ban"
glan@sxjhu.edu.cr
Shanghai Jiao Tong University
Paul Weng'
paul. www.ke/wwdddukcknnnsnnneeee.....
Duke Kunshan University
Reviewed on OpenReview: https: // opw.reieee.r..w....mmmm net/ forum? id 2NssfOnss 7a
Abstract
Data augmentation is widely applied and has shown its benefits in different machine learning tasks. However, as recently observed, may have an unfair effect in multi-class classification. While data augmentation generally improves the overall performance (and therefore is beneficial for many classes), can actually be detrimental for other classes, which be problematic in some application domains. In this paper, to counteract this phenomenon, we proposc CLAM, CLAss-dependent Multiplicative-woihht, method. To derive it, we first. formulate the training of a classifier as a non-linear optimization problem that aims at simultaneously maximizing the individual class performances and balancing them. By rewriting this optimization problem as an adversarial two-player game, we propose a novel multiplicative weights algorithm, for which we prove the convergence. Interestingly, our formulation also reveals that the class-dependent effects of data augmentation is not due to data augmentation only, but is in fact general phenomenon. Our empirical results over six datasets demonstrate that the performance of learned classifiers is indeed more fairly distributed over classes, with only limited impact on the average accuracy.
1 Introduction
Data augmentation is a popular technique used in the field of computer vision to improve both training efficiency and model performance (Kunar et al. 2023 Ma et al.) 2022). It involves creating new training samples by applying semantically pww.llvviieeeeeigggggg transformations (Shorten & Khoshgoftaar 2019)), such as random shift (Kostrikov et al.) 22000] Yarats et al. 2021), random convolution (Hansen al.) 2021) and random resized cropping, to the original data. The aim is to increase the diversity of the training dataset and prevent overfitting, leading to better generalization to the lesting dataset. However, recent research (Balestriero et al. 2022) has highlighted potential issue with data augmentation: although it is beneficial in improving the overall performance, data augmentation can lead to disparities in performance across different classes. This means that while there are significant improvements on the performance some classes, others may experience a decline. certain classes are consistently under-performing, it can lead to biased outcomes and lack of equitable treatment for all classes (Lin et al. 2017), which is important, for instance, in autonomous driving where good label recognition in all classes (e.g., road signs) is crucial to achieve road safety, As result, there is growing interest in developing methods to achieve equitable performance among all classes within classification tasks, notably to ensure that the benefits of data augmentation are more
* *Corresponding authors.
For concreteness, we focus accuracy. approach could also applied with other performance measures.
1

---

Published in Transactions on Machine Learning Research (06/2025)
V1
V2
Vi
V2
Improve Overall
Accuracy
Max player
V6
V/
V3
2 Improve Fairnsss
V/5
V4
Min player
Figure 1: CLAM formulates the classification problem as an adversarial two-player game, where is represents the accuracy of the jt class in the classification problem. The min player determines the weights for classes, while the max player aims to maximize the weighted accuracy of the model.
evenly distributed across all classes, rather than being concentrated in a few. This involves finding ways to mitigate the potential negative impact of data augmentation on certain classes while maintaining the positive effects on others.
In this paper, our goal is to address the challenge of reducing the class-dependent effects of data augmentation in classification tasks without much compromise to the overall model performance, aiming to limit the degradation on certain classes caused by data augmentation while maintaining a balance between overall accuracy and class equity. We relate this problem to fair optimization (Ogryczak et al., 2014), which seeks to obtain good solutions with a balanced prosile in the context of multi-objeclive optimization. been applied in resource allocation problems ensure fair treatment of the multiple involved parties. In work, we apply fair optimization to the realm of classification tasks with data augmentation. As illustrated in Figure our fair optimization problem is formulated game. The max player aims at maximizing the overall accuracy of the model while the min player focuses on ensuring fairness over all classes. Instead of the typical max-min fairness formulation, we propose a novel refinement of it to ensure that the accuracies of all classes are optimized, not only that of the worst performing class. Tn this strategic interaction, the min player determines the weight for each class, assigning larger weights to corse-performing classes, while the max player uses these weights and optimizes the model so that the weighted overall accuracy is maximized. Unlike approaches based on maxmin fairness that concentrate solely improving the accuracy of the worst-performing class, our strategy ensures that the accuracies of all classes are enhanced, leading to more balanced distribution of class accuracies. As a side note, our proposed approach actually also extends the popular fair optimization formulation based on the Generalized Gini social welfare function (GGF) [Weymark] 1981), which achieves equily by assigning larger weights to ranked objectives. An important difficulty when using GGF in practice is deciding how to set its fixed weights, which our proposition avoids.
has notably
adversarial zero-sum two-player
To lackle this two-player game, propose employing an adaptation of the multiplicative weights algorithm (Freund& & Schapire| 1999). Our method dynamically adjusts the weights assigned to each class to ensure that more focus is given to classes with weaker performance. By adopting this multiplicative weights method, we aim to achieve a state of equilibrium where the model's overall accuracy is optimized while simultaneously ensuring fair treatment among all classes. Traditional classification frameworks commonly prioritize overall accuracy while neglecting individual class disparities. By contrast, our innovative method introduces a mechanism that encourages the min player to consider the max player's objective, leading a trade-off between overall accuracy and class equity.
Contributions Our main contributions are three-fold: (i) We propose to reformulate classification using a novel variant of fair optimization to reduce disparities among different classes, which can be viewed as an adversarial zero-sum two-player game. This formalization highlights that the class-dependent effects of data augmentation is actually a more general phenomenon and not only due to data augmentation, which we have also experimentally confirmed. (ii) We propose multiplicative weights optimization method to solve this
2

---

Published in Transactions on Machine Learning Research (06/2025)
two-player game and further analyze the upper bound of the average loss, which proves the convergence of our method and gives an upper bound on the convergence rate. (iii) Experimental results show that our method mitigates the class-dependent effects of data augmentation, reduces the increase of these effects with stronger data augmentation, enhances the worst class accuracies, and meanwhile maintains a minimal impact on overall average accuracy.
2 Related Work
In this section, we introduce related works in fair optimization, fair machine learning, and class equity. Fair optimization integrates fairness-related objectives into the optimization process. In contrast, fair machine learning focuses on developing algorithms that do not discriminate against any particular group based on sensitive attributes. Most relevant to our work, class equity addresses the fairness of a classification model's 's predictions across all classes.
Fair Optimization In the context of fair oplimization (gryezak et al. 2014), the Generalized Gini social welfare function (GGF) [Weymark] 1981), which includes the standard max-min fairness (Rawls) 1971], 1 plays a pivotal role in multi-objective settings. The application of GGF for assigning weights is to ensure that the learned model is more equitable and treats all objectives fairly. For instance, in more recent applications in machine learning, (Siddique et al. (2020) utilize GGF as a means to guarantee fairness over users in deep reinforcement learning. With the help of GGF, it becomes possible to reduce bias towards any single objective, thus fostering a fairer learning process. More generally, fair social welfare functions have been advocated
Heidari et al. 2018 Speicher et al. 2018 Weng 2019 as a tleoretically-fouddee approach to tackle fairness in machine learning.
Fair Machine Learning Tn fair machine learning, previous works consider different definitions of fairness, such as individual fairness and group fairness. Individual fairness focuses on the uniformity of predictions for subjects who possess comparable features relevant to the recision-making process. In group fairness
(Krishnaswamy el al. 2021 Jung et al., 2021 Alghamdi et al. 2022 Park el al. 2022 Anchlia & Choi 2023). the focus is on ensuring that the prediction outcomes for different groups are balanced or equal. In binary classification problems, an algorithm learns a model to predict a target variable and there is a sensitive variable that defines the groups within which to measure the fairness. Group fairness considers three main criteria, independence, separation and sufficiency (Caton & Haas 2020).
In fair machine learning field, some existing works leverage adversarial two-player game strategy. Xu et al. (2019) have trained generative adversarial networks (GAN) designed to concurrently achieve equitable data generation and classífication. This is accomplished by co-training the generative model and the classifier through joint adversarial games with discriminators. Martinez et al.] (2020) concentrate on group fairness concerning disparities in predictive risk. They concept.ualize Tairness as a minimax problem, aiming to discover the classifier that offers the smallest maximum group risk among all efficient models. To address this, they propose the Approximate Projection onto Star Sets (APStar) algorithm, which iteratively refines the minimax risk by updating a linear weighting vector. Closest to our work, (Agaraal el al. (2018) reduce fair classification to series of cost-sensitive classification challenges. They reformulate the issue as adversarial two-player game and resolve it by optimizing the Lagrangian. Even though the above methods use the max-min two player game strategy, our problem and the formulations, detailed in Section are distinct.
More generally, the objective of our fair classification problem different to these fair machine learning methods. Our emphasis on obtaining a classifier whose performance is fair with respect classes. In our context, the sensitive variable is the same as the target. Consequently, the three main criteria, independence, separation, and sufficiency, no longer apply in our scenario. Moreover, our focus is on reducing the class- dependent effects of data augmentation across significantly larger number of classes rather than achicving fairness among constrained number of groups
Class Equity Specialized loss functions have been introducod achieve class equity. These approaches are general algorithmic fairness methods for class equity, which can be applied independently of data augmentation. Notably, Lin et al. (2077) introduce the focal loss for the task of dense object detection, which strategically
3

---

Published in Transactions on Machine Learning Research (06/2025)
decreases the training loss from correctly classified instances, thus concentrating the model's learning efforts on the challenging samples. This approach aims to achieve instance fairness, which can also be leveraged in our context for class equity. Szabó et al. (2021) propose tilted cross entropy (TCE) loss, which fosters fairness in semantic sogmentation tasks by dynamically assigning weights to various classes based on the training losses. Addressing the problem of model pruning, Meyer & WWng (2022) recognize that while pruning might minimally impact overall performance, it can inadvertently introduce biases, leading to performance degradation for certain sample subsets. Inspired by focal loes, the authors propose performance weighted loss function aimed at mitigating such unfairness. Bitterwolf et al. (2022) extend evaluation metrics to measure the performance a group of worst classes, leading to natural approach that applies the max-min formulation to achieve class equity. In our work, we reformulate classification using fair optimization. Similarly to these methods, our approach can reduce the class-dependent effects of data augmentation while remaining generic and applicable regardless of data augmentation.
Another line of work considers label noise (Scott & Zhang 2020 as a possible cause of class disparities. Thus, one strrtegy to mitigate the unfairness of per-class accuracy is through the process of relabelling (Kirichenko) et al. 2023). Aiming at reducing the class-dependent effects of data augmentation, like in work, they demonstrate that by leveraging the more procise multi-label annotations available in ImageNet, the typically negative impact of data augmentation on individual class accuracies is substantially reduced. While their work emphasizes the enhancement of label quality, we directly aim at reducing the class-dependent effects of data augmentation by reformulating the maximization of the average accuracy (see Equation (3)) into a max-min problem (see Equation
( encoding fairness over classes. Our approach is complementary to their method and can potentially be combined with theirs.
3 Background
In this section, we provide detailed introduction to the generalized Gini social welfare function and the multiplicative weights method, both of which are closely related to our work. Before that, we first introduce a few notations, which will be useful in the following weighting schemes: A, ww E [], Vi, w; 2 0; E, w, 1} demoles the (n - 11]simppee which is the set of all non-negative n-dimensional vectors w that sums to one. This simplex is the space to which the weight vectors used in our proposed two-player game belongs. For vectors v, v' E R", the notation v S o' means that each clement of is less than or equal to the corresponding element in v. Denote the symmetric group of order n (i.c., set of all permutations of n elements) with Sn and let u. (Wa(1); (.(2))--.;; Wa(n)) be a reordering of the weight vector w according to the permutation s.
Generalized Gini Social Welfare Function (GGF) GGF (Weymark 1981 extends the concept of Gini coefficient, often used to measure income equality, to measure of social welfare that encodes fairness:
GG) Σ w. w.(a(f)
(1)
where E An such that wi > 12 >> ... Wn. IR is a utility vector, and Sn is a permutation such that (1(1) S Us(2) S... S Ms(x). As can be seen from its definition, GGF encodes fairness by assigning higher weights to worsc-off components. With different settings of w, higher GGF value indicates cither higher overall utility or more equitable distribution. Interestingly, by setting w close to (1,0,..., O), GGF can encode the classic max-min fairness. One difficulty with exploiting GGF in applications is that it is not always clear how to choose this weight vector w, which actually controls a trade-off between efficiency and equity.
Multiplicative Weights Method This method (Fround & Schapire 1999,, also known as hedge, is a learning algorithm, which can be applied by a player in adversarial (wo-player games. It updales the probabilities of the player's strategy based on the outcomes of previous timesteps. Without loss of generality, we assume that the player the min player. The player initializes a mixed strategy, i.c., probability distribution w E An over all her strategies (e.g., with a uniform distribution). Then at each timestep t, the player chooses strategy sampled according to her current mixed strategy w and incurs loss us according
4

---

Published in Transactions on Machine Learning Research (06/2025)
to the other player's choice and the payoff matrix. The player then updates her mixed strategy using the multiplicative weights method. The probability of strategy is updated as:
w --
with Z
w
(2)
Zz.
i=1
where 7 > 0 is hyperparameter. This process is repeated for multiple rounds until convergence. The hyperparameter T controls the update rate of the probabilities. Freund & Schapire (1999) have proved that with an appropriate choice of 7, the mixed strategy is guaranteed to asymptotically suffer average loss close to the minimum loss achievable by any fixed strategy.
4 Methodology
In this section, we introduce our proposed approach, CLAss-dependent Multiplicativv--eighhts (CLAM), where the solution should satisfy two generally contradictory objectives. Firstly, the average accuracy over all classes should be maximized. Secondly, the classifier should not favor any particular class over others. For this purpose, we formulate the classification problem adaptation of the multiplicative weights method, We then provide a theoretical proof of its convergence to an optimal fair classifier. Finally, we conclude this section with the practical implementation of our adapted multiplicative weights method.
advensarial two-player game, which motivates our
4.1 Classification as An Adversarial Two-Player Game
To motivate our formulation of the classification problem (see Equation 5
we first recall the standard classification problem. Traditionally, an n-class classification problem is solved by simply maximizing the average performance over all classes:
1 Σ (0),,
max
(3)
where 0 is the parameter of the classification model (e.g,, neural network) and v0()) is the accuracy achieved by the model for the tth class. As usual in deep learning, this optimization problem is solved approximately by training the model (c.g., minimizing an empirical risk expressed with a convex loss such as the cross-cntropy loss). Training is usually performed with the assistance of data augmentation, which can significantly improve the sample efficiency.
However, this standard approach often results in skewed performance distribution over classes, favoring some others. To counteract this imbalance, naive approach would be to resort to the standard minmax fairness F which can be interpreted as zero-sum two-player game. This alternative perspective shifts the focus from the average performance to the performance of the worse class
max min 8 WEA,
Σ (.)
(4)
This formulation aims to enhance the worst-class performance by assigning them greater weights. This two-player game can be solved using the multiplicative weights algorithm: At the end of training epoch t, the min player chooses w, probability distribution over classes. With the knowledge of w, the max player computes her best response: she optimizes the weighted objective function Σ. w,,((0) by updating 0. After the model parameter update at epoch ++1, the loss of the min player is computed by the class accuracy weighted by w. Due to the nature of the zero-sum game, the loss of the max player is the negative of the loss incurred by the min player.
Nonetheless, this max-min approach is insufficient. Indced, the min player would assign the weights w such that the worst class receives weight of while all the others receive 0. By focusing solely on the worst class,
Here, this leads max-min formulation since we want maximize performance.
9hhis is equivalent to max min, v0))
5

---

Published in Transactions on Machine Learning Research (06/2025)
the performance of the other classes is neglected when training the classifier (i.e., max player), leading to sub-optimal outcomes of the classification model as whole. As result, another class would achieve the worst performance and at the next iteration, the weight assigned by the min player would shift to it. This oscillating phenomenon is observed in our experiments: the accuracy fluctuates and the training does not converge.
To avoid this extreme case, we instead use constrained max-min formulation:
max min Σ (O) # WEB;
(5)
where A is a compact convex set, which is a strict subset of An such that Umin min, min w; > 0 (excluding the extreme points of A,). We assume that any mixed strategy of the max player can be represented by 0. Under this assumption, the max-min problem is equivalent to the min-max problem by the Minimax theorem. Depending on applications, different A could possibly be defined. In contrast to choosing fixed weights as in GGF, we expect that its definition may be easier since one may define A, such that whole ranges of GGF weights are allowed.
Set A needs to satisfy some properties such that f(v) = min Ae the following proposition whose proof can be found in Appendix A Proposition 4.1. If Aú satisfies the following property: Ve E A Vs E Sn, w. E An then f is
>,
w models class equity, as stated by
monotonic: Ye, v' v S o' f(v)= S f(0'),
Schur-cononve: Yu,u; Vj, VE 0, 1/4 Uj, s(v cej -ce)) 2 f(v),
symmetric: f(v) f(va);
where ej represents a vector that is 1 at the jth element and 0 otherwise.
The mononicity property ensures that all the components should be maximized. Schur-concavity states that if we take any two components i2 and 0j of the vector v such that 0/ > Uy and we increase vy by small amount while decreasing U; by the same amount c, the value of will not decrease, which is consistent with the idea that redistributing resources from more advantaged component to less advantaged component should not result in a worse outcome. Symmetry states that the value of the function f remains unchanged regardless of how the components of the vector U are permuted, which ensures that the final outcome does not favor any particular component of the vector v. In other words, the function f treats all components of o equally, which is a fundamental aspect of fairness.
The refined formulation in Equation (5) ensures that the maximization of the worst-class performance occurs with controlled set of weights. Then the efforts to enhance the accuracies of the worst classes may not come at the cost of substantial reductions in the performance of other classes. This offers more balanced approach to improving the worst-class performance while minimizing the impact on the other classes. This constrained max-min formulation defines our fair optimization problem. Interestingly, when contrasting the traditional formulation (Equation (3) with our proposed one (Equation (5), becomes evident that this issue of class disparities is not specifically related to data augmentation but rather a general problem. While data augmentation can exacerbate this effect, the underlying issue still exists even without it. Also, as another consequence of our formulation, this class-dependent disparity in performance would occur regardless of the network architecture used.
Before explaining how this refined problem can be solved by adapting the multiplicative weights method, we further discuss how it is related to fair optimization as modeled by the GGF, which may also help better understand our novel formulation. This refined problem shares a close relationship with the GGF (Equation 11) By maximizing GGF during training, the model not only optimized for higher overall accuracy but also for fairness among classes, leading to an increase of the accuracies of the worst classes Interestingly, GGF can be rewritten as follows:
GGFw(v(f)) = -min Σ W.(t) W.(o),;()) (0),
(6)
sE3,,

---

Published in Transactions on Machine Learning Research (06/2025)
Epoch
Epoch
NN
Per-class accuracy
Per-class accuracy
Payoff Matrix M
Max
Payoff Matrix g
Player
Update
W
WitH
Min Player Update w
Min Player Update Wa12
Our method reduces the class-dependent effects of data augmentation iteratively epochs. At epoch L, the max player updates the model parameters Of by loss function that incorporates weights wt- Subsequently, the min player adjusts the wy through a multiplicative weights algorithm based on the accuracy of each class, (8*), to obtain the weights for the next epoch W1+1- Consequently, the max player loss of epoch + 1 is calculated with the updated weights wt+1- The above process repeats for each epoch. For the payoff matrix M, each row corresponds to a class i, each column represents a set of model parameters 0, and each entry (0) indicates the accuracy achieved by the model with parameters when evaluated on class i.
Figure 2:
where Wx(s) 's are reordered based on s. The min over ensures that at the end, higher ws(() are assigned lower (s)sss. Using this rewriting, optimizing GGF corresponds to the following max-min problem:
max min SEB,
Σ ((
(7)
This problem formulation indicates that when we train the model to optimize the GGF-based objective function, the min player chooses a permutation s, while the max player tune the weights 0 to maximize the weighted sum defined by the min player. Therefore, our fair optimization problem defined in Equation 5 includes the GGF-based formulation by setting the constrained simplex to only contain weights that corresponds reorderings of GGF weights However, our generic fair optimization problem is more flexible since it does not require to define precisely weight vector w, which is hard to choose in practice.
4.2 Fair Optimization via Multiplicative Weights Algorithm
As defined above, the min player has mixed strategy over classes while the max player maximizes the sum of weighted accuracies by optimizing 0. This adversarial two-player game is well studied. The min player can minimize its loss by applying the multiplicative weights method to attach significance to the worst classes. The max player can directly apply the same weights of all classes w as the min player, which ensures that the accuracies of the classes v² emphasized by the min player are improved most the next training epoch L ++1 In Equation (5) we introduce a constrained simplex A,, This modification introduces some consequences on the theoretical guarantee, which will be discussed in Section 4.3 The overview of our method is shown in Figure 2 According to this modification, our min-player update rule can be transformed into the following:
t +- w xx -T
2
(8)
++1 t π( Σ. u w
where 7 > 0 is a hyperparameter and æ to denote the Euclidean projection on AS
The pseudocode provided in Algorithm [ outlines the adapted multiplicative weights algorithm. Compared to the standard multiplicative weights algorithm, the key difference lies in the update of the weight voctor w in line of Algorithm | After training the classifier, the accuracies for all classes 8² are used update w It is important to note that w is confined to restricted simplex Aę rather than the entire simplex A..
7

---

Published in Transactions on Machine Learning Research (06/2025)
Algorithm 1 Adapted Multiplicative Weights Algorithm
Hyperparameters: total number of training epochs T, the restricted simplex A.
1: Initialize model parameters 0, weight vector w".
2: for = O do
3: // update 0 (max player) Update such that arg max E, (O).
// update w (min player)
Compute the accuracies of all classes o² at iteration
Update wt according to Equation
end for
4.3 Theoretical Proof of the Convergence
In this section, we prove that the average loss suffered by the min player in Algorithm is approximately equal to the minimum loss achievable by any fixed strategy, which shows the convergence of Algorithm D Theorem 1. The average loss of the min player's strategy compared to the optimal fixed strategy has
upper bound if we set as In(1 + I+maxa,
ln
T
T= S
T= Vū, + Inn
Inn T
EV T
Ve
Σ wEA:
min
T - (1 + max an)
(9)
T
where T is the number of total epochs, to be any arbitvary weight vector in Dú w is the weight vector chosen by our method at epoch t, v² is the accuracy vector for all classes at epoch t, Vw w is the weighted accuracy of choosing u when the accuracy veclor is to and con also be regarded as the loss for the
min player, 04 En 10
Eneso uae
x++
w and c x(x(())
16.0)
Ve.
x
The detailed proof of Theorem I can be found Appendix B Here we give an outline of the proof. We introduce an intermediate variable t which can be any arbitrary weight vector and compare the difference in KL divergence between consccutive timesteps: K KL (L@l|l=*+1 K KL(Wl| lw").. This difference expressod in termss of the original variables w² and v², allowing to find an upper bound involving т, an, Va,e* and Yet. Compared to the proof given by Freund & Schapire (1999), at this step our proof involves another difference term induced by the restricted simplex A which can also be represented by and Vwe, By summing this bound from = 1 to t = T, we obtain an overall bound for KL( Finally, using the property that w a uniform distribution by default and setting a specific value for т, we derive an upper bound on the average loss of our strategy. This bound gets smaller as T gets larger, showing that our strategy approaches the optimal performance achievable by any fixed strategy the constrained simplex.
K KL(Wll['))
Algorithm I suggests that the min player should play the average (over time) of the w as its stationary strategy, which would then yield a near-optimal lair classifier. However, this would require costly computation to obtain it. Instead, we observe that if weight vector w' converges, then using the last iteration is sufficient. We formalize this point in Lemma in Appendix C
4.4 Practical Implementation
In practice, the restricted simplex A in Algorithm D is selected such that the ratio between the maximum and minimum class weights is lower than a given value. This is lo ensure that the max-min approach does not converge to an extreme point, where all probability weight is assigned to one worst class and zero probability all the others. Proposition 4.1 requires that when the order of weights in the weight vector E A changes, we can still find the modified weight vector w, in A, The A we choose in practice does not restrict the order of weights in the weight vector that it always satisfies Proposition 4.1 in diverse scenarios.
8

---

Published in Transactions on Machine Learning Research (06/2025)
Algorithm CLass-dependent muultipliiative-weihht Algorithm (CLAM)
Hyperparameters: total number of training epochs T, total number of itcrations in every epoch K, multiplicative weights update parameter 7, the lower bound of the weights umin- 1: Initialize model parameters 0, weight vector wº.
2: for t ... T do
3: // update 0
4: for k= 0 do
5:
Sample a minibatch of samples from training set and compute loss L for class Update the model with the gradients of the loss La Di=1 w/ L. end for
8: // update w²
9: Compute the training accuracy of all classes of at epoch t.
10: Update w² using T, Umin and of via Equation S
11: end for
A practical implementation of the adapted multiplicative weights algorithm is essential due to the realization of optimization through gradient descent. In the thcoretical formulation, the algorithm involves updating to maximize the weighted training accuracies. However, this theoretical approach requires finding an exact solution, which can be computationally infeasible for large-scale problems and can actually also be undesirable due to overfitting issucs. To address this challenge, we approximately maximize the weighted training accuracy by updating 0 using gradient. descent. The pseudocode in Algorithm / describes the practical implementation of Algorithm [ Note that in contrast to Algorithm I where the max player A aims to maximize the weighted training accuracy, in Algorithm 2 loss function is used surrogate due to its continuous and differentiable nature, enabling gradient- based optimization for 6.
Theoretically, to ensure unbiased estimation, we should update the weight vector based on accuracios evaluated using a held-out validation dataset. However, empirical observations indicate that this approach can lead to decreased test accuracy. We conjecture that this could be due to the reduced size of the training dataset and possibly higher variance of accuracy estimates in the validation dataset. Consequently, in our proposed method, the accuracy in the training dataset to update the weight vector.
5 Experimental Results
5.1 Baselines, Setups, and Evaluation Metrics
To investigate whether our method alleviate the class-dependent effects of data augmentation, we perform a set of experiments comparing our method with baselines across six different classification tasks.
Basclines Except for normal cross entropy loss (Normal), we further select four bascline mothods, which are specifically designed for training a fair classifier. These include focal loss (Focal) (Lin et al. 2017 tilted
) cross entropy loss (TCF) (Szabó et al., (202), performance weighted loss (PW) (Meyer & Wong 2022 and
GGF-enhanced cross entropy loss (GGF) (Weymark, 1981 Siddique et al. 2020 The detailed descriptions
and hyperparameters are listed in Appendices and G
Experimental Setup To evaluate the performance of our method, we run experiments on six classification
tasks: CIFAR-10 10, CIFAR-100 Krizhevsky & Hinton| 2009), Fashion-Mnist (Xiao et al. 2017). iNaturalist2018
(Hrrn et al. 2018), Mini-ImageNet and ImageNet (Russakovsky et al. 2015). Regarding the network architecture, since the issuc of class-dependent effects is brought up by Balestricro et al. (2022) where they employ ResNet-50 models, we follow their experimental setup and train a ResNet-50 model from scratch as well. As discussed in Section 4.1 our mathematical formulation reveals that this issue and our proposed method are independent of the neural architecture of the classifier and one may expect similar results with other architectures (c.g., ViT (Dosovitskiy et al. 2021)). Either random resized cropping or color jittering is applied during training. For each method and each task, we run experiments with 8 data augmentation parameters,
9

---

Published in Transactions on Machine Learning Research (06/2025)
Table 1: We present the fairness metrics of different methods on five datasets in the following two tables. In the second column of each table, 'std', 'COV', and 'range' respectively denote the standard deviation, coefficient of variation, and range of class accuracies on test datasets. For each method used on each dataset, the reported result is averaged over crop lower bounds. The std and range are listed in % while COV is listed in values. Lower is better.
CIFAR-10
CIFAR-100
Pushion-Mnist
Mini-ImageNer
ImageNet
Method
sid COV
range std
COV
range
std
COV
range std
COV
range std
COV
range
Normal
4.55
0.049
12.6
11.8
0.157
39.1
5.94
0.063
18.3
11.2 0.152
37.0
17.3
0.247
58.5
Focal
4.76
0.052
14.1
11.3
0.150
37.2
6.05
0.064
18.9
11.2 0.151
36.6
17.0
0.252
57.7
PW
4.65
0.050
13.4
11.4
0.152
37.8
6.12
0.066
18.7
11.2
0.151
36.5
17.4
0.256
58.7
TCE
4.56
0.049
13.0
11.6
0.156
38.7
6.14
0.066
19.0
11.4
0.155
37.2
17.1
0.245
58.0
GGF
4.12
0.044
11.9
11.2
0.149
36.9
5.98
0.064
19.4
12.1
0.172
41.3
16.9
0.242
57.2
CLAM
3.86 0.042 10.7
10.8 0.143 35.9
5.44
0.058 16.7
10.6 0.143 35.1
16.7
0.241 56.8
(Ours)
remel
lue C_liiant
(LiMioen
GuPkar
12
1000
Figure 3: Fairness metries over training epochs on CIFAR-100 and dini-Imagenet. 'std' and 'COV" denote the standard deviation and coefficient of variation for evaluation accuracics. (Lower is better.)
crop lower bound for random resized cropping, and brightness, contrast, and saturation adjustment for color jittering.
Evaluation Metrics To investigate whether our mothod can alleviate the dass--depeneent effects of data augmentation, we calculate the range, standard deviation, and the coefficient of variation (COV) of class accuracies test datasets. COV measures relative variability and is calculated as the ratio of the standard deviation to the mean. We also compare the worst class accuracics achicved by our method on test datasets with those of the baseline methods. The mathematical definitions of these metrics can be found in Appendix
5.2 Results and Analysis
Through these experiments, we aim to provide detailed answers to several key questions. First, we investigate whether our method alleviate the class-dependent effects of data augmentation. Next, we assess if our approach roduces the increase of these effects cansed by data augmentation. We also examine the improvement in performance for the worst classes. Additionally, we present the evolution of the weight vector during training. Furthermore, we evaluate whether our method significantly degrades the overall performance. Due to the page limit, we present the comparison of our method and group fairness methods in Appendix D and the hyperparameter sensitivity study on in Appendix L
Does our method alleviate the class-dependent effects of data augmentation? To effectively address whether our method alleviates the class-dependent effects of data augmentation, we present the results from five datasets in Table [ Consistently across all five datasets, our method leads to a remarkable roduction in the standard deviation and range of class accuracics. This reduction indicates a narrower spread of accuracies among different classes, which is a positive indicator for fairness. The cocfficient of variation further corroborates the trend by demonstrating a reduced relative variability in class accuracies,
10

---

Published in Transactions on Machine Learning Research (06/2025)
Table 2: Difference in test accuracy range (with DA - without DA) across five datasets, averaged over seven crop lower bounds with data augmentation. Lower values indicate lower increase of class-dependent effects of data augmentation. The difference values are listed in %. Lower is better.
CIFARIO CTFAR100 FMuist
MImageNel ImageNet.
Method
Difference in Test Accuracy Range (mean + std)
Normal (2007)
1.0=1.0
0.5
0.5:00.9
0.6:0.88
-3.0+10.5
Focal (ICCV2017
1.7=0.5
-1.1
2.3
1.520.7
-3.5+0.6
PW (NeurIPS2022
-0.5+0.6
2.9+0.8
1.141.8
0.1+0.6
0.0+0.88
TCE (CVPR202))
-1.240.5
-1.840.5
0.54116
-5.240.2
-1.510.6
GGF (ICMI 2020
-1.740.6
0.3+0.7
-0.7#1.1
0.4+1.7
0.0+1.0
CLAM((Ors)
-2.4+0.3
-2.3+0.7
-1.4+1.4
-3.5+0099
-4.0+-0.8
Table 3: The worst class test. accuracies of different methods on five datasets, calculated from the worst 10% classes and averaged over crop lower bounds. The accuracies are listed in %. Higher is better.
CIFAR1O CIFAR100 FMnist
MImageNet ImageNet
Method
Worst Class
Test Accuracies (mean + std)
Normal (007
84.4=1.9
53.6+3.0
81.0#1.6
53.8+4.55
35.9+2.6
Focal (TCCV2017 7
82.4+1.1
55.1+2.44
80.5+1.4
52.
34.8+-2.1
PW (NeurIPS2022)
83.4+1.3
54.7=3.00
80.613.6
53.943.8
34.422.1
TCE (CVPR2021
84.3+144
53.1+3.7
80.2#3.4
52.8+3.2
36.2+2.5
GGF (ICMT 2020
85.1+119
55.5+13.1
80.0+2.0
47.74-3.3
36.64-2.2
CLAM(Ours)
85.8
56.412.3
82.5
54.813.7 36.742.9
which implies that the distribution of class accuracies more uniform. Results of color jittering data augmentation technique are included in Appendix M Additional results iNaturalist2018 and ImageNet with reassessed labels [Beyer et al., 2020 Kirichenko et al. 2023) are provided in Appendices N and O These results strongly support the assertion that our method learns a classifier that exhibits a more balanced distribution of accuracies across classes. Plots of fairness metrics along the training process are included in Figure 3 We can observe that our method consistently outperforms others with lower standard deviation and COV along the training process. Due to the page limit, plots of other datasets are included in Appendix J
Docs our method reduce the increase of class-dependent effects of data augmentation? In this part, we aim to quantify the effectiveness of our method in reducing the increase the class-dependent effects associated with data augmentation. We measure the increase of class-dependent effects by calculating the difference in accuracy ranges when data augmentation is applied versus when it is not. A smaller difference value indicates that data augmentation leads to a reduced accuracy range. A larger difference value suggests an increased accuracy range, implying a higher increase of class-dependent elfects of data augmentation, which we aim to avoid. Table illustrates that our method consistently yields the lowest difference values. This observation suggests that our method effectively reduces the increase of these effects induced by data augmentation Therefore, our approach ensures that the performance gains from data augmentation are not disproportionately influenced by the characteristics of certain classos, leading to a more balanced and reliable enhancement in accuracy across all classes.
Does our method improve the worst class performance? One overarching goal of our method is to ensure that the performances of the least accurate classes receive some enhancement. The accuracies of the worst 10% classes across five diverse datasets are shown in Table B We can observe consistent trend of improvement of the worst-class accuracies with method across all five datasets, indicating that our
11

---

Published in Transactions on Machine Learning Research (06/2025)
Clers Vesight your Mediad CHEAUEEE
Eaikeddss POCK DIYY MAST CIAMBLEE
5110
Figure 4: Evolution of class weights in CIFAR-10. From left to right: class weights, evaluation accuracy, evaluation accuracy gap during training, and final evaluation accuracy gap.
Table 4: The average test accuracies of different methods five datasets. We average the results over 8 crop lower bounds. The accuracies are listed in %. Higher is better.
CIFAR10 CIFAR100 FMnist MImageNet ImageNet
Method
Average Test Accuracies (mean = std)
Normal (007
92.8=0.6
75.2+2.0
94.110.55
74.412.6
70.3:2..6
Focal (ICCV2017
91.9=0.5
7..5+166
94.0+#.3
74.242.8
67.7+2.5
PW (NeurIPS2022)
92.410.6
75.241.8
93.611.
74.343.0
68.01:2.4
TCE (CVPR202I)
92.8+0.8
74.72244
98.7+0.9
73.742.2
70.2+2.3
GGF (ICML 2020)
93.0#1.2
75.3+1.8
94.2=0.4
70.612.1
70.0+2.2
CLAM(Ours) 92.6+10.9 75.7
94.0+1..4
74.5+12.3
69.6412.4
approach is effective in mitigating the disparity in class accuracies, resulting in more balanced classifiers. To visualize the enhancements achieved by our method, we provide plots of worst class accuracies in Appendix I
How does the weight vector evolve during training? We show the evolution of the weight vector during training. In the loss function we normalize the weights such that the sum equals rather than 1, a deliberate implementation choice that does not affect the order of weights. We present the changes of the weights for certain classes in the CIFAR-10 dalaset, as shown Figure | We can observe that the order of the weights alternates during the initial phase of training. Then they begin to converge around the 45th epoch, and subsequently the test accuracies converge at approximately the 60 epoch. Notably, the order of the converged weights is inversely related to the order of the converged test accuracies, which is desired. Plots of the other tasks are provided in Appendix K
Does our method significantly degrade the overall performance? We display the average test. accuracies in Table 4 In three out of the five datasets, our method results in slight decrease in overall accuracy compared to the baselines. However, this degradation is relatively limited compared to our improvement of worst class accuracies, suggesting that our approach manages to balance the contradictory demands of accuracy and equity. Remarkably, in the remaining two datasets, our method surpasses all baselines, underscoring its potential to enhance both class equity and overall accuracy simultaneously.
6 Conclusion
In this paper, focus the class-dependentt effects of data augmentation in classification. We propose CLAM, novel approach that reformulates classification a two-player game and adjusts class weights to achieve a balance between class equity and overall accuracy. CLAM is validated both empirically and theoretically. Importantly, our method is effective with very few hyperparameters. This simplicity makes it casily adaptable various other machine learning applications. For future work, could potentially be applied other scenarios, such as multi-task problems to achieve an equitable distribution of performance across various tasks.
12

---

Published in Transactions on Machine Learning Research (06/2025)
Acknowledgments
This work has been supported by the program of National Natural Science Foundation of China (No. 62176154) and by Shanghai Magnolia Funding Pujiang Program (No. 23PJJ1404400). We would also like to thank Yann Chevaleyre for insightful discussions.
References
Alekh Agarwal, Alina Beygelzimer, Miroslav Dudik, John Langford, and Hanna Wallach. A reductions approach to fair classification, 2018. URL https://arxiv org/ httg//bb//800.... .24553
Wael Alghamdi, Hsiang Hsu, Haewon Jeong, Hao Wang, P. Winston Michalak, Shahab Asoodeh, and
Flavio P. Calmon. Beyond adult and compas: Fairness in multi-class prediction, 2022. URL https
//axxiv.rrg/ab////. wtt.g///////.....//////////// 07801
Saurav Anchlia and YooJung Choi. A probabilistic approach to fairness under label bias. In The 6th Workshop on Tractable Probabilistic Modeling, 2023. URL
Randall Balestricro, Loon Bottou, and Yann LeCun. The effects of regularization and data augmen- (ation are class dependent. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Newral Information Processing Systems, volume 35, pp. (1878-3789)) Cur- Associates, Inc., 2022. URL Pan http://eww...ppoccoppppppeeeee/e (cccl/S866aaa000000566667777PP00
pdf
Lucas Beyer, Olivier Hénaff, Alexander Kolesnikor, Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet?, 2020. URL https://arxiv: http//aawwi..rggb///////////// 07159
Julian Bitterwolf, Alexander Meinke, Valentyn Boreiko, and Matthias Hein. Classifiers should do well even on their worst classes. In ICML 2022 Shift Happens Workshop, 2022. URL httphtttt/////eeeeeeee..ro net/
forum?id=0x 1o
Simon Caton and Christian Haas. Fairness machine learning: A survey. CoRR, abs/2010.04055: 2020.
URL https://arxiv..rg// abtp:/10.b000.0... 2000000000000
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale, 2021. URL httts:///axxi:
org/abs/2010 11929
Yoav Freund and Robert E. Schapire. Adaptive game playing using multiplicative weights. Games and Econmicc Behavior, 29:79-108 103, 1999. URL. https://ppp. semanticscholas org/CorpusID. 152956666
Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistionl Association, 102(477):359 -788 2007. doi: 10. 10.1198/011 0111880000214550000000033333 URL 10. http:///.....1110001110000000010
Nicklas Hansen, Hao Su, and Xiaolong Wang. Stabilizing deep q-learning with convnets and vision transformers under data augmentation. CoRR, abs/ /bs//2107000644, 2021. URL https://arxiv http:///axxvvv..aaaab/. htrg/abs/2107.00 00644
Hoda Heidari, Claudio Ferrari, Krishma P. Gummadi, and Andreas Krause. Fairness behind a veil of ignorance: A welfare analysis for automated decision making. In NewrIPS, 2018.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset, 2018. URL
httpa://arxiv.org/abba. 1111177700000 1667.
Sangwon Jung, Sanghyuk Chun, and Tacsup Moon. Learning fair classifters with partially annotated group labels. CoRR, abs/2111.1455 2021. URL https://arxxw..rr/abb//1111 14581
13

---

Published in Transactions on Machine Learning Research (06/2025)
Polina Kirichenko, Mark Ibrahim, Randall Balestriero, Diane Bouchacourt, Ramakrishna Vedantam, Hamed Firoaz, and Andrew Gordon Wilson. Understanding the detrimental class-level effects of data augmentation, 2023. URL https://arxiv.org/ wwtp://wrggbbb/.....oo 01764
Nya Kostrikov, Denís Yarats, and Rob Fergus. Tmage augmentation is all you need: Regularizing deep reinforcement learning from pixels. CoRR,, abs/ abs/2004.13649, 2020. URL
13649
Anilesh K. Krishnaswamy, Zhihao Jiang, Kangning Wang, Yu Cheng, and Kamesh Munagala. Fair for all: Best-effort fairness guarantees for classification, 2021. URL https://arxiv.orr http://ww..rrgaa///...... http//abb/212..21111
A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.
Teerath Kumar, Alessandra Mileo, Rob Bremnan, and Malika Bendechache. Image data augmentation approaches: A comprehensive survey and future directions, 2023.
Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2999 3009,007,, 2017. doi: 10.1109/ICCV.2017.2 2017.324.
Guozheng Ma, Zhen Wang, Zhecheng Yuan, Xueqian Wang, Bo Yuan, and Dacheng Tao. A comprehensive survey of data augmentation in visual reinforcement learning, 2022.
Natalia Martinez, Martin Bertran, and Guillermo Sapiro. Minimax pareto fairness: A multi objective perspective, 2020. URL https://arxiv org/ http/abb/22100000/ 018221
Robbie Meyer and Alexander Wong. fair loss function for network pruming. In Workshop on Trastworthy and Socially Responsible Machine Learning, NewrIPS 2022, 2022. URL 1d=Y@PmDhBdmv/
Wlodzimierz Ogryczak, Hanan Luss, Michal Pióro, Dritan Nace, and Artur Tomaszewski. Fair optimization and networks: A survey. Journal of Applied Mathematics, 2014, 2014.
Sungho Park, Jewook Lee, Pilhyeon Lee, Sunhee Hwang, Dohyung Kim, and Hyeran Byun Fair contrastive Icarning for facial attribute classification. In 2022 IEEE/CVF Conference on Computer Vision and Pattern WWWPPP5668820220001111 John Rawls. The Theory of Justice. Havard university press, 1971.
Recognition (CVPR), pp. 10379 10388, 2022. doi:
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211 2112222,, 2015.
doi:
Clayton Scott and Jianxin Zhang. Learning from label proportions: A mutual contamination framework, 2020. URL / wttp//b///000..0///333
Connor Shorten and Taghi M. Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of Big Data, 6:1 48, 2019. URL https://api. semanticscholar. wrwp:roooo///ooo110000 195811899
Umer Siddique, Paul Weng, and Matthicu Zimmer. Learning fair policies in multi-objective (Deop) re- inforcement learning with average and discounted rewards. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 8905 8915. PMLR, 13 18 Jul 2020. URL https://procedings..tt press//v11//siddiueeeaa..
Till Speicher, Hoda Hcidari, Nina Grgic-Hlaca, Krishna Gummadi, Adish Singla, Adrian Weller, and Muhammad Bilal Zafar. A unified approach quantifying algorithmic unfairness: Measuring individual & group unfairness via inequality indices. In KDD, pp. 2239 2248, 2018.
14

---

Published in Transactions on Machine Learning Research (06/2025)
Attila Szabó, Hadi Jamali Rad, and Siva-Datta Mannava. Tilted cross entropy (TCE): promoting fairness in semantic segmentation. CoRR, abs/2103.1405/ 2021. URL ftttp::/aaxxiv.orr/bbb///// 14051
Paul Weng. Fairness in reinforcement learning. In AI for Social Good Workshop IJCAI, 2019.
John A. Weymark. Generalized gini inequality indices. Mathematical Social Sciences, 144):409 430, 1981.
ISSN 0165-4896. doi: http:///bbooorg/1100000/00066660 900010 http://caaaaaattttieeeaa/////66e
190011-44 URL https:///ww..ccienddrrreet
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. ColR,, abs/1708.07747, 2017. URL
Depeng Xu, Shuhan Yuan, Lu Zhang, and Xintao Wu. Fairgan+: Achieving fair data generation and classification through generative adversarial nets. In 2019 IEEE International Conference Big Data (Big Dala), pp. 1401-1406, 2019. doi:
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved daga-eugee reinforcement learning. CoRR, 2bs/2107.09645, 2021. URL https://aaaas org/ arg/abs/2107.. 09645
15

---

Published in Transactions on Machine Learning Research (06/2025)
A Proof of Proposition 4.1
Let f(u)= min E; wyy,, If AS satisfies the following properly: Yw A Vs Sny w. E Ar, where Sn is the symmetric group of order i.e., set of all permutations of n elements and w, is reordering of the weight vector w based on 8, then f is
monotonic: v', v'= + f(v) S f(0'),
Schur-concave: Ve, > Vij Ve > 0,e Vjj f(v + cEj -ee) 2 f(Ø):
symmetric: f(v) = f(ve),
where Ej represents vector that is 1 at the jth element and 0 otherwise.
Proof. Firstly, it obvious that f is symmetric. f(v) = f(v,) because Ver E 4 Vs, ws E A.
Then, we prove that f monotonic. Let w' arg minl v'.
f(v') = min w v¹
wEA
w' o
2 w/ v
(10)
2 min w we WEA
= f(v)
The first inequality holds because v 2 v. The second inequality holds by definition.
Finally, we prove that f is Schur-concave. Let y = arg min w (v + eej ces). There are two scenarios depending on the value of 6.
I <e S {(u - Uz), then 0j + S -. By definition of y, yj 2 yh.
f(u + cej-
tei) min wEA
(u + cej - cei)
y (v cej- cei)
yi (v: t) + yj (vj 6) + Σ
yk UK
e (y) - yi) + Σ y/ Uk
(11)
Σ yk Uk
min w v WEA;
f(v)
< < (v) -1) If
v, uy we introduce an intermediate variable U1 Uy -e, satisfying 0 ( +e 14 and 12 - ny Since is symmetric,
f(v ces - cet) f(v - -ée
tee----e - des)
(12)
2 f(v)
Therefore, ʃ(v + cej- -ee)) f(0).
16

---

Published in Transactions on Machine Learning Research (06/2025)
B Proof of Theorem
The average loss of the min player's strategy compared to the optimal fixed strategy has an upper bound if we set T as In(1 + I+maxa,
lan T
T
In
w ve I min
Ve.u vt +
T wEAS
T
1=1
(13)
Inn
+ (1 + max I1a (4)
T
where T is the number of total epochs, v can be any arbitrary weight vector, w is the weight vector chosen by our method at epoch ot is the accuracy vector for all classes at epoch t, Vw. -E=1 t0y2 is the weighted accuracy of choosing when the accuracy vector is and can also be regarded as the loss for the
min player, at En aaot Eneso Naet
x++1
and =(x(+))
Vo.
Σ, af
Proof. We introduce an intermediate variable xt+1 follows.
Then our update rule can be rewritten as
Σ, u
ut t wt xe r.
(14)
w1+1 +- x(x*+1).
We begin the proof by analyzing KL ( w - K (K((u (i (wn), where w is any arbitrary weight vector.
K [ - K L T8 ww0)
= =[KT(ui]|l0 w K ( (
(15)
+ [KL(w)lx - K (L(wi|l'
We firstly prove that for any w. KL KL(z0ll0'+)
K L( has upper bound.
Let
o T(x
x t+I
(16)
Eneso Das!
Ex taut
04
E, t.
V
KT(Wollu 100 +11
K
i=n
1+1
x
W In
w
x 1+1
(17)
Wi In
(+++
1=1
Wiro = VV..v
The inequality holds because In 5(x 241
for e = and (xx max(o: (Bmin)
Eauin Za
> Emin e Tiof
x.
Z,
z. 1+1 e --
for of > O, where Z, Di=n inn
max(x ++1 .miin)
Σ ii-n
max (2, .mmmn)
Endloo max(u ++ (raun)
141 Dis
iet #!
E Eneoso Tann
Engos Ammuse min(ut) S e-mi((()))
17

---

Published in Transactions on Machine Learning Research (06/2025)
We then prove that for any w. (L(W]a - KL(xil/w" Vw,2 Ei ways and 2, Σ, ut.
has upper bound. For simplicity, define
KL(W0|a
(0
K
KL( (LL(2t) (w0l(=))
i=n
w
Σ
ü; In
Ze
Wg In(
e-r-0,
i=n
Σ
ii 21
In Z + VV
(18)
k=
wi e T. + -V VV
< hn
Σ
w/ (1 (1-6- + +VV..
i=1
In
1- (1 E 7 Σ wi 2/ VVo.v
In(1 - (1 -- Vot, + VV
S (1 - - VoV'' + VV
The first inequality holds because e -- <= <= -1 for + and 0 <= <=<
<= 1. The second
inequality holds because In(1 (1 - e Ver,,, S
for <=
<=<==111 (1 (1 V <
Sum eq. 17 and eq. 18] together, we get the upper bound of (( KL(Wūllw*)
KL (L(WiIl=++1 KL(W\ll") $ -11 CTVVe (1 av u
(19)
Sum eq. 19 from t = 1 to t = T, we get the upper bound of KL(w ((bbl(wT+)) K KL( (w')
KL w T+1) KL(Wūl
t=T
(20)
(-(1 - c Voo' of + (1+a))Van.))
From eq- 20 we get the upper bound of E|= Vet Note that since fi for all i, K KL( /
K T.(z lw0 Inn for any Here, we set 7 as In(1 + maxos
18

---

Published in Transactions on Machine Learning Research (06/2025)
T=T
Σ Vwn ut S --e-r
1
(KL(T - KL(ūollo*+1
l=
+ + + +1))
V Vo,u))
t=1
5 1
Inn
+ (1 maxcx)
7
Vū, ut
1
1 e-
1
$ (1 +
lun
67
1 1
+ (1 maxas)
V
(21)
2e
(1 + e ynn
c
t=
+ (1 + (-mxaa))
Σ
2
T
+ (1 maxar)
In
Inn
+ (1 + maxaz)
Inn m T
1 2(1 + maxa:)
Vin
Dividing both sides by T, we obtain the upper bound of Ei=T Vwv pt.
T=T T
Inn T
lnn
T
Vwn
S
+ (1 + max az)
V
T
max Ot
1
In n
+
2 V
Vin bt
T
T3
+ max Or
Inn
+
T
2
73
t=
(22)
min
V VaL
WEA
[=1
+ Inn
Inn
+ (1 + max ar)
T
T
1 Inn
l=T
+ T3
min wEA:
Va
In
Inn
+ T + (1 + 11=maaa)-
T
EE Vur represents the average performance of our strategy while mingea; 1 EE= Ve, represents the best performance we can achieve by any fixed t that is in the constrained simplex. The above inequality
19

---

Published in Transactions on Machine Learning Research (06/2025)
guarantees that the average loss of our strategy compared to this optimal to is approximately bounded by In. + (1 maxa:))
/ In T which can be minimized when T gets larger.
]
C Lemma for the Loss at the Last Iteration
We show that if the weights w generated in Algorithm converge, then the one-step value V e² also converges to the average value of the best weight. This result justifies only considering the last weight given by Algorithm I
Lemma 1. VI > 0, 3T, Vl,, 12 > To, 1 - -w || e, then the loss of the mén player's s stralegy converyes to the average loss of an optimal fixed strategy:
T
lim VTT lim
min wEAg
Σ
Vi
(23)
T-000
T-00
Proof-
T
t=
V wwt
V v u
T
t=T
Vw wt +
V Ywntut
To
l=
Σ
Vwe
+
w²
-0))
T
E=
To
1=2
t=
T
7
Σ
Vwn
+
w
(w w T) r²)
t=Ta
(=Tb
(24)
lim T+x
Vue ve
lim
V
w
T
(w w .0)
T+000
T
t=T
(=Tu
t=T
t= Tp
t=
lim
w T
lim
Vut
Σ (w - -w²) -0))
T
T-x
T
t=To
1=1
t=
t=
T=T
lim T-200
w
v
lim
Vi
(w" - w²) x²)
T-000
T
T=
lim w v- +0= = lim
wT DT
lim
V .
T-000
T0000
T-000
Therefore, as T' approaches xo, the loss at the last epoch w" oT is equal to the average loss $ - Then using the upper bound derived in Theorem we can conclude that the final loss w" v also has an upper bound as T approaches cc.
-==1 Vwt
l=
lim Vur .T lim T-000
V v
T-000
T
t=
In
Inn
lim
min
Va +
T + (1 + max 0n) .
(25)
T-x T
wEA:
T- T
lim T-
min
Σ
Va w
T
BE A
t=1
20

---

Published in Transactions on Machine Learning Research (06/2025)
Since lim f min Ei=l Va,vi is the lower bound for limy- Vur u2
lim Vo o²
lim
min WEA
Va
(26)
7-000
700
Then from Equation 25 and Equation (6 we oblain
lim T-000
Vor N²
lim T000
V Vi
(27)
T WEAS
Comparison of our Method and APStar
Can fair classification method developed for group fairness be applied to reduce the class- dependent effects of data augmentation? Most. fair ML methods do not apply to our problem, but some could potentially be adapted (e.g., APStar (Martinez et al., 2020)) As such, investigate how it performs here compared to proposition. They achieve fairness by adding vector with ones for the worst groups to the previous weight vector. Adapting their method involves replacing groups by classes. In our setting, some classes inherently exhibit lower accuracy due to the increased difficulty in classifying them, and APstar tends to over-emphasize these difficult classes while neglecting others, impeding convergence during training. Additionally, as the number of classes grows, there is greater chance that marginally better classes will be weigr-we and consequently ignored, which further hurts training. As shown in Figure 5 our experiments show that the APStar's weight vector has difficulty converging with many classos. Comparing our method with APStar on the CIFAR-100 dataset, the weight vector and test accuracies of APStar show less stability. We conclude that our method is more effective than APStar as the number of classes grows.
Cisa Welights Dur Mathect in CIAP-100)
Euskation Ascurace Our Methed CTASACO0)
Weights Jazder cieer000
Accurncy Iepaber Efer100)
CR
Figure 5: Comparison of class weights and test accuracies between method and APStar CIFAR-100 dataset. These results show that the APStar's weight vector has difficulty converging with large number of classes.
E
Detailed Descriptions and Hyperparameters of Baselines
cross entropy loss The first bascline is normal cross entropy loss.
CE(pt) = In(p:)
(28)
focal loss This method adds a factor to the standard cross entropy loss.
Fl(p)) = In(p:) * (1 p()
(29)
where Pt is the model's estimated probability for the class with label y and y is hyperparameter.
21

---

Published in Transactions on Machine Learning Research (06/2025)
tilted cross entropy loss This method assigns weight to different classes. The weight of each class is obtained from the previous training loss of this class by the following equation.
C'((p)) = =- -In(p)) * wi
cLe(e)
(30)
we(c) (1 (()) + +2 Ei el((1)
where (i) is the weight assigned to class i and (t( is the training loss of class i. wo(() is initialized as 1 for each class
performance weighted loss (PW Loss) cross entropy loss.
Similar to focal loss, this method adds a factor to the standard
FL(p)) In(p:) * ((1 pc) + 0)
(31)
where 0 is an additional hyperparameter that makes this loss more flexible than focal loss
GGF enhanced cross entropy loss This approach utilizes the GGF loss at frequency specified by f. When f = 1, the GGF loss is consistently applied. For f = 2, the model alternates between applying the standard loss and the GGF loss. The weights assigned to different classes are determined based on the ranking of their training accuracies of the last epoch. The weight for class i at epoch t can be written as:
w max(a Rank(nj 1)-1 Wmin]]
(32)
where v/-1 is the training accuracy of class at epoch - 1, a and Wiin are two hyperparameters.
Table 5: Hyperparameters of Baselines
Focal Loss
CIFAR-10
CIFAR-100
Fashion-Mnist
Mini- Mini-emagenet
Imagenet
Y
2.0
2.0
2.0
2.0
2.0
TCE Loss
CIFAR-10 10
C1FAR-100
Fashion-Mnin
Mini- Aini-Imagenet
Imagenet
0.5
0.5
0.5
0.5
0.5
PW Loss
CIFAR-10
CIFAR-100
Mishion-Mnis
Mini- Mini-Imaeene
Imagenet
2.5
2.5
2.5
2.5
2.5
0.8
0.8
0.8
0.8
0.8
GGF Loss
CIFAR-10
CIFAR-100
Fashion- Mnist
Mini- Imageemteent
Imagenet
O
0.9
0.98
0.98
0.95
0.998
Wmin
f 0.1
0.1
0.1
0.01
0.2
2
F Preprocessing of Datasets
For CIFAR-10, CIFAR-100 and Fashion-Mnist datasets, we use the built-in datasets from the library of lorchvision. dalassts. For the Mini-Imagenet and Imagenet datasets, we retrieve them from their respective websites and utilize the predefined partitions for training and evaluation.
G Hyperparameters of Our Method
There are only two hyperparamoters our method, the parameter of multiplicative weight update and the lower bound of class weights Wmin- The simplicity in the number of hyperparameters not only eases the implementation of our method but also reduces the complexity of hyperparameter tuning to achieve optimal
22

---

Published in Transactions on Machine Learning Research (06/2025)
performance. Among all five datasets, do very little tuning and set and umin 1 where n is the number of classes in that specific dataset.
Note that in our theoretical proof appendix B we set T to be In(1+ + 14maxor
Ian
However, it is important
to mention that the robustness of our approach is not solely contingent upon this exact value. In practice, have observed that even when T is set to 1, a commonly used default value in experiments, our method still demonstrates strong performance. The consistency of the performance across five datasets underscores our method's 's resilience and adaptability, indicating that the benefits of our approach are not exclusively tied to the precise tuning of hyperparameters. Results of our method with different. т values are provided in Appendix
H Mathematical Definition of Evaluation Metrics
Mathematically, we define the range rt, standard deviation or and the coefficient of variation e of class accuracies i, at epoch follows.
max o/ min r
14
1=1
(33)
Σ 10 - (u) - -mee)
C
Plots of Worst Class Accuracies
For an in-depth analysis of the performance improvement in the worst class accuracies, detailed plots are shown in fig. 6 fig. 7 fig. S fig. 9 and fig. 10 for five datasets. In each of the five figures, the x-axis represents the th worst class, while the y-axis denotes the evaluation accuracy of that particular class. This arrangement allows for clear visualization of the effectiveness of our method in improving the test accuracy of these worst classes within each dataset. By examining the figures, we can observe how our approach consistently outperforms the other baseline methods across different datasets and almost all worst classes, highlighting its robustness and effectiveness in improving the worst class performance.
Analysis of Worst Class Performance Specifically, we invest whether CLAM identifies the same worst class as other methods and examined the training and testing errors for these classes. We observe that in CIFAR-10 all methods identify "cat" and "dog" as the worst classes while in CIFAR-100 "boy", 'girl', "man", 'woman', 'seal', and 'otter' are identified as the worst classes. Notably, while the training accuracy for these classes always 100%, indicating perfect fitting to the training data, CLAMM improves the testing accuracy for these worst classes, thereby reducing the gap between training and testing performance and highlighting CLAM's effectiveness in reducing class-specific generalization gap.
J Plots of Fairness Metrics during Training
As shown in fig. II fig, in CIFAR-10, 100, Fashion- Aashion-Mnist, Mini-ImageNet and ImageNet datascts. From the plots, we can observe that our method consistently outperform other methods with the lowest standard deviation of class accuracies and the lowest coefficient of variation (COV).
12 fig.
13 fig. and fig. we include plots fairness metrics during training
23

---

Published in Transactions on Machine Learning Research (06/2025)
worst class acc in CIFAR-10
worst class acc in Fashion-Mnist
normal
normal
focal
focal
90
pw
tce
tce
86
GGF
GGF
CLAM(ours)
CLAM(ours)
88
84
86
82
eval acc
eval
80
Figure 6: Worst class accuracies of different methods Figure 7: Worst class accuracies of different methods in CIFAR-10 dataset.
in Fashion-Mnist dataset.
24

---

Published in Transactions on Machine Learning Research (06/2025)
worst class acc CIFAR-100
CLAM/ourt]
15
034 022
woret classes
Figure 8: Worst class accuracies of different methods in CIFAR-100 dataset.
worst dasss acc in Mini-mageNet
local
CLAMiouss
50
40
30
went clases
Figure 9: Worst class accuracies of different methods Mini-Imagenet dataset..
worst dass acc n mageNet
FRrrmal local
BM tce
0301 665
15
Figure 10: Worst class accuracios of different methods in ImageNet datasct.
25

---

Published in Transactions on Machine Learning Research (06/2025)
Rindd
D0F CLAR
CLAMOuRS
2
233
150h
Dach
Figure 12: Fairness metrics during training in CIFAR-
Figure 11: Fairness metrics during training in CIFAR- 10 dataset.
100 dataset.
durin
durliy by st
Portal
rormal
10
Figure 13: Fairness metrics during training in Fashion- Mnist dataset.
Figure 14: Fairness metrics during training in Mini- ImageNet dataset.
std during training in ImageNet
COV during training in Imagenet
28
0.7
26
coefiiciont variation
24
22
normal
normal
focal
focal
20
0.4
PW
pw
18
GGF
GGF
CLAM(aurs)
x
CLAM(ours)
30
40
50
60
70
80
30
50
60
70
80
Spoch
Epoch
Figure 15: Fairness metrics during training in ImageNet dataset.
26

---

Published in Transactions on Machine Learning Research (06/2025)
K
Plots of Class Weights Evolution during Training
Here we present the plots depicting the evolution of class weights during training for the CIFAR-100, Fashion- Mnist and Mini-Imagenet datasets in Figure 17 Figure 18 and Figure 19 From these figures, trends similar to those observed in Figure 16 for CIFAR-10 dataset. Initially, the order of weights fluctuates during training but eventually converges at certain point, after which the evaluation accuracies also converge. Furthermore, the order of the converged weights is inversely related to the order of the converged evaluation accuracies.
discern
Class Weights (Our Method in CIFAR-10)
Evaluation Accuracy (Our Method in CIFAR-10)
100
class1
class2
class
class4
80
1.6
dasss
class weight
eval acc in %
60
1.2
1.0
class1
class2
dlass3
0.6
class4
60
20
Epoch
Epoch
Evaluation Accuracy (Ours Normal in CIFAR-10)
Final Evaluation Accuracy Gap in CIFAR-10
ours-normal
dlass2
2.5
dasss
dlass4
eval acc (ours norral) in %
2.0
20
class5
1.5
eval acc in
1.0
-20
440
-0.5
-1.0
20
40
60
80
100
5
Epoch
daass
Figure 16: Evolution of class weights CIFAR-10 dataset. Top Left: class weights; Top Right: evaluation accuracies of certain classes; Bottom Left: evaluation accuracies of certain classes (ours baseline); Bottom Right: Final evaluation accuracy gap of certain classes (ours baseline).
27

---

Published in Transactions on Machine Learning Research (06/2025)
Class Weights (Our Method in CIFAR-100)
Evaluation Accuracy (Our Method in CIFAR-100)
classl
1.8
80
class4
1.6
class
class weight
60
eval acc
class1
0.8
class4
class5
25
50
100
125
150
25
50
75
100
125
150
Epoch
Epoch
Evaluation Accuracy (Ours Normal CIFAR-100)
Final Evaluation Accuracy Gap CIFAR-100
40
classl
ours-normal
class2
30
eval acc (ours. normal) in %
20
class
eval acc in %
10
-10
-1
-2
-2
-3
-40
100
125
Epoch
dasss
Figure 17: Evolution of class weights in CIFAR-100 dataset. Top Left: class weights; Top Right: evaluation accuracies of certain classes; Bottom Left: evaluation accuracies of certain classes (ours baseline); Bottom Right: Final evaluation accuracy gap of certain classes (ours baseline).
28

---

Published in Transactions on Machine Learning Research (06/2025)
Class Weights (Our Method FMnist)
Evaluation Accuracy (Our Method in FMnist)
classl
class3
class4
80
class5
1.6
cass weight
eval acc in %
60
1.0
class1
class4
class5
0.4
20
30
40
50
10
20
30
Epoch
Epoch
Evaluation Accuracy (Ours Normal FMnist)
Final Evaluation Accuracy Gap in Fmnist
100
classl
ours-normal
class2
75
class3
class5
3
eval acc (ours. normal)
eval acc in %
25
2
1
-25
-75
-1
50
Epoch
dasss
Figure 18: Evolution of class weights in Fashion-Mnist dataset. Top Left: class weights; Top Right: evaluation accuracies of certain classes; Bottom Left: evaluation accuracies of certain classes (ours baseline); Bottom Right: Final evaluation accuracy gap certain classes (ours baseline).
29

---

Published in Transactions on Machine Learning Research (06/2025)
Class Weights (Our Method in Mimagenet)
Evaluation Accuracy (Ours in Mimagenet)
classl
www mmm
class2
dlass
80
class4
1.6
class
class weight
1.4
eval acc
50
class1
0.8
20
class4
class5
20
40
60
80
100
20
40
80
100
Epoch
Epoch
Evaluation Accuracy (Ours Normal Mimagenet)
Final Evaluation Accuracy Gap Mimagenet
classl
ours-normal
30
class2
class3
eval acc (ours. normall in %
20
class
eval acc in %
10
2
-10
-20
-30
60
Epoch
dlass
Figure 19: Evolution of class weights in Mini-Imagener dataset.
Top Left: class weights; Top Right: evaluation accuracies of certain classes; Bottom Left: evaluation accuracies of certain classes (ours baseline); Bottom Right: Final evaluation accuracy gap certain classes (ours baseline).
L Additional Results for CLAM with Different Values of
We present additional results for CLAM CIFAR-100 and Fashion-Mnist with varying 5 values in Table b including the bascline (r 0) for reference. Across both datasets, CLAM consistently improves class equity in terms of "std', "COV", 'range', and 'worst acc'. Notably, optimal performance is achieved at r= 0.1 or == 0.5, aligning with theoretical values derived in Theorem [ For CIFAR-100 (100 classes, 150 training epochs), the theoretically optimal r z 0.15, and for Fashion-Mmist (10 classes, 50 training epochs), the theoretically optimal T R 0.18. The approximations come from assuming maxa, s 0.1 both cases.
30

---

Published in Transactions on Machine Learning Research (06/2025)
Table 6: We present the fairness metrics of CLAM with different values on CIFAR-100 dataset in the following table. In the columns, "std", 'COV', 'range', 'avg acc', and 'worst acc' respectively denote the standard deviation, coefficient of variation, range of class accuracies, average accuracy of all classes, and average accuracy of worst 10% classes test datascts. For each method used each datasct, the reported result is averaged over 8 crop lower bounds. The std, range, and accuracies are listed in % while COV is listed in values. Lower is better for "std", "range", and "COV". Higher is better for accuracies.
CIFAR-100
Fashion-Maist
|st
COV
range avg acc
worst acc |std
COV
range avg ace worst acc
11.8
0.157
39.1
75.2
53.6
5.94
0.063
18.3
94.1
81.0
(Normal)
0.1
10.8
0.112
35..
75.8
56.0
5..17
0.055
14.8
194.1
84.7
0.5
10.6
0.140
35.3
75.5
56.1
5.11
0.055
15.1
94.1
84.4
1.0
10.8
0.143
35.9
775.7
56.4
5..44
0.058
16.7
911.
82.5
2.0
11.3
0.151
37.3
75.1
54.5
5.30
0.055
15.4
99399
83.7
10.0
|| 11.3
0.151
37.3
775.
54.9
|5.50
0.060
17.1
922.8
81.7
Table 7: We present the fairness metrics of different methods with color jitter as data augmentation four datascts in the following table. In the sccond column of the table, 'std", "COV", and 'range" respectively denote the standard deviation, coefficient of variation, and range of class accuracies on test datasets. For each method used on each dataset, the reported result is averaged over 8 color jitter parameters. The std and range are listed in % while COV is listed in values. Lower is better.
CIFAR-10
CIFAR-100
Fashion-Mnist
Mini-ImageNet
Method std COV
range/std
COV
range std
COV
range std
COV
range
Normal 8.95
0.106 28.7
16.89
0.273
57.1
15.8
0.238
48.7
17.4
0.312
58.2
Focal
8.66 0.102 26.4
15:27
0.242
51.3
15.7
0.218
46.9
17.3 0.326
58.5
GGF
8.88 0.104
25.9
17.72
0.308
59.0
15.9
0.225
47.4
22.0
0.492
73.5
CLAM 3.86 0.042
24.3
15.06
0.236 50.1
15.5
0.214 44.7
16.3
0.290 54.1
(Ours)
M Additional Results with Color Jitter as Data Augmentation
We present additional results on CIFAR-10, CIFAR-100, Fashion-MNIST, and Mini- Imag-em datasets with color jitter as a data augmentation method in Table Our method consistently achieves the best class equity across all datasets, demonstrating its robustness diverse augmentation strategies.
N Additional Results on iNaturalist2018
Here we include additional results on 22018, a class-imbalanced dataset, as shown in Table g As shown in the table, "Focal" and 'GGF" reduce accuracy standard deviation and range but sacrifice significant average accuracy. In contrast, our method achieves lower coefficient of variation, a more advanced and critical metric for class equity, while maintaining better average, worst-class, and best-class accuracies, demonstrating stronger robustness and balance across classes.
31

---

Published in Transactions on Machine Learning Research (06/2025)
Table We present the fairness metrics of different methods with random resized cropping as data augmentation on 1Naturalis22118 in the following table. In the second column of the table, 'std", "COV", 'range", 'vv acc', 'worst acc', and "best acc' respectively denote the standard deviation, coefficient of variation, range of class accuracies, average accuracy of all classes, average accuracy of worst 10% classes, and average accuracy of best 10% classes test datasets. For each method used on each dataset, the reported result is averaged over 8 crop lower bounds. The std, range, and accuracies are listed in % while COV is listed in values. Lower is better for "std", "range", and "COV". Higher is better for accuracies.
INaturalist2018
Method
std
COV
range
avg acc worst acc best
Normal
30.5
0.843
50.7
42.1
11..1
61.9
Focal
29.3
0.933
47.9
37.9
|8.3
56.2
GCF
29.2
0.875
18.1
40.2
9.3
57.5
CLAM
30.3
0.813
50.2
42.9
12.5
62.7
(Ours)
Table 9: We present the fairness metrics of different methods on ImageNet with reassessed labels in the following table. In the second column of the table, 'std", 'COV" 'range', 'worst acc', and "best acc' respectively denote the standard deviation, coefficient of variation, range of class accuracies, and average accuracy of worst 10% classes on test datasets. For each method used on each dataset, the reported result is averagod over crop lower bounds. The std, range, and accuracies are listed in % while COV is listed in values. Lower is better for "std", "range", and "COV". Higher is better for accuracies.
ImageNet with ReaL labels
Method std
COV
range
worst acc
Normal
13.6
0.179
45.9
50.0
CLAM
13.3
0.176
44.9
50.2
(Ours)
0 Additional Results Imagenet with ReaL (Beyer et al., 2020; Kirichenko et al. 2023)
Here we include additional results on Imagenet dataset with reassessed labels, as shown in Table 9 From the table, we can observe that our method continues to improve class equity on ImageNet with reassessed labels.
P Compute Resources We Use
In all our experiments, utilize GPU server equipped with 8 cards that have either RTX-4090 A6000 GPUs and are powered by AMD EPYC 7763 CPUs. For the CIFAR-10 dataset experiments, training resnet-50 model for 100 epochs on single GPU card takes approximately 3 hours. When working with the CIFAR-100 dataset, à resnet-50 model trained for 150 epochs on one GPU card requires about 4.5 hours. On the Rishion-MNISTT dataset, a resnet-50 model trained for 50 epochs on one GPU card completes less than hours. For the Mini- Imagenette dataset, an experiment involving resnet-50 model trained for 150 epochs on one GPU card consumes around 4.5 hours. Lastly, for the Imagenet dataset, training a resnet-50 model for 88 epochs GPU card demands roughly 2 days of computational time.
32

---

Published in Transactions on Machine Learning Research (06/2025)
Q
Limitations
While our approach has yielded commendable improvements in fairness metrics, it is essential to acknowledge a limitation that emerged from our experiments. We observe small decrease in overall test accuracy across out of the datasets we examined, as discussed in Table 4 This trade-off between fairness and accuracy presents a limitation our work, as enhancing fairness may sometimes come at the expense of the overall model performance.
R Broader Impacts
Positive societal impacts By proposing and validaling a novel method that balances accuracy with fairness, we contribute the development of a more equitable machine learning system. This kind of system have the potential to positively affect various domains, such healtheare and education, where unbiased decisions can lead to more just outcomes for all individuals.
Negative socictal impacts To the best of our knowledge, we don't see any negative socictal impacts of our work.
33


Reviews:
[{"review_id": "w8kdzLEAHY", "reviewer": "Reviewer_Fbdn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper addresses an important and understudied issue of class-dependent bias introduced by data augmentation. The approach offers a practical solution to ensure equitable performance across classes.\n- A robust theoretical justification, including proof of convergence for the multiplicative weight algorithm, is provided.\n- Extensive experimentation across multiple datasets and comparison with relevant baselines demonstrate consistent improvements in fairness metrics.\n- The proposed method is easy to implement, with only two hyperparameters, making it practical for real-world applications.\n- The paper is well-written, clear, and logically organized, facilitating understanding.\nWeaknesses\n- A slight degradation in overall accuracy observed in some datasets needs more explicit analysis and potential remedies.\n- While simplicity is a strength, the impact of hyperparameter tuning is not extensively analyzed, potentially limiting generalizability.\n- The iterative updating of weights could introduce computational overhead, which is not thoroughly discussed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents CLAM (CLAss-dependent Multiplicative-weights), a novel approach to mitigating the class-dependent effects of data augmentation in multi-class classification. It formulates the classification problem as a fair optimization challenge through an adversarial two-player game. The min player assigns weights to classes emphasizing fairness, while the max player optimizes the weighted accuracy. The authors introduce an adapted multiplicative weight algorithm to dynamically balance overall accuracy and class fairness. Theoretical analysis demonstrates the convergence of the proposed algorithm, and extensive empirical evaluations across five datasets confirm its effectiveness in improving worst-class performance with minimal impact on overall accuracy.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Further Analysis of Accuracy-Fairness Trade-off: Provide deeper insights or a theoretical exploration explaining under which conditions the trade-off between fairness and overall accuracy occurs. This would strengthen the theoretical foundation and provide practical guidelines for applying CLAM effectively.\n- Hyperparameter Sensitivity Study: Conduct a detailed analysis of hyperparameter sensitivity. While simplicity is beneficial, understanding the robustness of the approach to hyperparameter choices is crucial for practical adoption.", "broader_impact_concerns": "The broader impact statement provided is clear and adequate, highlighting positive societal impacts through fairer decision-making systems and not identifying negative impacts. However, to strengthen it further, the authors could explicitly discuss scenarios where class-level fairness might inadvertently lead to suboptimal outcomes in critical applications, advising caution when deploying fairness-aware methods without domain-specific considerations.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2N68hQVGqP", "reviewer": "Reviewer_8JWK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n\n1. Paper is well written and easy to follow.\n2. Detailed theoretical analysis are provided.\n3. Experimental results look good, justifies the effectiveness of the model\n\nCons:\n\n1. My major concern is the motivation of the problem. In short, it is not straightforward to me that the problem of Class-Dependent Effects is essential in practical deployment of Neural Network models. And from my reading experience of this paper, the importance of the Class-Dependent Effects is not emphasized (only the autonomous driving example in the intro is not sufficient ). I would recommend the author(s) provide stronger examples that the Class-Dependent Effects are crucial.\n\n2. The effectiveness of the proposed method may be limited. This can be seen In Fig.2, where the per-class accuracy of the proposed method and the corresponding weight of loss seems converge at around 60 epoch. And the final per-class accuracy still some what differs, implying that the proposed method is mitigating (instead of solving) the problem of Class-Dependent Effects . I would recommend to plot the same metric without using the proposed methods in contrast to demonstrate the effectiveness of the model. And it would be better if Fig.2 for larger dataset can be provided, as CIFAR10 is a really small dataset.\n\n3. In the experiment section, it seems that only a particular data augmentation technique -- cropping, is tested. What about other data augmentation methods? As most of the previous sections argues that the proposed method is for Class-Dependent Effects when using data augmentation methods, I would expect at least two common types of data augmentation methods are tested.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes a min-max optimization approach to tackle the problem of class-dependent accuracy of the Neural Network classifiers trained with data augmentation methods. Theoretical analysis is provided to justify the validity of the model and empirical evidences are provided to justify the effectiveness of the model.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see weakness.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3dcLfV2Onb", "reviewer": "Reviewer_mh54", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. CLAM is easy to implement, requires few hyperparameters, and is architecture-agnostic.\n\n2. The authors provide a theoretical guarantee for CLAM. \n\nWeaknesses: \n\n1. My primary concern is that this paper mentioned the effect of data augmentation on class fairness, but the proposed method is reweighting each sample in a two-player game way. Could the authors explain more about how it is related to data augmentation? and explain why it can solve the bias between the accuracies among different classes? For example, as mentioned in the first paragraph of the Introduction, data augmentation for image classification includes random shifts and random clips. Why this semantic-preserving transformation can induce class bias? Why CLAM can alleviate it?\n\n\n2. Based on the first disadvantage, if I understand correctly, I do not think the authors involve the effect of data augmentation in the theoretical analysis of CLAM. Usually we consider data augmentation as a label-invariant function. Please refer to [1].\n\n\n3. The datasets utilized in the experiments are all class-balanced datasets, but class bias occurs more significantly via class-imbalanced datasets (e.g., iNaturalist[2]). I suggest authors add more experiments on imbalanced datasets. \n\n4. The baselines shown in Tables 2/3/4 are quite old (2017-2022). Can the authors compare CLAM with more recent baselines for class fairness?\n\n5. In Table 3, the performance gain for the worst-case accuracy is quite limited on CIFAR10. I suggest the authors conduct more analysis on that. For example, does CLAM have the same worst class as other methods? What is the worst class actually? As the basic idea for CLAM is to assign more weight to the worst-case class, what is the training error for the worst class? By comparing the training error and testing error for the worst class, the authors can identify whether the error is brought by insufficient optimization or the generalization gap. \n\n[1] Yang S, Dong Y, Ward R, et al. Sample efficiency of data augmentation consistency regularization[C]//International Conference on Artificial Intelligence and Statistics. PMLR, 2023: 3825-3853.\n\n[2] https://paperswithcode.com/dataset/inaturalist", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the issue of inherent class fairness introduced by data augmentation. The authors propose CLAM (CLAss-dependent Multiplicative-weights), a novel framework as an adversarial two-player game, where the max player (Eq 3) aims to maximize the overall performance across different classes, and the min player (Eq 4) aims to reweight more on the worst-case class. The authors theoretically prove the convergence of CLAM under constrained simplex settings and empirically validate their method on five datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to my comments in the section of disadvantages.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)