Learn2Mix: Training Neural Networks Using Adaptive Data Integration
Authors: Shyam Venkatasubramanian
Vahid Tarokh
Venue: NeurIPS 2025 poster ()
Keywords: adaptive training
deep learning
optimization
Primary area: deep_learning
TL;DR: This work introduces learn2mix, a new training strategy that adaptively adjusts class proportions in batches to accelerate neural network convergence in resource-constrained environments.

Abstract: Accelerating model convergence in resource-constrained environments is essential for fast and efficient neural network training. This work presents learn2mix, a new training strategy that adaptively adjusts class proportions within batches, focusing on classes with higher error rates. Unlike classical training methods that use static class proportions, learn2mix continually adapts class proportions during training, leading to faster convergence. Empirical evaluations on benchmark datasets show that neural networks trained with learn2mix converge faster than those trained with existing approaches, achieving improved results for classification, regression, and reconstruction tasks under limited training resources and with imbalanced classes. Our empirical findings are supported by theoretical analysis.

Learn2Mix: Training Neural Networks Using Adaptive Data Integration
Shyam Venkatasubramanian
Vahid Tarokh Duke University vahid. tarokhčduke edu
Duke University shyam www.:eeeeaaaaaaaannaaaa anduuee edu
Abstract
Accelerating model convergence in resource-constrained environments is essential for fast and efficient neural network training. This work presents learn2mix, new training strategy that adaptively adjusts class proportions within batches, focusing classes with higher eπrоr rates. Unlike classical training methods that use static class proportions, learn2mix continually adapts class proportions during training, leading faster convergence. Empirical evaluations that neural networks trained with learn2mix converge faster than those trained with existing approaches. achieving improved results for classification, regression, and reconstruction tasks under limited training resources and with imbalanced classes. Our empirical findings supported by theoretical analysis.
benchmark datasets show
Introduction
Deep neural networks have become essential tools across various applications machine learning,
including computer vision Krizhevsky et al., 2012 Simonyan and Zisserman, 2014 He et al.) 2016)]
natural language processing |Vaswani et al. [017 Devlin et al. [0018] Radford et al. [0199] Touvron
al. 2023 and speech recognition [Hinton et al. 2012 Baevski et al. [2020]. Despite their ability to learn and model complex, nonlinear relationships, deep neural networks often require substantial computational resources during training. In resource-constrriind environments, this demand poses significant challenge [Goyal et al. 2017). making the development of efficient and scalable training methodologies increasingly crucial fully leverage the capabilities of these models.
Training deep neural networks relies on the notion of empirical risk minimization (Vapnik and Bottou
1993 and typically involves optimizing loss function using gradient-based algorithms [Rumelhart Bottou| 2010 Kingma and Ba 2014). Techniques such as regularization [Srivastava 2014 loffe and Szegedy 2015 and data augmentation [Shorten and Khoshgoftaar] [019]] learning rate scheduling, (Smith 2017] and early stopping [Prechelt] 1998], are commonly employed to enhance generalization and prevent overfitting. However, the efficiency of the training process itself remains a critical concern, particularly in terms convergence speed and computational resources.
ct al.
1986
et al.
Within this context, adaptive training strategies, which target enhanced generalization by modifying aspects of the training process, have emerged as promising approaches. Methods such as curriculum learning [Bengio et al. 2009 Graves et al. 2017 Wang et al. [2021] adjust the order and difficulty of training samples facilitate more effective learning. Insights from these adaptive training strategies can be extended to the class imbalance problem [Wang al.) 2019], where underrepresented classes are intrinsically harder to learn due to data scarcity [Buda et al. 2018), a challenge intensified in adversarial settings where safe data collection is severely limited (Wang and Gursoy) [022].. These
methods are typically categorized into data-leve methods, such as oversampling and undersampling Chawla et al. 2002 and algorithm-level schemes, including class-balanced loss functions [in et al.
GitHub repository: httts:///githb. wwccshuub.omm/siii http:///hahaaaaeeeaarraaaaa
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Initial class proportions
Adaptive evolution class proportions, et, ecross epochs
Finall class progortions
(original training detaset)
[(#*)) [x]
Class 1
Closs
o = 0.4
Mixing parameters
#; 0.4
Class 2
Class 3
65 0.3
of = 0.2
Class 4
- 8.1
Batch camposition at stophh
Batch composition epoch
Training epoch,
Figure 1: Illustration of the learn2mix training mechanism. The class-wise composition of batches is adaptively modified during training using instantaneous class-wing error rates.
2017]. However, developing adaptive neural network training methodologies that accelerate model convergence, while ensuring robustness class imbalance, remains an open problem.
Building upon these insights, a critical aspect of training efficiency lies in the composition of batches used during stochastic gradient descent. Classical training paradigms maintain approximately fixed class proportions within each shuffled batch, mirroring the overall class distribution in the training dataset (Buda al. 2018, Peng al.) 2019). However, this static approach fails account for the varying levels of difficulty associated with different classes, which can hinder optimal convergence rates. For example, classes with higher error rates those that inherently more challenging may require greater emphasis during training to enhance model performance. While existing approaches address class imbalance by adjusting sample weights or dataset resampling, they do not dynamically change the fassswwise composition of batches during training via real-time performance metrics.
This observation motivates the central question of this paper: Can adaptively adjust the proportion of classes within batches, across training epochs, to accelerate model convergence? Addressing this question involves developing approaches that dynamically modify class proportions using real-time performance metrics, directing learning towards underperforming classes. Such batch construction has the potential to enhance convergence rates and training efficiency, especially in scenarios with
imbalanced classes or heterogeneous class difficulties [Liu et al. 2008 Ren et al. 2018).
To address these considerations, in this work, introduce learn2mix, novel training strategy that dynamically modifies class proportions in batches by emphasizing classes with higher instantancous error rates. In contrast with classical training schemes that utilize fixed class proportions, learn2mix continually adapts these proportions during training via real-time class-wise error rates. This dynamic adjustment facilitates faster convergence and improved performance across various tasks, including classification, regression, and reconstruction. An illustration of the learn2mix training methodology provided in Figure [ demonstrating the adaptive class-wise composition of batches.
This paper is organized as follows. Section [ we formalize learn2mix, and prove relevant properties. In Section we detail the algorithmic implementation of the learn2mix training methodology. In Section 4 we present empirical evaluations on benchmark datasets, demonstrating the efficacy of leam2mmx in accelerating model convergence and enhancing performance. Finally, in Section summarize our paper. Our main contributions are outlined as follows:
1. We propose learn2mix, novel adaptive training strategy that dynamically adjusts class proportions within batches, utilizing class-wise error rates, to accelerate model convergence.
2. We prove that neural networks trained using learn2mix converge faster than those trained using classical approaches when certain properties hold, such that the class proportions converge stable distribution proportional to the optimal class-wise error rates.
3. We empirically validate that neural networks trained using learn2mix consistently observe accelerated convergence, outperforming existing training methods in terms of convergence speed across classification, regression, and reconstruction tasks.
The landscape of neural network training methods comprises various approaches aiming to enhance model performance and training efficiency. Handling class imbalance has been extensively studied, with methods such as importance sampling (Katharopoulos and Fleuret, 2018). oversampling Chawla et al., 2002), undersampling (Tahir et al. 2012), and class-balance loss
Related Work.

---

functions Lin et al., 2017 Ren et al. 2018 being proposed to mitigate biases towards majority classes. In parallel, curriculum learning [engii et al. 2009] and reinforcement learning approaches [Florensa et al.] 2017] have introduced methods to facilitate more effective learning trajectories. Meta-learning, or learn2learn methodologies [Arnold et al., 2020). including model-agnostic meta- learning (MAML) [Finn et al., [017]] focus on optimizing the learning process itself to enable rapid adaptation to new tasks. Additionally, adaptive data sampling strategies [Liu et al. [2008) and boosting algorithms [Freund and Schapire, [997] emphasize the significance of prioritizing harder or misclassified examples to improve model robustness. Despite these advances, most existing training methods either adjust sample weights, resample datasets, or modify the sequence of training examples without specifically altering the class proportions within batches in an adaptive manner. Our proposed learn2mix strategy distinguishes itself by adapting batch class proportions throughout the training process, targeting classes with higher error rates to accelerate convergence. This approach offers a unified framework by addressing class imbalance through adaptive training principles.
2 Theoretical Results
Consider the random variables X Rd and Y E R where X denotes the feature vector, are the labels, and & is the number of classes. We consider the original training dataset, J = {Tz, M5)
}
where (zz. Mi} ~ (x, Y), Vj E (1...., N). The class proportions for this dataset are given by the vector of fixed- preed-prpootion mixing parameters, à = [ā,,.... ... , reflecting the distribution of classes. We define = [al,..., oxl" as a variable denoting the vector of mixing parameters, where Og E [0, I] and Σ a; = 1. The value of a determines the class proportions used during training, and can vary depending on the chosen training mechanism. In classical training, over time and reflects the class proportions within the original training dataset, wherein a' = a, Vt E N. In learn2mix training,
Lid.
a² is constant
a' is time-varying, and is initialized at time 0 as a â.
Let H. C {h: Fd +R be the class of hypothesis functions that model the relationship between X and Y. For our empirical setting, we let H denote the set of neural networks that have predetermined architectures. We note H is fully defined by a vector of parameters, A E I where H ho denotes a set of parameterized functions. The generalized form of the loss function for classical training and the loss function form under learn2mix training are given below.
Definition 2.1 (Loss Function for Classical Training). Consider à E [0, 1]& as the vector of fixed- proportion mixing parameters, and let L((") E RR. denote the vector of class-wise losses at time t. The loss for classical training at time t is given by:
c(0', ā) = ==c(()'==a"" = 2(0").
Definition 2.2 (Loss Function for Learn2Mix Training). Consider a² a (0. 1) as the vector of mixing parameters at time t and time t - 1, and let L(O'), L(00-1) E R² denote the respective class-wise loss vectors at time and time - 1. Consider & E (0, 1) as the mixing rate. The loss for learn2mix training at time & is given by the following:
L(0', a²) = Σ
-0) =(a))T<(0))
1=1
((0--1) 1E -a--1) -
Where: a¹ = a² aa-1 +7
We note that the denominator, 1f C(0'-1), is the sum of losses across all classes, and dividing by it converts 1() ) into a probability distribution. We update a' by nudging the mixing parameters toward this probability distribution, so classes with higher losses receive a larger share of samples in the next time step. The scalar mixing rate, 7, is a user-defined step size hyperparameter that controls how aggressively d moves. We note that classical training is recovered by setting 7 = 0.
Suppose that H is sufficiently expressive and can represent the true conditional expectation function, wherein there exists 0" E IR with (x) = E X] almost surely. In the following proposition, we demonstrate that via maddient-based optimization under learn2mix training, the parameters con- verge to 0*, with the mixing proportions converging to a stable distribution that reflects the relative difficulty of each class under the optimal parameters.
3

---

Proposition 2.3. Let ( [(8)) ERR denote the respective class-wise loss vectors for the model parameters at time and for the optimal model parameters. Suppose each class-wise loss C,(0) E R is strongly convex in 0, with strong convexity parameter μt E &R Vi E (1....,k) k), and each class- wise loss gradient (0) E TTT Lipschitz continuous 0, having Lipschitz constant L, E Rzo. Vi E {1...., k.. Let μ² m Pb I = (1...( 14- Then, if the model parameters at time t + 1 are obtained via the gradient of the loss for learn2mix training, where:
0++1 = 0² -11V v/VeL(0' ,a',
with: 11 VRz0:
It follows that for learning rate, 1) E (0,2 and for mixing rate, 7 E (0, 1):
and: lim a' = ===a== [(6))
lim 0 0"
(5)
1/ 1(6')
The complete proof of Proposition 2.3 is provided in Section of the Appendix. We now detail the convergence behavior of the learn2mix and classical training strategies, and suppose that a' ā We first present Corollary 2.4, which will be used to prove the convergence result in Proposition 2.5 This corollary leverages Lipschitz continuity and strong convexity to bound the loss gradient norm.
Corollary 2.4. Let L(0*) R* denote the class-wise loss vector at time t. Suppose each class-wise loss, C,)0) E IR, is strongly convex in 0 with strong convexity parameter μti E R20. Vi E {1...., k). and suppose each class-wise loss gradient Vel.(0) E Im Lipschitz continuous in 0 with Lipschitz constant L; E Rzo, Vi E {l,..., k). Let p² = m Hú. L = (a... Li. Then, the following condition and inequality hold, Vo E [o, 1]a where Σ az 1:
14 1100 - 0-11 S VoC(O' a)ll - -- - -0"1,
(6)
Wherein: VeC((0' a')ll + Vo R)ll s 21" - -0
(7)
The proof of Corollary is4is provided in Section of the Appendix we note that the inequality in Eq. (7) relates the loss gradient norm under classical training with that under learn2mix training. We now present Proposition 2.5] which demonstrates that under the condition expressed in Eq. (8) updates obtained via the gradient of the loss for learn 2mix training bring the model parameters closer to the optimal solution than those obtained via the gradient of the loss for classical training.
Proposition 2.5. Let L()'), [(0*) E TR* denote the respective class-wise loss vectors for the model parameters at time and for the optimal model parameters. Suppose each class-wise loss, C(()) E R strongly convex 0 with strong convexity parameter μ4: E RR Vi [L....,k). and each class- wise loss gradient (0), E R.mh is Lipschitz continuous in 0, having Lipschitz constant Li E R20. di [l....,k}, k... Moreover, suppose the loss gradient VoC(0, a) IRn is Lipschitz continuous in a, having Lipschitz constant La E R20, and let μ¹" (I....) M4g L' = max.e(1,....)) Li. Then, if and only if the following condition holds:
11- +ap+aTcco
1100 -011-(00) (C(O') -((0)) >0, 0, (8)
follows that for every learning rate, 1j > 0, and for every mixing rate, 7 E 3]:
[| VoL' a²)) 00 s (1(0' n/V a)) 0
(9)
The complete formula for 3 can be found in Section of the Appendix.
The complete proof of Proposition 2.5 is provided in Section of the Appendix.
3 Algorithm
In this section, we outline our approach for training neural networks using learn2mix. The lear2mmx mechanism comprises a bilevel optimization procedure, where we first update the neural network parameters, 0', before updating the mixing parameters, a', using the vector of class-wise losses, L(0"). Considering the original training dataset, J, define J = {( Vj) 1= )o) Vi E (1...., k) as each class-specific training dataset, with J = U J. These & class specific training datasets are leveraged to speed up batch formation under learn2mix. We consider neural network training using

---

Algorithm 1: Neural Network Training Via Leam2MMxx
Input: J (Original Training Dataset), 0 (Initial NN Parameters), á (Initial Mixing Parameters), ")
(Learning Rate), y (Mixing Rate), M (Batch Size), P (No. of Batches), (Epochs) Output: (Trained NN Parameters)
for i 1,2,...k do
(Initialize class- spesisse training datasets) a; < 0; (Initialize time-varying mixing parameters)
Ji < {2z,. (z,,,;)) 33)1
for epoch = 1,2 do for 1,2..... do L J - Shuffle(J:) (Randomly shuffle each class-specific training dataset)
7
for p ... for i 1........ SP. - -Sample(J, M) (Select o; M distinct examples from J,)
P. do
h do
10
SP - Ei.st ("(0,a) a)) + M
b S. (Aggregate to form batch SP)
11
E (e ((ho(z)). Us) (Compute loss on batch ST)
((0,a) - 1 E (0, (Obtain total loss)
12
13
1--0-NVeL(t.c (Update model parameters, 0)
for 1,i==12,.. do
L.) - Ep=r 1 a.M Ees ((ho(zy), (Compute loss for class 1)
15
Params (()),))
17 return 0
batched stochastic gradient descent, where for training epoch, f, the empirical loss computed over P = N/M total batches, where M Z+ is the batch size. Each batch is formed by sampling of distinct examples from the ith class-specific training dataset, denoted as S S Jr, for 5" = +. Sp. where (t) is the set union operator that preserves duplicate elements. For learn2mix training, the class-wise errors, (,(0),, Vi (1,...,k)) at training epoch are empirically computed as:
P
(0) = P p=1
Σ
Σ (E
E(har (x)), Ub)
(10)
a
Where f: y: x x - R>0 bounded per-sample loss function and computes the error between the model prediction, hos (3.j), and the true label, yj- Accordingly, the overall empirical loss training epoch, f, under the learn2mix training mechanism given by:
2(0',a')
Σ -C))(00) = Σ o 1=1
k
P
((ho) (1)S6))
Σ
|]
(1I)
OCM
1=1
PE1
Utilizing the empirical loss formulation from Eq. we now detail the algorithmic implementation of the learn2mix training methodology
per-sample basis, for consistency with the mathematical
preliminaries Section 2 We note that the batch processing equivalent this procedure trivial extension to the domain of matrices, and was used to generate the empirical results from Section g
Algorithm]] [ outlines the primary training loop, where for each epoch, the class-specific datasets, J. shuffled. Within each epoch, we iterate a; examples from every Jg. The empirical loss within each batch is computed and aggregated to obtain the overall loss, L(O, a). which is then used to update the neural network parameters through gradient descent. Lastly, the vector of class-wise losses, L(0), is calculated to inform the adjustment of the mixing parameters, a, using Algorithm
the P total batches, forming each batch by choosing
Algorithm 2 outlines the method for adjusting class proportions using the mixing parameters, a, based on the computed class-wise losses. For each class, E (1,..., k). we first calculate the normalized loss L,, by dividing the class-specific loss (0) by the total cumulative loss summed over all classes. Each mixing parameter, as, is then updated incrementally towards this normalized loss value Li-
5

---

Algorithm 2: Updating Mixing Parameters Via Learn2Mix
Input: a (Previous Mixing Parameters), (() (Class-wise loss vector), 7 (Mixing Rate)
Output: ( (Updated Mixing Parameters)
for i 1,2,...,k k do
2
L +- Σi=1 ()) (Compute normalized losses)
2,(0)
0g < 02 + 7 (I4 az) (Update mixing parameters)
return a
The magnitude of the update step is controlled by the mixing rate, 7, determining how quickly the proportions adapt. Thus, classes exhibiting higher relative losses are progressively given greater emphasis in subsequent training epochs, ensuring balanced reduction of errors across all classes.
Finally, we recall that during the batch construction phase, for each class, E ( k), we select .M examples from each J, to form the subset S C Jj. Given the dynamic nature of the mixing parameters, a, is possible that this cumulative selection across batches may exhaust all the samples within a particular Jj before the epoch concludes. To address this, we incorporate a cyclic selection mechanism. Formally, we define an index T.,, Vi E {1,...,k) k) and p E (1,..., P), such that:
Tf 7 aaM) mod a;N,
(12)
Where -- = 0, (1..... .......................... Accordingly, when selecting S.. if +
a, > N,N, we wrap around to the beginning of Jr, effectively resetting the selection index, T. this ensures that every example in Jj is selected uniformly and repeatedly as needed throughout the training process. Thus, the selection procedure to construct S. is defined as:
SP =
Ja [T. w) mod a.N]
(13)
w=0
This cyclic selection procedure ensures that the required number of samples, a; for each class in every batch is maintained, even as ar is adaptively updated across epochs.
4 Empirical Results
In this section, we present our empirical results on classification, regression, and image reconstruction tasks, across both benchmark and modified class imbalanced datasets. We first present the classifica- tion results on three benchmark datasets (MNIST (Deng 2012). MMSIST [Xiao et al. 2017)
CIFAR-10 (Krizhessky al. 2009). ). and three standard datasets with manually imbalanced classes
(Imagenette [Howard] 2020]. CIFAR- 1-0 Krizhevsky et al. 2009], and IMDB [Maas et al. 2011
We note that for the imbalanced case, we only introduce the manual dlass-imbllancing to the training dataset, J, where the test dataset, K = {I yj) }
Nay, Nmm, is not changed. This choice ensures that the generalization performance of the network is benchmarked in a class-balanced setting. Next, for the regression task, we study two benchmark datasets with manually imbalanced classes (Wine Quality [Cortez et al. 2009]. and California Housing [Gronn 2022]. D. and a synthetic mean estimation task, where the manual class-imbalancing parallels that of the classification case. Finally, we reconsider the MNIST, Fashion MNIST and CIFAR-IU 10 datasets for image reconstruction, again with manual class-imbalancing. The comprehensive description of these datasets and class- btass-imbalancing strategies is in Scction the Appendix. For further performance verification, we include ablation studies on architecture, optimizer, batch size, learning rate, and worst-class error in Section B of the Appendix.
The intuition behind the application of learn2mix to regression and reconstruction tasks stems from its ability to adaptively handle different data distributions. For regression tasks with categorical variable taking & distinct values, the samples from that correspond each of the i values, can be aggregated to obtain each class-specifis training dataset, J: Here, each J; denotes distinct underlying data distribution. As in the classification case, learn2mix will adaptively adjust the class-specific dataset proportions during training. For image reconstruction, we can similarly treat the & distinct classes being reconstructed as the values taken by categorical variable, paralleling the regression context. This formulation supports the adaptive adjustment of class proportions under learn2mix training.

---

Table 1: Test accuracies for learn2mix (L2MM), classical (CL), FCL. SMOTE, 1S, CURR training.
Elapsed Time: [ 0.25
Dataset
MNIST
Fsh, MNIST
CIFAR-10
Imagenette
CIFAR- 100
IMDB
Acc (L2M)
95.4220.22 28
77.6220.69
51.3420.13
24.1220.46
30.03
70.28 666
Acc (CL)
93. 47
74.1320.79
49.2640.15
15.5510.18
23.23
60.13
Acc (FCL)
91.3220.57
74.08:0.75
47.90=0.22
20.37
27.15#1.13
0..30=0.67
Acc (SMOTE)
92.41
73.67
47.76=4.115
23.93
50.941=0.00
Acc (IS)
92. 4..64
74.23=#0.29
40.60
23. 23.10
27.97 +0.07
58. 08
Acc (CURR)
93.30#0.64
75.06#0.63
49.11
18.82
27.15
50.02=0.04
Elapsed
Time: 0.
Dataset
MNIST
Fsh. MNIST
CIFAR-I0
Imagenette
CIFAR-I00
IMDB
Acc (L2M)
97.61 10.15
83. 63.1687..8
55.84+0.19
33.6/1.0.42
46.80
76.02=2.1 77
Acc (CL)
96.74=0.10
79. 79.
54.50+0.34
23.63
43.00
71.99
Acc (FCL)
95.92:10.13
78.9410.62
54.15:0.00 os
29.44:10.43
40.26
68. 1..300:..00
Acc (SMOTE)
96.51
19.17
53.7211.
28. 10.90
39.10:11.63
62.72:00554
Acc (IS)
96.60.0.25
79, 10..35
52.56
28, 1..32
42.61 1:2.61
=4.00
Acc (CURR)
96.53
79.08
53.49 10.33
2..26
39.48
71. 71.68
Elapsed
Time: t =
Dataset
MNIST
Fsh. MNIST
CIFAR-10
Imagenette
CIFAR-100
IMDB
Acc (L2M)
98.46
85,85
60.49
10.35
54.50
82. 22.38
Acc (CL)
98. 14
84.23
59.62.40.18
0..53
52.30+00.30
80, 0..84
Acc (FCL)
97. 97.8620.00
83.68:0..11
59.37=2064
40.60:20.71
49.33
79.09
Acc (SMOTE)
98. 20.09
83.57
58.4620.11
39.59
50.63
74.64=1.22
Acc (IS)
98. 14
84.3322022 29
57.4420.42
35.33
52.83
79.08 000.57
Acc (CURR)
98.13+0.05
83.32#+.48
59.15++042
35.26++0.48
50.88
00.04
Skgese time jecando
Slpiac Time Since
Chasse line Jecends
(a) MNIST
(b) Fashion MNIST
(c) CIFAR-10
Clacke
Dae baa
D044
Dupeed Time ussart
Cacoed DSCSPIC
Oapaee aroonds
(d) Imagenette
(e) CIFAR-100
( IMDB
Figure 2: Comparing model classification errors for learn2mix, classical, FCL, SMOTE, IS, and CURR training. The x-axis is the elapsed [training] time, while the y-axis is the classification error.
For the evaluations that follow, all training was performed on an NVIDIA GEForce RTX 3090 GPU. To ensure a fair comparison between learn2mix and classical training, we utilize the same learning rate, 1), and neural network architecture with initialized parameters, 0, across all experiments for given dataset. Additionally, we train each neural network through learn2mix (with mixing rate 7) and classical training for E seconds (or E epochs), where E is dataset dependent [1] In classification tasks, we also benchmark Jearn2mix and classical training versus 'FCL training'. .SMOTE training', "IS training". and 'CURR training' (training using focal loss [Lin et al.) 2017). SMOTE oversampling
CCawla et al., 2002). importance sampling [Katharopoulos and Fleuret 2018), and curriculum
learning [Hacohen and Weinshall 2019
Sections D.3 D.4 D.5 and D.6 the Appendix).
The complete list of model architectures and hyperparameters is in Section Di the Appendix.
Praactically, observe that choosing y [0.01, 0.1] improves performance (see Section of the Appendix).
7

---

4.1 Classification Tasks
As illustrated in Table and Figure we observe consistent trend across all considered classification benchmarks, whereby neural networks trained using learn2mix converge faster than their classically- trained, FCL-trained, AMOTE-trained IS-trained, and CURR-traines counterparts. We first consider MNIST, and train LeNet-5 [Lecun al. [998] via the Adam optimizer [Kingma and Ba 2014] and Cross Entropy Loss for E = 50 s, leveraging learn2mix, classical, FCL., SMOTE, IS, and CURR training. We see that the learn2mix-traine within 30 whereas the remaining CNNs achieve this test accuracy after 40 s. We next consider the more challenging Fashion MNIST dataset, and train Large LeNet-5 for E = 50 s with the Adam optimizer and Cross Entropy Loss, leveraging learn2mix, classical, FCL. SMOTE, IS, and CURR training. Paralleling MNIST, we observe that the learn2mix-trained CNN converges faster, yielding a test accuracy of 83% within 20 s, whereas the other CNNs achieve this test accuracy after 33 The last class-balanced benchmark dataset we investigate is CIFAR- 10, which offers a greater challenge than MNIST and Fashion MNIST. We train Large LeNet for E 200 s using the Adam optimizer and Cross Entropy Loss, utilizing learn2mix, classical, FCL. SMOTE, IS, and CURR training. We observe that the lear22mix-traie trained CNN achieves faster convergence, yielding test accuracy of 60% after 170 s, whereas the remaining CNNs exceed this test accuracy after 200 s. Cumulatively, these evaluations demonstrate the efficacy of learn2mix training in settings with balanced classes, wherein the adaptive adjustment of class proportions accelerates convergence.
CNN converges faster, eclipsing a test accuracy of 98%
We now consider several class-imbalanced training datasets. We first benchmark Imagenette, which comprises a subset of 10 classes from ImageNet [Deng ct al. 2009), and modify the training dataset so the number of samples from each class, E {1, ...,, k). in J decreases linearly. We train R [He et al. 2016] with the Adam optimizer and Cross Entropy Loss for E = 230 s. using learn2mix, classical, FCL, SMOTE, IS, and CURR training. We see the learn2mix- ResNet-18 converges faster, achieving a test accuracy of 40% after 185 s. whereas only the FCL-trained model achieves this test accuracy after 230 We now consider CIFAR-100, and modify the training dataset so the number of samples from each class, E {l.....A ). in J decreases logarithmically. We train MobileNet- V3 Small (Howard et al. 2019 for E- 200 s leveraging the Adam optimizer and Cross Entropy Loss, using learn2mix, classical, FCL. SMOTE, IS, and CURR training. We see that the learn2mix-traine MobileNet-V V3 Small model converges faster, achieving a test accuracy of 50% within 120 s, whereas the other models exceed this test accuracy after 140 s. As the k = 100 mixing parameters are a small fraction of the total model parameters, this overhead is negligible. For IMDB, we modify the training dataset so the positive class keeps 30% of its original samples. We train a transformer for E = 150 s with the Adam optimizer and Cross Entropy Loss, using learn2mix, classical, FCL, SMOTE, IS, and transformer converges faster, reaching test accuracy
CURR training. We see the of 70% within 35 s, whereas the other models exceed this test accuracy after 60
In the above evaluations, we see learn2mix not only accelerates convergence, but also has a tighter alignment between training and test errors versus classical training. This correspondence indicates reduced overfitting, as learn2mix inherently adjusts class proportions based on class-specificc error rates, L, By biasing the optimization procedure away from the original class distribution and towards L,, learn2mix achieves improved generalization. We note this property is not unique to classification and also applies to regression and reconstruction (see Sections 4.2 and 4.3)
4.2 Regression Tasks
As illustrated in Table and Figure 3, observe that learn2mix maintains accelerated convergence in the regression context, wherein all the considered datasets are class imbalanced. We first consider the synthetic Mean Estimation dataset, which comprises sets of samples gathered from k 4 unique distributions and their associated means. Using the Adam optimizer and Mcan Squared Error (MSE) Loss, we train a fully connected network for E = 500 epochs on Mean Estimation via learn2mix and classical training. We see that the maanmmixx-trineee neural network converges rapidly, achieving a test error below 2.0 after 100 epochs, at which point the cessscclll-ttrainee network has a test error of 13.0. For the Wine Quality dataset, we modify the training dataset so the white wine class has 10% of its original samples. Using the Adam optimizer and MSE Loss, we train a fully connected network for E = 300 epochs on Wine Quality via learn2mix and classical training- We observe that the learn2mix-trainn
neural network yields faster convergence, achieving a test error below 2.5 after 200 epochs. at which point the classically trainedd network has test error of 5.0. Finally, on the
8

---

Table 2: Test mean squared error for learn2mix (L22M and classical (CL) training.
Epoch f. = 0.25 E
Epoch t = 0.5.E
Bpoch t = E
Dataset
Err (I.2M)
Err (CL.
Err (I.2M)
Err (CL)
Err (I.2M)
Err (CL)
Mean Estim.
1.81
6.51=1552
0.45++0.26
1.52=0.27
1.07=0.09
1.17=0.06
Wine Quality
17.7
19.8
1.26
0.72.1.94
1.75.10.21
2.13
Cali. Housing
2.52
2.9510.67
1.33
1.39
0.7710.08
0.99
MNIST
19.6:10.81
20.8
12.9:1.39
0.50
9.31:10.24
10.1
Fsh. MNIST
89.3
91.9112.37
65.1+1.21
70.9
45.511.21
51.6=11.0
CIFAR-10
1.3+
194+1.98
17542285
17943.87
144+1.71
148+1.37
bons Eree
Canakol Orer
Ceocial Learoike Irin iner
Learcinc Train over
Learrditi Thai Emn"
Leamonic Orer
Laaer Erar
LeamONic the Crez
(a) Mcan Estimation
(h) Wine Quality
(c) California Housing
Clasice Fain Ever
Cama cal Thii Crer entrent
clasdical main
Clssical That Ever
Leart Mic les Oree
Leeoceee Text Ertr
of
Nurbber
(d) MNIST
(c) Fashion MNIST
(f) CIFAR-10
Figure 3: Comparing model performance errors for classical training and learn2mix training. The
x-axis is the number of elapsed training epochs, while the y-axis is the mean squared error (MSE).
California Housing dataset, we modify the training dataset such that three of the classes have 5% of their original samples. Using the Adam optimizer and MSE Loss, train fully connected network for E = 1200 epochs on California Housing via learn2mix and classical training. We again see that the learn2mix- trained network converges faster, achieving test error below 0.8 after 1200 epochs, while the classically-trained network has a test error of 0.99. These empirical evaluations support our previous intuition pertaining to the extension of lear22mix imbalanced regression settings.
4.3 Image Reconstruction Tasks
Per Table and faster convergence using learn2mix. For the MNIST case, we modify the training dataset such that half of the classes retain 20% of their original samples. Leveraging the Adam optimizer and MSE Loss, we train an autoencoder for E = 40 epochs on MNIST using learn2mix and classical training. We observe that the leam2mixx-tranne autoencoder exhibits improved convergence, achieving test error less than 1.0 after 35 epochs, which the llassiaally-traine autoencoder achieves after 40 epochs. Correspondingly, for Fashion MNIST, we modify the training dataset such that half of the classes retain 20% of their original samples (paralleling MNIST). Using the Adam optimizer and MSE Loss, we train an autoencoder for E 70 epochs on Fashion MNIST, leveraging learn2mix and classical training. We observe that the below 54.0 after 50 epochs, which the messically-trainee autoencoder achieves after 65 epochs. We also consider 110 wherein we modify the training dataset such that all but two classes retain 20% of their original samples. Utilizing the Adam optimizer and MSE Loss, we train an autoencoder for = 110 epochs on CIFAR-10, leveraging learn2mix and classical training. We observe that the learm2mix-trainee autoencoder also converges faster and achieves a test error below 148.0 after 100 epochs, which the mussically-trainee autocncoder achieves after 110 epochs.
and Figure B note that the ilass-imbllanced image reconstruction tasks also observe
meremmxxxttrined autoencoder converges faster, achieving test error

---

Conclusion
In this work, we presented learn2mix, a new training strategy that adaptively modifies class propor- tions in batches via real-time class-wise error rates, accelerating model convergence, We formalized the learn2mix mechanism through bilevel optimization framework, and outlined its theoretical advantages in aligning class proportions with optimal error rates. Empirical evaluations across classi- fication, regression, and reconstruction tasks on both balanced and imbalanced datasets confirmed that lear2mmix not only accelerates convergence compared classical training methods, but also reduces overfitting in the presence of class-imbalances. Accordingly, models trained with learn2mix achieved improved performance in constrained training regimes. Our findings underscore the potential of dynamic batch composition strategies in optimizing neural network training, paving the way for more efficient and robust machine learning models in resource-constrained environments.
Acknowledgments and Disclosure of Funding
Shyam Venkatasubramanian and Vahid Tarokh supported in part by the Air Force Office of Scientific Research under award FA9550-21-1-0235.
References
Sébastien M. R. Arold, Praateek Mahajan, Debajyoti Datta, Ian Bunner, and Konstantinos Saitas Zarkias. learn2leam: A library for meta-learning research, 2020.
Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. In 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 776-780. IEBE, 2020.
Yoshua Bengio, Jean Louradour, Ronan Collobert, and Jason Weston. Currieulum learning. In Proceedings of the 26th Annual International Conference on Machine Learning, pages 41-48. ACM, 2009.
Léon Bottou. Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT'2010, pages 177-186. Springer, 2010.
Mateusz Buda, Atsuto Maki, and Maciej A Mazurowski. A systematic study of the class imbalance problem in convolutional neural networks. Neural Networks, 106:249-2550 2599,92-9,,99 2018.
Nitesh V Chawla, Kevin W Bowyer, Lawrence Hall, and W Philip Kegelmeyer. Smote: Synthetic minority svep-sampling technique. In Proceedings of the 2002 Joint Conference IEEE Interna- tional Conference on Knowledge Discovery and Data Mining and IEEE European Conference on Machine Learning, pages 878-884 IEEE, 2002.
Paulo Cortez, António Cerdeira, Fernando Almeida, Telmo Matos, and José Reis. Modeling wine preferences by data mining from physicochemical properties. Decision support systems, 47(4): 547-553, 2009.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-25- Icce, 2009.
Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 1916)141 2012.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 18 18.00000 2018.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1126-1135 PMLR, 06-11 Aug 2017.
10

---

Carlos Florensa, Yoshua Bengio, and Aaron Courville. Automatic goal generation for reinforcement learning agents. International Conference on Learning Representations, 2017.
Yoav Freund and Robert E Schapire. A application to boosting. Journal of computer and system sciences, 55(1):119-15) 1997.
generalization of on-line learning and an
Aurélien Géron. Hands-on machine learning with Scikit-Learn, Keras, and TensorFlow. "' O'Reilly Media, Inc.", 2022.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis. Lukasz Wesolowski, Ari Kyrola, Joshua Tulloch, Yangqing Jia, and Kaiming Hc. Accurate, large minibatch sgd: Training imagenet in hour. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1214,1 2017.
Alex Graves. Generating sequences with recurrent neural networks. arXiv preprint arXiv: 1308,000, 2013.
Alex Graves, Mare Bellemare, Jacob Menick, Remi Munos, and Koray Kavukcuoglu. Automated curriculum learning for neural networks. In international conference machine learning, pages 1311-1320. Pmlr, 2017.
Guy Hacohen and Daphna Weinshall. On the power of eurriculum learning in training deep networks. In International conference on machine learning, pages 2535-2544 PMLR, 2019.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 770-778, 2016.
Geoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patriek Nguyen, Tara N Sainath, et Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 29(6):82-97, 2012.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang. Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF international conference on computer vision, 2019.
Jeremy Howard. Imagenette. Sttps://githube httpp:://tttttttccooooooo//////t http///uw..ooo/aaaaaaaaaaaaaaees imtgeteeiiiimmneeeteeeeeeee 2020.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pages 448 4566. PMLR, 2015.
Mathias Johansson and Emma Lindberg. Importance sampling in deep learning: broad investigation on importance sampling performance, 2022.
Angelos Katharopoulos and François Fleuret. Not all samples are created equal: Deep learning with importance sampling. International conference on machine learning, pages 2525-2534, PMLR, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6988. 2014.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in Neural Information Processing Systems, pages 1097-1105, 2012.
Y. Lecun, L. Bottou, Y. Bengio, and Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2222 1998. doi: 10.1109/5.72679
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollar. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision, pages 2980-2988, 2017.
11

---

Xu- Ying Liu, Jianxin Wu, and Zhi-Hua Zhou. Exploratory undersampling for learning. IEEE Transactions on Systems, Man, and Cybernetics. Part B (Cybernetics), 39(2): 539-550, 2008.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150, Portland, Oregon, USA, June 2011 1. Association for Computational Linguistics.
Minlong Peng, Qi Zhang, Xiaoyu Xing. Tao Gui, Xuanjing Huang, Yu-Gang Jiang, Keyu Ding, and Zhigang Chen. Trainable undersampling for class-imbalance learning. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 4707-4714, 2019.
Lutz Prechelt. Early stopping but when? In Neural Networks: Tricks of the trade, pages 55-69. Springer, 1998.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 1(8), 2019.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In International conference on machine learning, pages 4334-4343, PMLR, 2018.
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back-propagating errors. Nature, 323(6088):333-536 1986.
Connor Shorten and Taghi M Khoshgoftaar. survey on image data augmentation for deep learning. Journal of Big Data, 6(1):60, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 2014.
Leslie N. Smith. Cyclical learning rates for training neural networks. In 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 464 472. IEEE, 2017.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, lya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, (3(1::1929-195) 2014.
Muhammad Atif Tahir, Josef Kittler, and Fei Yan. Inverse random under sampling for class imbalance problem and its application to multi-label classification. Pattern Recognition, 45(10))3738-77 -3550, 2012.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
Vladimir Vapnik and Léon Bottou. Local algorithms for pattern recognition and dependencies estimation. Neural Computation, 5(6):893-909, 1993.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998-6008, 2017.
Xin Wang, Yudong Chen, and Wenwu Zhu. A survey on curriculum learning. IEEE transactions on pattem analysis and machine intelligence, 4409):4555-4577 2021.
Xueyuan Wang and M Cenk Gursoy. Resilient path planning for uavs in data collection under adversarial attacks. IEEE Transactions on Information Forensics and Security, 18, 2023.
Yiru Wang, Weihao Gan, Jie Yang, Wei Wu, and Junjie Yan. Dynamic curriculum learning for imbalanced data classification. In 2019 JEEE/CVF International Conference on Computer Vision (ICCV), pages 5016-5025, 2019, doi: 10. 10.1109//CCV.221990055..
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint
2017.
12

---

Appendix
A Proofs of the Theoretical Results
In this section, we present the proofs of the theoretical results outlined in the main text.
Proposition 2.3] Let C [(0)) E Ie* denote the class-wise loss vectors for the model parameters at time t and the optimal model parameters. Suppose each class-wise loss C,(0) E R is strongly convex in 0, with strong convexity parameter μt E R>,, Vi E {l...., k,,, and each class-wise loss gradient V (0),(0) Th. is Lipschitz continuous in having Lipschitz constant L; E Rzo. Vi E (1....,kk, k)- Let J = minne m (I...k) p4i, L' = maxic (....k) Lg. Then, if the model parameters at time t + 1 are obtained via the gradient of the loss for learn2mix training, where:
gi+1 = 0 - ,a),,
with: 17 nER>0
(14)
follows that for learning rate, 1] E L'),, and for mixing rate, 7 E (0,
lim a² a²=aa== L((*) 60000
lim 0 0", 20000
and:
(15)
1 L(Ar)
Proof. We begin by recalling that (0) is strongly convex in 9 with strong convexity parameter μa. Vi E {1....,A k). Accordingly, Va E 1jk, with Σ a 1, the loss function L(O, a:) is strongly convex with parameter, / E Rys. which is lower bounded by μ² E Ik>0, as per Eq. (1))
1 2 4 >0, 0,
where:
p
min (E.......k)
Mi:
and: n'
appi-
(16)
We note that this lower bound on the strong convexity parameter, μ' 2 n², holds independently of a. Now, recall that (0), is Lipschitz continuous in 0 with Lipschitz constant L,, Vi {l....,k} k]. Accordingly, Va E [0, 1j%, where Ek= = 1, the loss gradient V (o a is Lipschitz continuous in with Lipschitz constant, L' E R20, which is upper bounded by L R2u, as per Eq. (17)
L L'2L00,
where: L
max Li, :E(I.....k)
and: L' = 1=1
L'=E=
agLi
(17)
We affirm that this upper bound on the Lipschitz constant, L' < L' holds independently of a. Now, suppose that a a', where C(6, a²) is strongly convex in 0 with parameter μz' 2 μ² and V VoC(O, a') is Lipschitz continuous in 0 with constant I,' < I." Let p max{ [1] n/11,11 nI=1))- By the gradient descent convergence theorem, for learning rate, 17 20.,/L"), it follows that:
lim 100 0*11 < lim p² 1000 - 00 = |100 0' lim p² =. 10000
(18)
t
t
Therefore, lim 0 = 0" Let gr-1 L(0 // [ c(Br [ wherein f E [0, 11 Unrolling the recurrence relation from Eq. (5) and expressing it in terms of 3t-1, we obtain:
a² = (1- (1-
Σ (1 -
(19)
1=0
Taking the limit and re-indexing the summation using n t - and / 1= -
n, we obtain:
t-1 Σ (1 7)
lim a' lim (a - 7)
+ lim
3t-1n ((-
(20)
20000
t+000
80000
0
t-1
Og + lim
Σ
>) 3t
(21)
++00
We proceed with the steps to invoke the dominated convergence theorem. We note that for fixed n:
((1'-1)
2())
lim (1--))))---m)] (1 -7)" lim
= (1 (1-7)) 7)
(22)
c(00)
13

---

Now, consider g(n) = (1 -7)". For this choice of g(n), we have that:
[I - -" 1| S -1-77 11- 1-"1 S g(m), Vl, N
(23)
Σ (()) = - (1 1 - (1 -1) = 80. ==0 ==0
(24)
We now invoke the dominated convergence theorem. Recalling Eq. (1 we observe that:
lim a' y lim
(25)
1000
1000
0
_())) ((8')
7)th
lim g0-1-n 7
> (1 7)th
(26)
(4000
n=0
n=
L((*)
L(00)
(7)
a²
(27)
7 1 L(B*) 1 ((0))
Therefore, lim a² = a- = [/0')/[1 [(+)] Cumulatively, for E (0,2/L*) and y E (0, 1). under learn2mix training, lime 0¹ = 0", and lim,- a² - ( ((0"))
Corollary 2.4 Let C((') E R* denote the class-wise loss vector time t. Suppose each class-wise loss, (,( E R, is strongly convex in 6, with strong convexity parameter μ4 E Rso. Vi E {1...., k). and suppose each class-wise loss gradient (6) E R.m Lipschitz continuous @ with Lipschitz constant L; E Rzo. Vi E
.k. Let p = min (1....k) H4, L = ma...(1.....))) Li. Then, the
following holds, Va E [0, 1],, with E = 1:
2 1100 - I I Vo a)ll S L' - 11
(28)
Wherein: |VoL(0', a')]l + [oL((0', a))) S 21. 21.' - -011-
(29)
Proof. We begin by recalling that (0) is strongly convex in 9 with strong convexity parameter 14; Vi E {l...., ,k.. Accordingly, Ya E 1], with Ek1 a; = the loss function L(0, a:) is strongly convex 0 with parameter, μ E Rso, which is lower bounded by μ² E <R as per Eq. (30).
and: v = = 0:P4
r 2 4² >0, 0,
where: p
min Mi: kE
(30)
Now, recall that (oL, is Lipschitz continuous in 8 with Lipschitz constant L,, Vi f1....,,k k}. Accordingly, Va [0, 1]" where D a, = 1, the loss gradient L a:) is Lipschitz continuous in with Lipschitz constant, L' E Rzo, which is upper bounded by L E R2z, as per Eq. (31)
L'20'0,
where: L max Lis
and:
L' =
-E, o; L.
(31)
:E(
Note that VeL(O", ,)) -0m- Since L(0, a) is strongly convex in 0, the following inequalities hold:
[(0', a) L("", a) 2 VoL(0", a)" (0' 0*) + 1² 110 1-
n¹
2 100 0-13
(32)
((0', a) - C( a) < VoL(0', a (0' 0*) < | al] 8"
(33)
Combining Eq. (2 and Eq. 33 and recalling Eq. (30). we obtain the following inequality:
L( a) L(
||VoL(0' a]]l
10 1-
(34)
||0 8*1
Furthermore, since Vo is Lipschitz continuous in 0 and recalling Eq. 31 it follows that:
||VoC(0', V VeC(0", a) < L'| 1' = ||Ve((0' S L" 110 11
(35)
Altogether, combining Eq.
34p and Eq. (35) we arrive at the final inequality:
n |
0- VoL(O' a) L L 1-
(36)
Furthermore, since Eq.
35
holds Va E [0, 1 where Σ a; = 1, it follows that:
VeL(0' aa)ll + & a)ll < 21 1 0'
(37)
14

---

Let L(0"), ((8) E B. denote the respective class-wise loss vectors for the model parameters at time and for the optimal model parameters. Suppose each class-wise loss, (;()) E R is strongly convex in O with strong convexity parameter μ4i E > Vi E {1.....ki, k), and each class- wise loss gradient (0) E ERTT is Lipschitz continuous in 0, having Lipschitz constant T, E Rzo. di E {1,...,k) ... Moreover, suppose that the loss gradient VgL o:) E TRm is Lipschitz continuous in a, having Lipschitz constant La E R22. and let μ³ = ( Mbs - = max Li.
Proposition 2.5,
Then, if and only if the following holds:
-I-1100 - 001 1 + a² (C(0') - [(01)) [100 - 0-1l (C(()) - [(0')) > 0,
(38)
It follows that for every learning rate, 1) > 0, and for every mixing rate, 7 E (0.13)
11(0' NVL((0 a²) )) 0 - 11(0' nVVo a)) - 001
(39)
(62 - L 10 1| + ( - ((0)))
Where: B
(40)
LL.L L"
100 - 8 (C(0t) - C(O-)) 1. 1j ā
110t-1)
Proof. We note that for all subsequent derivations, F(O', 0" n1 a²) 11000 Vo ,²)) 0'1
and G(0', 0r, n, ū) = 11(0' '/VoL((0' ā)) -0" ||, where a² 1- = ã. We begin by observing that:
[F(" 0" 77, a²) = =100' - 0r - 2n( 87 V C a') + " a²))
(41)
[F(8', 8" 77 a) 110* 0+ 12 27y V a) Vo a)1
(42)
Accordingly, the difference between [F(0", 0, n, ,,,0, a²) and [ 0,, .7 ] 2 is given by: 0*, - a.)² = -2n - 030 (V V0' a²) - VoL(0' .a))]
[F(O', 0-, n,,,a)'' [9(00
(43)
+ + a - {a)]]
Consequently, suppose that H(0 0-, n, aa) = (0 - 0 (Vo at) VeL(0' a))] and let J J(0", 17, ã, a²) = n² [IV
a) 1/I. Suppose the loss gradient, VoL(O,a),
is Lipschitz continuous in a with Lipschitz constant, La. We now upper bound J(0", n7, a, a.)):
J(B' n, n,,) = v' a²) VoL(0) a)] T [Vo a²) + VoL(0' a)]
- I|VoL(0' a²) - VoL(o a)llI a²)+ + Vol(O')
(44)
s 2n Lalla' ā [ ((00,))))) a')ll + (oC(0')))
(45)
S 2n 2a La b''laaa - -all10' - 0'11
(46)
2(1--1)
- 100 1
-212 1i
à + 7
1 2(00-1) -a -
(47)
1(0 1/. C(Ot-1-
=2n Ln 1
ā 110 0-|
(48)
We note that this upper bound follows from the Cauchy- Schwarz inequality and Corollary 2.4 We now proceed by lower bounding H(0', 0-,, n, ā, a²):
H100, 0", ,,, a²) = 2// (0' 00)0 VoL(O', . (0' 0-2² V VoL(O' a)
(49)
2 27/
(0' "''"TTV ,a) 110* 1' []]l
(50)
2 27
(0' 07) Vo .09 I.' 110' 13]
(51)
2.7
100 0- + L(0', a²) L(1', aa)) T." 11' - 0-1(2)
(52)
2
27
"1"
11- 11)110 110 - 1- + (T(((()) C())))
2
17 [(B(1111) -a) (C((T) - -(07))
C(0L-1)
66)
+7
(53)
15

---

As in above, we note that this lower bound also follows from the Cauchy-Schwand inequality and
Corollary 2.4 and further invokes the strong convexity of L(O, a) in 8. Combining Eq. (48) and
Eq. 533 we obtain the following upper bound on [F(0', 00, ,, a a²) [9(0', 0", n, a)]
[F 0" n, n) [9(0', 8r, 17, a]]] S <K((0' 0", 17, 1,7,,, ,,,,
(54)
Where: K((0', 00 n, a, a²) ) = -2 n -1.² 110 - 0011 + at (2(0') - L()*))
2((0-1)
à1) T
(2(0') [(03)
7
1T 1(0--11
f(0(-1)
001100
+ 211² LaLin 7
10 0-1
(55)
1 C(Br-)
Now, consider the following chain of inequalities deriving from Eq. (54)
K(00 0-, n. n,7,a a²) s 0 = [F(0 0- 7, a.)') - [g(0', 000 17. a))
(56)
[F(0", 0r, 7. a²)] < [g(0",0" ,,,,,),
Accordingly, we aim to find a condition on the mixing rate, 7, under which the chain of inequalities is satisfied. We proceed by letting K(0", 0 77 7, ā. a²) I 0, and rearrange the terms:
1(0
2 - 0-112 - (=(('))))))))))))))))))c))) (()))) 2 7 nLab"
ā 100 10'- 001
(57)
1 C(H--))
(((--1)
1 L((1-1) a) (C(0") (C(0))-(00)))
We note that this chain of inequalities is satisfied if, for every 17 > 0:
( - L") 11 - 1- + (C(0t) - ((()))
0 0<75
SB, (58)
c(D--1) à 110 - 0-1 -0'1- (1-2001 - a) (C(O') C((-))
nhab* L+
15 Ln) 1'0 0- + (C(0') - -(0)))
Where:
B
(59)
nLLa L'
c(Be 1)
110 1100--011- (C(0)) - -((-)))
0-11
1 1(0-
However, > 0 iff the numerator and denominator from Eq. (59) have the same sign, ensuring that O. Accordingly, if and only the condition provided in Eq. (60) is satisfied:
1 - + at (C(O') [(0))) [10 - 0-ll (C(()) - C(0))) > 0, (60)
It follows that for every learning rate n > 0, and for every mixing rate E (0, 3] satisfying Eq. (59h (1 - VoL(0' a²)) - 0- < [1 V/V' a)) - 0-|
(61)
B Additional Empirical Results
For further performance verification of learn2mix, we present several ablation studies quantifying the effects of different architectures, optimizers, batch sizes, learning rates, and mixing rates for the considered classification tasks from the main text. We further present the worst-class classification accuracy on Imagenette further gauge the efficacy of learn2mix within imbalanced classification settings, and illustrate how the mixing parameters converge to stable distribution on Mean Estimation. We first consider CIFAR- 10 and CIFAR- (per Section C and evaluate whether the gains afforded by learn2mix persist across architectures. For CIFAR- 10, we recall the Large LeNet architecture, trained using the Adam optimizer and Cross Entropy Loss with learning rate r = 7e-5 for E = 200 s, and the MobileNet-V3 V3 Small architecture, trained using the Adam optimizer and Cross Entropy Loss with learning rate 7 = Te-4 for E = 750 s. For CIFAR-100 100, we consider again the MobileNer-V V3 Small architecture, trained using the Adam optimizer and Cross Entropy Loss with learning rate
16

---

Casatoal Turn Dner
30
Casacall
Clssiial teat Enter
Fece Train Frtr
swarls Faan trer
Focal
Chassift fan Eren (%6]
led lere
swore Bo1
Evorr
Leart 2014
Learr Mit bec Drcce
Cansiiii
Erai
is Tea Eog
clisA Trai
0000
CURR Eri
Elapserd Time (sercondis)
Elapsed Time (seconds)
(a) CIFAR-I0: Large LeNet
(b) CIFAR-10: MobileNet- V3 Small
Classifcation (40
Classifcaaiio Error 1%5)
Guaacel Tram Ener
Gemical Train Entr faeeiral liar
SMOTE Tran
SHOTE
SMCTE Tex: Tear
SHOTE That Ere
LANTOND Thin
bot brtrr
kist brice
CURR tran biu
Ri Tha cror
Elapoed Timm (ascoodda))
Elpped Tire (secuncla)
(c) CIFAR-100: Large LeNet
(d) CIFAR-100 100: MobileNet- V3 Small
Figure 4: Comparing model classification errors for leam2mix, classical, FCL, SMOTE, IS, and CURR training. The x-axis is the elapsed [training] time, while the y-axis is the classification error.
77 7= 1e-4 for E 200 and the Large LeNet architecture, trained using the Adam optimizer and Cross Entropy Loss with learning rate = le-4 for E = 50 s. The results are depicted in Figure We observe that for both Large LeNet and MobileNet- V3 Small, the learn2mix trained models converge faster than the classical, FCI., SMOTE, IS, and CURR trained models.
4
Next, we evaluate the robustness of learn2Mix to different optimizers and batch sizes. As we used the Adam optimizer in the main text, we now consider the RMSProp optimizer [Graves] 2013) with batch size M E {250, 500, 1000}. We train LeNet-5 on MNIST using Cross Entropy Loss with learning rate 7 = le-5 for E = 45 s. E = 60 s, and E 70 s. As depicted in Figure we see that learn2mix converges faster than classical, FCL, SMOTE, IS, and CURR training.
(a) MNIST: M= = 150
(b) MNIST: M = 500
(c) MNIST: M = 1000
Figure 5: Comparing model classification errors for learn2mix, classical, FCI., SMOTE, IS, and CURR training. The x-axis is the elapsed [training] time, while the y-axis is the classification error.
We further verify the robustness of learn2Mix to different learning rates. We train LeNet-5 on MNIST using Cross Entropy Loss with learning rate E {1e-5, le-4, 1e-3) for E = 75 s, = 50 s, and E = 45 Per Figure o observe that the faster convergence afforded by learn2mix is apparent for 17 E {1e-5, 1e-4]. For ア = 1e-3, we note that after convergence, the learn2mix train error continues to decreases at faster rate than the the classical, FCL, SMOTE, IS, and CURR train errors.
We now illustrate the worst-class classification accuracy on Imagenette and IMDB as an additional metric to gauge the efficacy of learn2mix for imbalanced classification settings. We train ResNet-18
17

---

Were Tat
14 Turter
Oksoed bucndd
Eacoed Tewresence
Dipred Intonda
(a) MNIST: 17 1c-5
(b) MNIST: 77 1e-4
MNIST: 1j == 1c-3
Figure 6: Comparing model classification errors for learn2mix, classical, FCL., SMOTE, IS, and CURR training. The x-axis the elapsed [training] time, while the y-axis is the classification error.
on Imagenette via Cross Entropy Loss with learning rate 77 1e-5 for E 240 s, and a transformer on IMDB using Cross-Fntroos Loss with learning rate 1e-4 for E = 150 s. and record the test classification accuracy of the worst class after each training epoch, To demonstrate relative insensitivity to the choice of y, we ablate the mixing rate for [0.01, 0.1]. The result is depicted in 7
Figure We see that learn2mix offers a considerable improvement the worst-class classification accuracy metric versus classical, FCL, SMOTE, IS, and CURR training, which matches intuition; the theoretical foundation learn2Mix is to increase the proportion of harder classes during training, which directly translates to stronger results for the most challenging classes.
(a) Imagenette
(b) IMDB
Figure 7: Comparing worst-class model classification accuracies using learn2mix, classical, FCL, SMOTE, IS, and CURR training Imagenette and IMDB, The x-axis is the elapsed [training] time, while the y-axis is the classification accuracy the worst-class.
To illustrate how the mixing parameters converge stable distribution during training (as detailed in Section 2) we train a fully connected network on Mean Estimation (where the Normal, Exponential, and Chi-squared cases have similar variance but the Uniform case is substantially more variable) using Cross-Entropy Loss with learning rate 5e-5 for E' 500 epochs. As depicted in Figure g lear22mix prioritizes the hardest class without overstating differences among the easier ones.
0.55
0.85
0.30
Geurral
0.05
0.00
@csitesaned
Gusttm
Epochs
Figure 8: Evolution of learn2mix mixing parameters across training epochs on Mean Estimation.
18

---

C Dataset Descriptions
C.1
MNIST Dataset
The MNIST (Modified National Institute of Standards and Technology) dataset is a collection of handwritten digits commonly used to train image processing systems. For the MNIST classification result from Section 4.1 the original training dataset, J, comprises N = 60000 samples, wherein the fixed- prceppropottion mixing parameters (for default numerical class ordering of digits from 1 10) are:
α [0.0987, 0.1124, 0.0993, 0.1022, 0.0974, 0.0904, 0.0986, 0.1044, 0.0975, 0.0991|²
The test dataset, K, comprises Nest 10000 samples, with class proportions equivalent to the class proportions in the base MNIST test dataset. For MNIST reconstruction (see Section 4.3). we utilize manual class imbalancing, reducing the number of samples comprising each numerical class - 10 by factor of 5. The original training dataset, J, now contains N 36475 samples, wherein the fixed-proportion mixing parameters (for default numerical class ordering of digits from 1 10) are:
ã 10.1 0.1848, 0.1633, 0.1681, 0.1602, 0. 0.0297, 0.0324, 0.0344, 0.0321, 0.0326]"
We note that the test dataset maintains the same class proportions in the base MNIST test dataset. The features and labels within MNIST are summarized as follows:
Each feature (image) is of size 28 x 28, representing grayscale intensities from 0 to 255.
Target Variable: The numerical class (digit) the image represents, ranging from 10.
C.2 Fashion MNIST Dataset
The Fashion MNIST dataset is a collection of clothing images commonly used to train image processing systems. For the Fashion MNIST classification result from Section 4.1 the original training dataset, J, consists N = 60000 samples, wherein the fixed- exed-prropottion mixing parameters (for default numerical class ordering of clothing from 1 10) are:
ā [0.1, 0.1, 0. 0.1, 0.1, 0.1, 0.1, 0.1, 0.1]" = 10.1)
The test dataset, K, comprises Nest = 10000 samples, with class proportions equivalent to the class proportions in the base Fashion MNIST test dataset. For Fashion MNIST reconstruction (see Section we use manual class imbalancing, reducing the number of samples within each numerical class 10 by factor of 5. The original training dataset J, now has N 36000 samples. The fixed-proportion mixing parameters (for default numerical class ordering of clothing from 10)
are:
ā = (0.1667) (0.0333)13]1 !7""
We note that the test dataset maintains the same class proportions as in the base Fashion MNIST test dataset. The features and labels within Fashion MNIST are summarized as follows:
Each feature (image) is of size 28 28, representing grayscale intensities from to 255.
Target Variable: The numerical class (clothing) the image represents, ranging from 1 to 10.
C.3 CIFAR-10 Dataset
The CIFAR-10 dataset is collection of color images categorized into 10 different classes, and is commonly used to train image processing systems. For the CIFAR-10 classification result in Section the original training dataset, J, comprises N 50000 samples, wherein the pixed-prooottion mixing parameters (for default numerical class ordering of categories from 1 - 10) are:
ā (0.1)1110
The test dataset, K, comprises Nist 10000 samples, with class proportions equivalent to the class
proportions the base CIFAR-10 test dataset. For CIFAR- 11 reconstruction (see Section 4.3p. we use manual class imbalancing, reducing the number of samples in numerical classes 1 - 4, 7 10 by a factor of 10. The original training dataset, J, now has N 14000 samples. The fixed- proporton mixing parameters (for default numerical class ordering of categories from 1 - 10) are:
á (0.0057) (0.3571) (0.035))177'
We note that the test dataset maintains the same class proportions found in the base CIFAR-10 test. dataset. The features and labels within CIFAR-10 are summarized as follows:
19

---

Each feature (image) is of size 32 x 32 x 3, with three color channels (RGB), and size 32 32 pixels for each channel, represented as a grayseale intensity from 0 to 255.
Target Variable: The numerical class (category) the image represents, ranging from 1 to 10.
C.4 Imagenette Dataset
The Imagenette dataset contains a subset of 10 classes from the ImageNet dataset of color images, and is commonly used to train image processing systems. The base Imagenette training dataset, T, comprises N, 9469 samples, and the base Imagenette test dataset, K, comprises Nost = 3925 samples. For the Imagenette classification result in Section 4.1 we utilize manual class imbalancing. Let N; E N be the number samples in each class, i (I...., 10). from I, where N, = - N, We define 6, 1 0.1i, Vi {1, {L..... 10} as the linearly decreasing imbalance factor. Accordingly, the original training dataset, J, has parameters (for default numerical class ordering of categories from 1 10) are:
Ene e = 5207 samples. The fixed- preddprroootion mixing
ā [0.1849] 0.1650, 0.1525, 0.1152, 0. 0.1083, 0.0918, 0.0737, 0.0536, 0.0365, 0.0184)
We note that the test dataset maintains the same class proportions found in the base Imagenette test dataset. The features and labels within Imagenette are summarized as follows:
Each feature (image) is of size 224 224 3, with three color channels (RGB), and size
224 x 224 pixels for each channel, represented as a grayscale intensity from 0 to 255.
Target Variable: The numerical class (category) the image represents, ranging from 1 to 10.
C.5 CIFAR-100 Dataset
The CIFAR-100 dataset is a collection of color images categorized into 100 different classes, and commonly used to train image processing systems. The base CIFAR-100 training dataset, /, has Nj = 50000 samples, and the base CIFAR- 100 test dataset, K, has Nist = 10000 samples. For the CIFAR-100 classification result in Section 4.1 we utilize manual class imbalancing, Let N; N be 100), from I, whereby N₁ = Em= N₁. We define 6 = 1000// 1/110 Vi E {1...., ,00) as the logarithmically decreasing imbalance factor. Accordingly, the original training dataset, J, has N = 100 e = 13209 samples. The fixed- sxed-propottion mixing parameters (for default numerical class ordering of categories from 100) are:
the number of samples in each class, i E {
â [ā1, ag
where: à (GNN)/N, Vi (1,..., 100}
We note that the test dataset maintains the same class proportions found the base CIFAR-100 test dataset. The features and labels within CIFAR-100 summarized follows:
Each feature (image) of size 32 x 32 x 3, with three color channels (RGB), and size 32 x 32 pixels for each channel, represented as grayscale intensity from 0 to 255.
Target Variable: The numerical class (category) the image denotes, ranging from to 100.
C.6 IMDB Dataset
The IMDB dataset is a collection of movie reviews, categorized positive negative in sentiment. We split the IMDB dataset such that the base IMDB training dataset, I, has N2 = 40000 samples, and the base IMDB test dataset, K, consists of Nist 10000 samples. For the IMDB classification result in Section 4.1, we leverage manual class imbalancing, wherein numerical class 1 retains 30% of its samples. Accordingly, the original training dataset, J, has N 26000 samples. The fixed- proporton mixing parameters (for default numerical class ordering of sentiment from 1, 2) are:
= (0.2307, 0.7693]
We note that the test dataset maintains the same class proportions as in the base IMDB test dataset. The features and labels within the IMDB dataset are summarized as follows:
Each feature (review) is tokenized and encoded as a sequence of word indices with a max length of 500 tokens. Sequences are padded or truncated to ensure uniform length.
Target Variable: The numerical class (sentiment) the review represents, either I or 2.
20

---

C.7 Mean Estimation Dataset
The Mean Estimation dataset a synthetic benchmark designed for regression tasks, wherein each example, (zy. comprises a 10-dimensional feature vector, T, of samples from one of four statistical distributions, and the mean, yj. of this distribution. We create an imbalanced original training dataset, J, with N 3000 samples, where J1 has 1000 examples drawn from normal distribution with o = 1, J2 has 1000 examples drawn from an exponential distribution, J3 has 800 examples drawn from chi-squared distribution, and J4 has 200 samples drawn from uniform distribution. The pixeoprrooottion mixing parameters (for numerical ordering of distributions from
1 4) are:
á = [0.333, 0.333, 0.267, 0.067]
The test dataset, K, is created balanced dataset that has 1000 examples from each distribution, wherein Nest = 4000. The Mean Estimation dataset features and labels are summarized as follows:
Each feature (vector of samples) is generated from of four statistical distributions (normal, exponential, chi-squared, uniform). The feature vectors are created by sampling from these distributions with means uniformly drawn from the interval [0, 1] for normal, exponential, and chi-squared distributions, and from [20, 50] for the uniform distribution.
Target Variable: The mean parameter used to generate the vector of samples, representing the underlying expected value of the chosen distribution.
C.8 Wine Quality Dataset
The Wine Quality dataset consists of physicochemical tests on white and red wine samples, and the corresponding quality rating. We treat the wine type (white 1, red = as a categorical variable, wherein k = 2. We split the Wine Quality dataset such that the base Wine Quality training dataset, J, has N = 3248 samples, and the base Wine Quality test dataset, K, has Nust 3249 samples. For the Wine Quality regression result in Section 4.2] we utilize manual class imbalancing, reducing the number of samples in numerical class 1 by factor of 10. The original training dataset, J, now has N = 1043 samples, where the fixed- proopprroootion mixing parameters (for numerical class ordering of wine type from 1, 2) are:
á [0.234, 0.766)7
We note that the test dataset maintains the same class proportions as in the base Wine Quality test dataset. The features and labels within the Wine Quality dataset summarized as follows:
Each feature (physicochemical tests) contains set of test results, and is of size 11 x 1.
Target Variable: The wine quality rating given the set of physicochemical tests.
C.9 California Housing Dataset
The California Housing dataset contains housing data from California and their associated prices. As the ocean proximity variable is categorical (<1H OCEAN 1, INLAND = 2, NEAR BAY = 3, NEAR OCEAN = 4), we denote l: = 4. We split the California Housing dataset such that the base California Housing training dataset, J, has N 10214 samples, and the base California Housing
test dataset, K, has Niest 10214 samples. For the California Housing regression result in Section
we use manual class imbalancing, reducing the number of samples in numerical classes 1, by a Tactor of 20. The original training dataset, J, now has N = 3641 samples. The fixed-proportion mixing parameters (for numerical class ordering of ocean proximity from 4) are:
ā (0.0615, 0.9055, 0.0154, 0.0176)7
We note that the test dataset maintains the same class proportions in the base California Housing test dataset. The features and labels in the California Housing dataset are summarized as follows:
Each feature (housing data) contains various housing attributes, and is of size 8 x 1.
Target Variable: The housing price associated with the housing data.
21

---

D Experiment Details
D.1 Neural Network Architectures
We provide comprehensive descriptions for six different neural network architectures designed for various tasks: classification, regression, and image reconstruction. Each of these architectures were employed to generate the respective empirical results pertaining to the aforementioned tasks.
D.1.1 Fully Connected Networks
We leverage fully connected networks in analysis for regression Mean Estimation, California Housing, and Wine Quality. The network consists of the following layers, wherein = 10 for Mean Estimation, d 11 for Wine Quality, and d for California Housing:
Fully Connected Layer (fc1): Transforms the input features from a d-dimensional space
a 64-dimensional space.
ReLU Activation (relu): Applies the ReLU activation function to the output of fc1.
Fully Connected Layer (fc2): Maps the 64-dimensional representation from relu to 1-dimensional output.
D.1.2 Convolutional Neural Networks
We utilize the LeNet-5 convolutional neural network architecture in our analysis for image classifica- tion on MNIST and Fashion MNIST, The network consists of the following layers:
Convolutional Layer (conv1): Applies a 2D convolution with input channel, output channels, and a kernel size of 5.
ReLUU Activation (relu1): Applies the ReLU activation function to the output of conv1.
Max Pooling Layer (pool1): Performs 2x2 max pooling on the output of relui. Convolutional Layer (conv2): Applies a convolution with input channels, 16 output channels, and a kernel size of 5.
ReLU Activation (relu2): Applies the ReLU activation function to the output of conv2.
Max Pooling Layer (pool2): Performs 2x2 max pooling on the output of relu2.
Flatten Layer: Reshapes the pooled feature maps into ID vector.
Fully Connected Layer (fc1); Maps the flattened vector to a 120-dimensional space.
ReLU Activation (relu3): Applies the ReLU activation function to the output of fc1.
Fully Connected Layer (fc Maps the 120-dimensional input a 84-dimensional space
ReLU Activation (relu4): Applies the ReLU activation function to the output of fc2.
Fully Connected Layer (fc3): Produces a 10-dimensional output for classification.
For image classification on CIFAR-10 and CIFAR-100, 100, we employ an adapted, larger version of the LeNet-5 model, which we call 'Large LeNet'. The network consists of the following layers, wherein 10 for CIFAR-I0 and k: = 100 CIFAR-100.
Convolutional Layer (conv1): Applies 2D convolution with 3 input channels, 16 output channels, and a kernel size 3.
ReLU Activation (relu1): Applies the ReLU activation function to the output of conv1.
Max Pooling Layer (pool1): Performs 2x2 max pooling the output of relu1. Convolutional Layer (conv2): Applies 2D convolution with 16 input channels, 32 output channels, and a kernel size 3.
ReLU Activation (relu2): Applies the ReLU activation function to the output of conv2.
Max Pooling Layer (poo12): Performs 2x2 max pooling on the output of relu2. Convolutional Layer (conv3): Applies convolution with input channels, 64 output channels, and a kernel size 3.
22

---

ReLU Activation (relu3): Applies the ReLU activation function to the output of conv3.
Max Pooling Layer (pool3): Performs 2x2 pooling the output of relu3.
Flatten Layer: Reshapes the pooled feature maps into a ID vector of size 4 x x 64.
Fully Connected Layer (fc1): Maps the flattened vector to a 500-dimensional space.
ReLU Activation (relu4): Applies the ReLU activation function to the output of fcl.
Dropout Layer (dropout1); Applies dropout with p = 0.5 to the output relu4.
Fully Connected Layer (fc2): Produces a k-dimensional d-douuesiidiimmiii
output for classification.
D.2 Mobile Neural Networks
For image classification on CIFAR-10 and CIFAR-100, we also employ the MobileNer- V3 Small architecture. The network consists of the following layers, where 10 CIFAR-10 and 100 for CIFAR-100.
Convolutional Stem (featuresO): 3 input channels, 16 output channels, kernel size 3, stride 2, padding followed by BatchNorm and Hard-Swish activation.
Inverted Residual Block (features1): expansion factor 1, 16 to 16 channels, kernel size 3, stride 2, SE disabled, activation ReLU.
Inverted Residual Block 2 (features2): expansion factor 4.5, 16 to 24 channels, kernel size 3, stride 2, SE disabled, activation ReLU.
Inverted Residual Block (features3): expansion factor 3.67, 24 24 channels, kernel size 3, stride 1, SE disabled, activation ReLU.
Inverted Residual Block 4 (features4): expansion factor 4, 24 40 channels, kernel size 5, stride 2, SE enabled, activation Hard-Swish.
Inverted Residual Block 5 (features5): expansion factor 6, 40 40 channels, kernel size 5, stride 1, SE enabled, activation Hard-Swish.
Inverted Residual Block 6 (features6): expansion factor 6, 40 to 40 channels, kernel size 5, stride 1, SE enabled, activation Hard-Swish.
Inverted Residual Block 7 (features7):
expansion factor 3, 40 to 48 channels, kernel size 5, stride 1, SE enabled, activation Hard-Swish.
Inverted Residual Block (features8): expansion factor 3, 48 to 48 channels, kernel size 5, stride 1, SE enabled, activation Hard-Swish.
Inverted Residual Block (features9): expansion factor 6, 48 96 channels, kernel size 5, stride 2, SE enabled, activation Hard-Swish.
Inverted Residual Block 10 (features10): expansion factor 6, 96 96 channels, kernel size 5, stride 1, SE enabled, activation Hard-Swish.
Inverted Residual Block 11 (features1l): expansion factor 6, 96 96 channels, kernel size 5, stride 1, SE enabled, activation Hard-Swish.
Convolutional Head (features12): 1x] Conv2d from 96 to 576 channels, followed by BatchNorm and Hard-Swish.
Adaptive Average Pooling (features13): global average pool to Ixl.
Conv Head (features14): 1x1 Conv2d from 576 1024 channels, followed by Hard- Swish.
Flatten Layer: reshapes the 1024x1x1 tensor to 1024-dimensional vector.
Fully Connected Layer (classifierO) linear 1024 1024, followed by Hard-Swish.
Dropout Layer (classifier)). dropout with 0.2.
Fully Connected Layer (classifier3) linear 1024 to & for classification.
23

---

D.2.1 Residual Neural Networks
For image classification on Imagenette, we employ the ResNet- 18 residual neural network architecture, which consists of the following layers:
Convolutional Layer (conv1): Applies 7x7 convolution with input channels, 64 output channels, and stride of 2.
Batch Normalization (bn1): Normalizes the output of conv1.
ReLU Activation (relu): Applies the ReLU activation function to the output bn1.
Max Pooling Layer (maxpool): Performs 3x3 max pooling with stride of 2 on the output relu.
Residual Layer (layer1): Contains two residual blocks, each with 64 channels.
Residual Layer 2 (layer2): Contains two residual blocks, each with 128 channels.
Residual Layer (layer3): Contains two residual blocks, each with 256 channels.
Residual Layer (layer4): Contains two residual blocks, each with 512 channels.
Average Pooling (avgpool): Applies adaptive average pooling to reduce the spatial dimen-
sions 1xl.
Fully Connected Layer (fc): Produces a 10-dimensional output for classification.
D.2.2 Transformer Models
For sentiment classification on IMDB Sentiment Analysis, we leverage a transformer architecture, which consists of the following layers:
Embedding Layer (embedding): Maps input tokens to 64-dimensional embeddings.
Positional Encoding (pos_encoder) Adds positional information to the embeddings with maximum sequence length of 500.
Transformer Encoder (transformer eew.reenneeeerrrrr Applies transformer encoder with 1 layer, 4 attention heads, and hidden dimension of 128.
Pooling Layer (pool): Averages the transformer outputs across the sequence length. Dropout Layer (dropout): Applies dropout with probability 0. to pooled output.
Fully Connected Layer (fc1): Maps the 64-dimensional pooled vector 32-dimensional
space.
ReLU Activation (relu1): Applies the ReLU activation function to the output of fc1.
Fully Connected Layer (fc2): Maps the 32-dimensional input to 2 output classes.
D.2.3
Autoencoder Models
For image reconstruction on MNIST, Fashion MNIST, and CIFAR-10 we employ an autoencoder. This network consists of the following layers, where d = 784 for MNIST and Fashion MNIST, and 3072 for CIFAR-10:
Fully Connected Layer (fc1): Transforms the input fcatures from a d-dimensional space a 128-dimensional space.
ReLU Activation (relu1): Applies the ReLU activation function to the output of fc1.
Fully Connected Layer (fc2): Reduces the 128- dimensional representation to 32- dimensional encoded vector.
Fully Connected Layer (fc3); Expands the 32-dimensional encoded vector back to 128-dimensional space.
ReLU Activation (relu1): Applies the ReLU activation function to the output of fc3.
Fully Connected Layer (fc4): Maps the 128-dimensional representation back to the original d-dimensional space.
Sigmoid Activation (sigmoid1): Applies the Sigmoid activation function to ensure the output values are between 0 and 1.

---

D.3
Focal Training
For the classification tasks outlined in Section 4.1 we compare learn2mix and classical training with focal boss-based neural network training (focal training). Let å E [0. 1] denote the vector of fixed- proportion mixing parameters, let (()) E IR& denote the vector of class-wise cross entropy losses at time t, and let w E IR denote the vector of class-wise weighting factors, where E [I....,k}:
[1/(G,N)]
x k. W/ Σ [1)
(62)
The vector of predicted class-wise probabilities, p E [0, 11 is given by p- -xxp (-L(0)) and we let R20 the focusing parameter. The focal loss at time (0", w) E 1Rz0. is given by:
(0',
E
(-w)) p.) log(p:).
(63)
k
Per the recommendations in [Lin et al. 2017), choose I = 2 in compiling the empirical results.
D.4 SMOTE Training
For the classification tasks outlined in Section 4.1 with neural networks trained on SMOTE-oversamppe original training dataset, where the number of samples in each class, E (1......k k} given by N. After applying SMOTE oversampling, we obtain new training dataset, JSMonE, with uniform class proportions, a; SMMOTE = t. Vi E kl,...,k). The total number samples in JSMOTE, given by:
we also compare learn2mix and classical training
datasets (SMOTE training). Let J denote the
max N;N) A-SMOTE_ = (E(......))
x k:
(64)
In the original training dataset, J. we use batch size of M, resulting in P N/M total batches. For consistency with learn2mix and classical training (see Section we perform SMOTE training on P batches of size M from the SMOTE oversampled training dataset, JSMOTE, during each epoch.
D.5 IS Training
For the classification tasks outlined in Section 4. we compare learn2mix and classical training with
importance sampling-base neural network training (IS training) adapted from (Katharopoulos and
Fleuret
2018 and [Johansson and Lindberg 2022). Let J denote the original training dataset, and let E R M denote the vector of individual cross-entropy losses at time f on a batch of size M
(0*
ind
drawn uniformly from J. We normalize these losses to sampling probabilities, P5- E [0, 1]. sample without replacement a subset of size b = M/2 according to {pi}. and update the model by taking gradient step on the average loss over that subset, where:
(0t)
Dj
and:
Lis(0') = Σ
CM i, (0').
(65)
1 C. (01)
r=1
In the original training dataset J, we use a batch size of M, resulting in P = N/M total batches, and perform IS training on batches of size M during each epoch.
D.6 CURR Training
For the classification tasks outlined in Section we compare learn2mix and classical training with curriculum learning- based neural network training (CURR) following the self-taught scoring and fixed exponential pacing scheme of [Hacohen and Weinshall 2019). Let J be the original training dataset, and denote by 8; 1 P. the self-taught score of sample j. where βú is the network's confidence in the correct label after preliminary convergence training on uniform mini-batcher (this warm-up stage is used only compute (5,) and is not included in our reported CURR timings, nor is any analogous stage required for learn2mix). We sort the samples by increasing &j (easiest first) to obtain sorted indices (11,-.., ). At epoch let the curriculum fraction be:
frac(t) min (starting (starting_percent percent x inc
1.0).
(66)
25

---

with starting percent = 0.5, inc = 1.2, and step_ length = 10. We form a curriculum subset of size [frac(t) N] by taking the first [frac(t) NJ sorted indices, and train on mini-batches of size M. The curriculum loss at time is then:
(frac()) Nj
CoURn(0') [Faa(()) NJ
Σ
Lind, i, (0)),
(67)
where Cnnd, (f") E IR is the individual cross-entropy loss sample and each epoch processes (frac(t) N /M batches of size M.
D.7 Neural Network Training Hyperparameters
The relevant hyperparameters used to train the neural networks outlined in Section D.I are given in Table 3
All results presented in the main text were produced using these hyperparameter choices.
Table 3: Neural network training hyperparameters (grouped by task).
Learning Rate (1))
Mixing Rate (?)
Batch
Dataset
Task
Optimizer
(Leam2Mix)
Size (MV)
MNIST
Classification
Adam
0.0001
0.1
1000
Fashion MNIST
Classification
Adam
0.0001
0.5
1000
CIFAR-10
Classification
Adam
7.0e-5
0.1
1000
Imagenette
Classification
Adam
1.0e-6
0.1
100
CIFAR-100
Classification
Adam
0.0001
0.5
5000
IMDB
Classification
Adam
0.0001
0.1
500
Mean Estimation
Regression
Adam
5.0e-5
0.01
500
Wine Quality
Regression
Adam
0.0001
0.05
100
California Housing
Regression
Adam
5.0e-5
0.01
1000
MNIST
Reconstruction
Adam
0.0005
0.1
1000
Fashion MNIST
Reconstruction
Adam
1.0e-5
0.1
1000
CIFAR-10
Reconstruction
Adam
1.0e-5
0.1
1000
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The paper presents new framework for accelerating neural network conver- gence. We provide comprehensive empirical results and theoretical guarantees to validate this claim.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: The performance gains (large or limited) afforded by learn2mix are explicitly quantified in the empirical results section, and all methods, alongside ablation studies, are thoroughly discussed in the appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
27

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We provide theorems (with all relevant terms defined) in the main text, along- side comprehensive proofs appendix to verify the proposed theorems.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Alongside the loss function and optimizer details presented in the main text, all neural network architectures and training hyperparameters are discussed and tabulated in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibilityy can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
28

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The complete code for reproducing all the empirical results for learn2mix are provided in the supplementary materials. All neural network architectures and training hyperparameters are also provided appendix.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
www.amissssssoooioioooommomoe oonioiicnlyy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
CooeeSumeeeeeeeeeeeeee ssi cerionPollin mollicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The appendix details all neural network architectures utilized to generate the results presented in the main text and in the ablation studies. All hyperparameter choices either explicitly specified in the main text/appendix, or tabulated in the appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All empirical results in the main text and the appendix include confidence intervals the figures and tables to explicitly declare the statistical significance of all experiments.
Guidelines:
29

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
The compute resources (GPUs) used to produce the empirical results in the main text are specified in the empirical results section.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The presented research conforms in all aspects with the NeurlPS Code of Ethics, and the authors have reviewed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
30

---

Justification:
Accelerating neural network convergence in resource constrained regimes is important capability to ensure fast and efficient neural network training the adoption of learn2mix can save compute cost and accelerate training. We find no negative societal impacts of our work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeett of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: All the code is provided in the appendix and the final version will be maintained in a GitHub repository by the authors. The authors contact information will also be provided in the final version to prevent misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All relevant code and models used in the paper have been properly cited.
Guidelines:
31

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thcode. com/ com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The appendix contains comprehensive descriptions of all considered neural network architectures and modified datasets used to generate the empirical results.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
32

---

Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
33


Reviews:
[{"review_id": "796c1j4SIV", "reviewer": "Reviewer_Ppx6", "summary": "This paper proposes a form of adaptive class rebalancing that uses ideas from imbalanced classification for a form of curriculum learning. Some theoretical results are presented in the strongly-convex smooth setting, and experimental results show relatively small but consistent improvements over all considered competitors for several small image and tabular classification problems.", "questions": "- What happens to $\\alpha$ during the training process? That is, you have an intuition that it orders the difficulty between classes or something along those lines, but...is that what actually happens? I'm very surprised that there were no experiments considering this.\n\n- What about cases where there is a real fundamental difference in class difficulty? I don't think this is particularly the case in any of the datasets you considered here, but e.g. standard ImageNet classification has some classes which are far easier to confuse with one another, like the fine-grained different breeds of dogs, or certain inherently ambiguous classes.\n\n- How do the per-class losses end up, compared to the standard case? The aggregate case is lower, but e.g. if you have a relatively rare class and a biggish $\\gamma$, then I wonder how often getting lucky early on could lead to poor performance on that class in particular.\n\n- Consider a situation where you can afford large batch sizes (in fact, you're running smallish models on a 3090, so you're basically in that situation). How would your sampling-based weighting scheme compare to an adaptive reweighting approach, where you still get a uniform / the dataset level of mixtures among classes, but the loss of each is reweighted?", "limitations": "Societal impacts: it's conceivable this could have some kind of fairness type impacts, but I don't think that's especially likely or necessary to evaluate in this kind of paper.\n\nLimitations of the proposed approach and analysis: not really discussed as thoroughly as they could be.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The\n proposed method is simple, easy to implement, and reasonably sensible. It also seems to consistently, though not earth-shatteringly, help on the settings you tried, although the considered experiments are a little more \"2017-era\" than a typical empirically-minded NeurIPS paper these days.\n\nIt would be worth mentioning, for readers who are not already familiar, that (a) a substantial portion (though certainly not all) of your theorems are textbook-standard results about the strongly convex smooth optimization setting, and (b) that neither of these conditions (particularly not strong convexity, but not smoothness either) apply to the neural networks you actually consider in experiments. Both are fine, but it would be better to be clear about this.", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Ppx6, \n\nThank you for your valuable feedback. Below, we have addressed your comments\n\n> It would be worth mentioning, for readers who are not already familiar, that (a) a substantial portion (though certainly not all) of your theorems are textbook-standard results about the strongly convex smooth optimization setting, and (b) that neither of these conditions (particularly not strong convexity, but not smoothness either) apply to the neural networks you actually consider in experiments. Both are fine, but it would be better to be clear about this.\n\nThank you for this suggestion. We will add a brief note in Section 2 stating that the propositions and their corollaries rest on standard results for strongly-convex objectives; we adopt this classical baseline intentionally because it lets us isolate how adaptive mixing influences convergence under well-understood conditions, before demonstrating the same mechanism’s empirical effectiveness for non-convex objectives via the architectures considered in Section 4.\n\n> What about cases where there is a real fundamental difference in class difficulty? I don't think this is particularly the case in any of the datasets you considered here, but e.g. standard ImageNet classification has some classes which are far easier to confuse with one another, like the fine-grained different breeds of dogs, or certain inherently ambiguous classes.\n\nWe provide empirical results on CIFAR-100 in Section 4 of the main text and in Section B of the Appendix, specifically to expose fine-grained confusion. The 100 classes in CIFAR-100 are arranged in 20 superclasses that group visually similar categories (e.g., fish types such as aquarium fish, flatfish, and trout). Learn2mix updates the class-sampling proportions, $\\alpha$, each epoch toward the normalised class-wise loss vector, so intrinsically harder to distinguish classes are automatically sampled more often during training. Empirically, this yields faster convergence: on CIFAR-100, the learn2mix-trained MobileNet-V3 reaches 50% test accuracy 20 seconds sooner than any fixed-proportion baseline, and on Imagenette (a compute-friendly subset of ImageNet), learn2mix lifts the worst-class accuracy curve throughout training (see Section B of the Appendix).\n\n> What happens to $\\alpha$ during the training process? That is, you have an intuition that it orders the difficulty between classes or something along those lines, but...is that what actually happens? I'm very surprised that there were no experiments considering this.\n\nWe note that the asymptotic behavior of $\\alpha$ is motivated by more than just intuition. Per Section 2 of the main text, learn2mix converges to a stable distribution that reflects the relative difficulty of each class under the optimal parameters (under standard strong convexity assumptions [1], [2]). Regarding experiments, on our Mean Estimation benchmark (where the Normal, Exponential, and Chi-squared cases have similar variance but the Uniform case is _substantially_ more variable), $\\alpha$ converges to the values given in Table 1. These proportions confirm that learn2mix prioritizes the hardest class without overstating differences among the easier ones. We will include this result in Appendix Section B of the revised manuscript to illustrate how $\\alpha$ empirically orders class difficulty.\n\n**Table 1: Mixing parameter evolution over time on Mean Estimation benchmark** \n\n| Epoch ($t$)   | $\\alpha_{\\text{Normal}}$ | $\\alpha_{\\text{Exponential}}$ | $\\alpha_{\\text{Chi-Squared}}$ | $\\alpha_{\\text{Uniform}}$ |\n| ------------- | -------- | ------- | --------- | --------|\n| $t = 0$       | 0.33                      | 0.33           | 0.264          | 0.076                     |\n| $t = 50$      | 0.1997                    | 0.1997         | 0.1598         | 0.4407                    |\n| $t = 100$     | 0.1243                    | 0.1226         | 0.1001         | 0.653                     |\n| $t = 150$     | 0.0865                    | 0.0794         | 0.0717         | 0.7624                    |\n| $t = 200$     | 0.0630                    | 0.0533         | 0.0546         | 0.8291                    |\n| $t = 250$     | 0.0479                    | 0.0372         | 0.0419         | 0.8729                    |\n| $t = 300$     | 0.0382                    | 0.0272         | 0.0337         | 0.9008                    |\n| $t = 350$     | 0.0325                    | 0.0210         | 0.0297         | 0.9168                    |\n| $t = 400$     | 0.0299                    | 0.0174         | 0.0273         | 0.9253                    |\n| $t = 450$     | 0.0280                    | 0.0151         | 0.0240         | 0.9330                    |\n| $t = 500$     | 0.0270                    | 0.0134         | 0.0227         | 0.9369                    |\n| $t = 550$     | 0.0258                    | 0.0121         | 0.0236         | 0.9385                    |\n| $t = 600$     | 0.0261                    | 0.0123         | 0.0233         | 0.9384                    |\n\n> How do the per-class losses end up, compared to the standard case? The aggregate case is lower, but e.g. if you have a relatively rare class and a biggish $\\gamma$, then I wonder how often getting lucky early on could lead to poor performance on that class in particular.\n\nIn Section B of the Appendix, we present the worst-class classification accuracy on Imagenette as an additional metric to gauge the efficacy of learn2mix for imbalanced classification settings. The class with the worst _per-class_ loss is recorded for each elapsed training time step, wherein the learn2mix-trained ResNet-18 model ($\\gamma = 0.1$) observes improved learning dynamics over existing training approaches. Regarding $\\gamma$, as we recommend in Section 4 of the main text, choosing $\\gamma \\in [0.01, 0.5]$ yields the best performance. We provide additional empirical results below for the worst per-class classification accuracy on Imagenette and IMDB for $\\gamma = 0.05$ and a “biggish” (per the reviewer’s comments) $\\gamma = 0.5$ to demonstrate relative insensitivity to the choice of $\\gamma$. We will include this result in the revised manuscript.\n\n**Table 2: Worst-class test performance comparison for ResNet-18 on Imagenette** \n\n| Method | Worst-Class Accuracy (57.5 s) |  Worst-Class Accuracy (115.0 s) |  Worst-Class Accuracy (230.0 s) |\n|--------|-----------------------------------:|-----------------------------------:|-----------------------------------:|\n| L2M ($\\gamma = 0.05$)    | 0.0000 %         | 2.3866 %      | 14.1623 %                  |\n| L2M ($\\gamma = 0.1$)      | **1.3149 %**    | **10.9318 %**    | 14.1451 %                 |\n| L2M ($\\gamma = 0.5$)    | 0.2542 %         | 3.2815 %      | **15.6304 %**                  |\n| CL     | 0.7779 %                         | 0.6205 %                         | 0.6325 %                         |\n| FCL    | 0.0000 %                         | 0.0000 %                         | 4.8162 %                         |\n| SMOTE  | 0.0000 %                         | 0.0000 %                         | 8.9119 %                         |\n| IS     | 0.0000 %                         | 0.0000 %                         | 0.7924 %                         |\n| CURR   | 0.0000 %                         | 0.2482 %                         | 2.8140 %                         |\n\n**Table 3: Worst-class test performance comparison for Transformer on IMDB** \n\n| Method | Worst-Class Accuracy (38.0 s) |  Worst-Class Accuracy (75.0 s) |  Worst-Class Accuracy (150.0 s) |\n|--------|-----------------------------------:|-----------------------------------:|-----------------------------------:|\n| L2M ($\\gamma = 0.05$) | 46.2067 % | **77.4133 %**   | 79.2200 %  |\n| L2M ($\\gamma = 0.1$) | 40.2867 % | 76.6267 %    | 80.7400 %   |\n| L2M ($\\gamma = 0.5$) | **60.3733 %** | 74.4467 %      | **81.8533 %**  |\n| CL     | 0.0000 % | 47.6400 %  | 67.3933 % |\n| FCL    | 0.0000 %   | 34.5333 % | 60.7200 %  |\n| SMOTE  | 11.3200 % | 28.7867 %  | 62.5400 % |\n| IS     | 24.8467 %   | 57.7200 %  | 69.6267 % |\n| CURR   | 0.0000 %   | 48.4400 %  | 66.0333 %  |\n\n> Consider a situation where you can afford large batch sizes (in fact, you're running smallish models on a 3090, so you're basically in that situation). How would your sampling-based weighting scheme compare to an adaptive reweighting approach, where you still get a uniform / the dataset level of mixtures among classes, but the loss of each is reweighted?\n\nWe agree that an adaptive loss-reweighting scheme is a natural baseline. In fact, our empirical results in Section 4 of the main text and Section B of the Appendix include the class-balanced focal loss (FCL) for this reason. Focal loss keeps class mixtures fixed while adaptively reweighting each class's contribution to the total loss, as the reviewer has proposed. Because focal loss relies on class-level weighting factors, $\\omega_i$, that can differ substantially across classes, its stochastic-gradient estimates generally exhibit higher variance. By contrast, learn2mix _samples_ more frequently from high-loss classes, a strategy known to reduce gradient variance in similar works on importance sampling [3].\n\nThank you again for your feedback, which has been instrumental in improving our paper.\n\n**References**:\n\n[1] Guanghui Wang, Shiyin Lu, Quan Cheng, Wei wei Tu, and Lijun Zhang. SAdam: A variant of adam for strongly convex functions. In International Conference on Learning Representations, 2020.\n\n[2] Arindam Banerjee, Pedro Cisneros-Velarde, Libin Zhu, and Misha Belkin. Restricted Strong Convexity of Deep Learning Models with Smooth Activations. In International Conference on Learning Representations, 2023.\n\n[3] Angelos Katharapolous, and Francois Fleuret. Not All Samples Are Created Equal: Deep Learning with Importance Sampling. In International Conference on Machine Learning, 2018."}, {"review_id": "PKiHKka3z0", "reviewer": "Reviewer_HCei", "summary": "This paper introduces learn2mix, a training strategy that dynamically adjusts class proportions within batches based on real-time class-wise error rates to accelerate convergence. Unlike classical training that maintains fixed class proportions mirroring the dataset distribution, learn2mix continuously adapts batch composition by emphasizing classes with higher instantaneous error rates. The authors provide theoretical convergence guarantees under strong convexity and Lipschitz continuity assumptions, and demonstrate consistent empirical improvements across classification, regression, and reconstruction tasks on multiple benchmark datasets.", "questions": "1. **Theoretical Gap**: How do the authors reconcile the restrictive theoretical assumptions with the practical success of neural networks, where these assumptions clearly don't hold? Can a tighter analysis be provided for the non-convex case?\n\n2. **Computational Overhead**: What is the computational cost of the dynamic batch construction and class-wise loss computation? How does this overhead compare to the time savings from faster convergence?\n\n3. **Scalability**: How does the method scale to datasets with very large numbers of classes?", "limitations": "The authors adequately acknowledge some limitations, but could strengthen this discussion. The gap between theoretical assumptions and practical applicability should be emphasized more prominently. Additionally, the paper would benefit from explicit discussion of computational overhead, hyperparameter sensitivity, and scenarios where the method might be less effective. \n\nThe evaluation is also limited to relatively small benchmark datasets and does not demonstrate scalability to larger, more challenging datasets. Furthermore, the comparison baselines for imbalanced learning are dated. More recent advances in imbalanced learning and adaptive training strategies from the past few years are not considered, which limits the assessment of learn2mix's competitiveness against state-of-the-art approaches.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n**Clear Problem Motivation**: The paper addresses a well-motivated problem in neural network training. The intuition that harder classes should receive more attention during training is compelling and well-articulated.\n\n**Solid Theoretical Foundation**: The authors provide rigorous convergence analysis showing that under appropriate conditions (strong convexity, Lipschitz continuity), learn2mix converges faster than classical training.\n\n**Consistent Performance Gains**: The results demonstrate consistent improvements across different settings, with learn2mix achieving faster convergence in nearly all evaluated scenarios. The improvements appear robust to architectural choices, optimizers, and hyperparameter settings based on the ablation studies.\n\n\n## Weaknesses\n\n**Narrow Baseline Comparison**: While multiple baselines are included, the comparison relies on dated methods for imbalanced learning (SMOTE from 2002, Focal Loss from 2017, curriculum learning variant from 2019). Recent advances in imbalanced learning and adaptive training strategies are not considered, limiting assessment of competitiveness against state-of-the-art approaches.\n\n**Limited Scale Evaluation**: The evaluation is restricted to relatively small benchmark datasets (MNIST, CIFAR-10/100, etc.) without demonstration on larger, more challenging datasets. \n\n\n**Restrictive Theoretical Assumptions**: The convergence guarantees rely on strong convexity and Lipschitz continuity assumptions that rarely hold for neural networks in practice.\n\n**Limited Algorithmic Novelty**: The core insight of emphasizing difficult examples is well-established in machine learning literature. While the specific implementation of adaptive batch composition is somewhat novel, the fundamental approach is incremental rather than groundbreaking.\n\n**Lack of Computational Analysis**: The paper provides no analysis of computational overhead. The dynamic batch construction and class-wise loss computation likely introduce non-negligible computational costs that could offset the benefits of faster convergence.", "was_revised": true, "final_justification": "**Computational overhead:** Measuring wall-clock time adequately addresses this concern.\n\n**Baselines:** Your justification doesn't resolve the core issue. Comparing against 2002-2019 methods in 2025 is problematic regardless of their \"wide adoption.\" Recent work in adaptive sampling and curriculum learning would provide more meaningful comparisons.\n\n**Theory-practice gap:** Saying strong convexity assumptions are \"standard\" sidesteps the real problem. These assumptions are violated in virtually all neural network settings, making the theoretical guarantees largely irrelevant. This limitation needs honest acknowledgment.\n\n**Scale:** You didn't demonstrate scalability to large datasets or many classes. Wall-clock comparisons on small benchmarks don't address whether the method works on datasets with thousands of classes.\nThe additional ablation results are useful, but the fundamental concerns about limited evaluation scope and outdated baselines remain unaddressed.", "rebuttal": "Dear Reviewer w9yV,\n\nThank you for your helpful feedback. Below, we have addressed your comments.\n\n> While multiple baselines are included, the comparison relies on dated methods for imbalanced learning (SMOTE from 2002, Focal Loss from 2017, curriculum learning variant from 2019). Recent advances in imbalanced learning and adaptive training strategies are not considered, limiting assessment of competitiveness against state-of-the-art approaches.\n\nOur chosen benchmarks intentionally cover representative methods that have achieved wide acceptance and continued relevance in the machine learning community. These benchmarks cover the principal methods of imbalanced learning, including class-balanced loss functions (2017), importance sampling (2022/2018), oversampling (2017/2002), and curriculum learning (2019), to provide a comprehensive comparison. While newer approaches continue to emerge, the selected benchmarks remain broadly adopted in practice.\n\n> The evaluation is restricted to relatively small benchmark datasets (MNIST, CIFAR-10/100, etc.) without demonstration on larger, more challenging datasets. How does the method scale to datasets with very large numbers of classes?\n\nInvestigating the efficacy of learn2mix in these settings remains an important aspect of our ongoing research. For the present study we intentionally chose benchmarks that are canonical in the imbalanced learning literature, admit _controllable_ imbalance factors, and can be reproduced without access to large clusters. CIFAR-100-LT (using a logarithmically decaying imbalance factor to exhibit long-tail behavior) and Imagenette (a subset of the high-dimensional ImageNet dataset) with stepwise imbalance satisfy these criteria and are standard in existing literature [1].\n\n> The convergence guarantees rely on strong convexity and Lipschitz continuity assumptions that rarely hold for neural networks in practice. How do the authors reconcile the restrictive theoretical assumptions with the practical success of neural networks, where these assumptions clearly don't hold? Can a tighter analysis be provided for the non-convex case?\n\nWe note that assumptions like strong convexity are standard in the machine learning literature when deriving convergence rates for gradient-based optimization methods [2], [3]. These assumptions provide a manageable framework to obtain rigorous and provable insights into the behavior of optimization algorithms. While the strong convexity assumption may not hold strictly in practical neural network training, our theoretical findings offer valuable intuition about the convergence properties of learn2mix. Moreover, our empirical results demonstrate that learn2mix effectively accelerates convergence in real-world non-convex settings, reinforcing the practical relevance of our work.\n\n> The core insight of emphasizing difficult examples is well-established in machine learning literature. While the specific implementation of adaptive batch composition is somewhat novel, the fundamental approach is incremental rather than groundbreaking.\n\nWhile prioritizing hard examples is a long-standing idea, our contribution lies in _how_ this prioritization is realized. Learn2mix is the first method, to our knowledge, that adapts _class proportions_ in real-time during training via a provably convergent bilevel optimization procedure, while also retaining the simplicity and efficiency of vanilla SGD. We bridge this theory with rigorous empirical validation, demonstrating that learn2mix accelerates convergence across various classification, regression, and reconstruction benchmarks.\n\n> The paper provides no analysis of computational overhead. The dynamic batch construction and class-wise loss computation likely introduce non-negligible computational costs that could offset the benefits of faster convergence. What is the computational cost of the dynamic batch construction and class-wise loss computation? How does this overhead compare to the time savings from faster convergence?\n\nThe manuscript _does_ measure and report computational cost: the classification results in Section 4 of the main text and all empirical results in Section B of the Appendix consider _elapsed training time_ rather than epoch count. Each experiment was run for $\\underline{E}$ seconds (wall-clock time) on the same NVIDIA GeForce RTX 3090 GPU, which includes the overhead of dynamic batch construction and per-class loss computations; because the evaluation metric is end-to-end wall-clock time on fixed hardware, learn2mix’s overhead is fully accounted for.\n\n> The authors adequately acknowledge some limitations, but could strengthen this discussion. The gap between theoretical assumptions and practical applicability should be emphasized more prominently. Additionally, the paper would benefit from explicit discussion of computational overhead, hyperparameter sensitivity, and scenarios where the method might be less effective.\n\nThank you for your comments. As detailed above, we report end-to-end wall-clock training time on a fixed NVIDIA GeForce RTX 3090 GPU, so any overhead from dynamic batch construction or class-wise loss computation is fully reflected in the results shown in Section 4 of the main text and Section B of the Appendix. The latter section also contains comprehensive ablations over batch size, learning rate, network architecture, and optimizer. To further demonstrate robustness, we provide worst per-class accuracy results on Imagenette and IMDB for $\\gamma \\in \\\\\\{0.05, 0.1, 0.5 \\\\\\}$ in Table 1 and Table 2 (we will include this result in the revised manuscript). These results confirm minimal sensitivity to this parameter. We recall that in Section 4 of the main text, we recommend choosing $\\gamma \\in [0.01, 0.5]$, as it yields similar improvements across all tasks.\n\n**Table 1: Worst-class test performance comparison for ResNet-18 on Imagenette** \n\n| Method | Worst-Class Accuracy (57.5 s) |  Worst-Class Accuracy (115.0 s) |  Worst-Class Accuracy (230.0 s) |\n|--------|-----------------------------------:|-----------------------------------:|-----------------------------------:|\n| L2M ($\\gamma = 0.05$)    | 0.0000 % | 2.3866 %  | 14.1623 %  |\n| L2M ($\\gamma = 0.1$)      | **1.3149 %**    | **10.9318 %**    | 14.1451 %  |\n| L2M ($\\gamma = 0.5$)    | 0.2542 %   | 3.2815 %   | **15.6304 %** |\n| CL     | 0.7779 %  | 0.6205 % | 0.6325 % |\n| FCL    | 0.0000 % | 0.0000 %  | 4.8162 % |\n| SMOTE  | 0.0000 %  | 0.0000 %  | 8.9119 %  |\n| IS     | 0.0000 %  | 0.0000 %  | 0.7924 %  |\n| CURR   | 0.0000 % | 0.2482 %  | 2.8140 %  |\n\n**Table 2: Worst-class test performance comparison for Transformer on IMDB** \n\n| Method | Worst-Class Accuracy (38.0 s) |  Worst-Class Accuracy (75.0 s) |  Worst-Class Accuracy (150.0 s) |\n|--------|-----------------------------------:|-----------------------------------:|-----------------------------------:|\n| L2M ($\\gamma = 0.05$)    | 46.2067 %  | **77.4133 %**      | 79.2200 %  |\n| L2M ($\\gamma = 0.1$)      | 40.2867 % | 76.6267 %    | 80.7400 %   |\n| L2M ($\\gamma = 0.5$)    | **60.3733 %**  | 74.4467 %      | **81.8533 %** |\n| CL     | 0.0000 %  | 47.6400 % | 67.3933 %   |\n| FCL    | 0.0000 %  | 34.5333 %  | 60.7200 %   |\n| SMOTE  | 11.3200 %  | 28.7867 %  | 62.5400 %   |\n| IS     | 24.8467 % | 57.7200 %  | 69.6267 %   |\n| CURR   | 0.0000 %  | 48.4400 % | 66.0333 %  |\n\nRegarding limitations, we note that the application of learn2mix to regression and reconstruction tasks necessitates samples to be partitionable into groups by similarity (e.g., via a categorical variable taking $k$ values). We will clarify this in the revised manuscript.\n\n> The evaluation is also limited to relatively small benchmark datasets and does not demonstrate scalability to larger, more challenging datasets. Furthermore, the comparison baselines for imbalanced learning are dated. More recent advances in imbalanced learning and adaptive training strategies from the past few years are not considered, which limits the assessment of learn2mix's competitiveness against state-of-the-art approaches.\n\nAs we detailed previously, for the present study, we intentionally restricted ourselves to CIFAR-100-LT (logarithmically decaying imbalance), Imagenette (stepwise imbalance), and similar benchmarks because they are canonical in the imbalanced learning literature [1], admit controllable imbalance factors, and can be reproduced without access to large compute resources. Our wall-clock results on an NVIDIA GeForce RTX 3090 GPU show that learn2mix introduces no scaling bottlenecks beyond standard SGD for extensions to larger datasets.\n\nThank you again for your detailed feedback, which has been significant in helping enhance our paper.\n\n**References:**\n\n[1] Zhang, Yongshun, et al. \"Bag of tricks for long-tailed visual recognition with deep convolutional neural networks.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 35. No. 4. 2021.\n\n[2] Guanghui Wang, Shiyin Lu, Quan Cheng, Wei wei Tu, and Lijun Zhang. SAdam: A variant of adam for strongly convex functions. In International Conference on Learning Representations, 2020.\n[3] Arindam Banerjee, Pedro Cisneros-Velarde, Libin Zhu, and Misha Belkin. Restricted Strong Convexity of Deep Learning Models with Smooth Activations. In International Conference on Learning Representations, 2023."}, {"review_id": "v2pMcYS7Cp", "reviewer": "Reviewer_wW5J", "summary": "This paper proposes an adaptive training strategy, learn2mix, which dynamically adjusts the proportion of classes within a batch based on real-time class error rates. The proposed method presents accelerated convergence in classification, regression, and reconstruction tasks, and is particularly effective in resource-constrained and class-imbalanced situations. Extensive experiments demonstrate the effectiveness of the proposed method.", "questions": "1. Whether the proposed training strategy is applicable to other network architectures besides CNN and Transformer.\n2. Whether the sequence length in Transformers impacts adaptive batching.", "limitations": "Please refer to the weakness.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. The proposed adaptive training strategy with real-time class-wise error rates is interesting.\n2. This paper builds a unified framework applicable to classification, regression, and reconstruction tasks.\n3. This paper provides a comprehensive theoretical foundation for the proposed methodology.\n4. The authors provide clear algorithmic implementations for reproduction.\n\n**Weaknesses**\n1. The authors conducted experiments using classical CNNs and Transformers, while lacking validation of recently advanced architectures such as diffusion models and Mamba.\n2. The computational overhead of real-time error calculation for massive k is unquantified.\n3. The experiments are conducted on small-scale datasets, lacking validations on large-scale datasets, such as ImageNet-21k.", "was_revised": true, "final_justification": "Although the authors do not provide evidence that the proposed method is equally applicable to other architectures, I still believe that the method is interesting and innovative enough to be published at NIPS. However, I suggest that the authors conduct experiments to prove the effectiveness of the proposed method on diffusion models and Mamba. I would like to keep my initial score.", "rebuttal": "Dear Reviewer wW5J, \n\nThank you for your constructive feedback. Below, we have provided responses to your comments.\n\n> The authors conducted experiments using classical CNNs and Transformers, while lacking validation of recently advanced architectures such as diffusion models and Mamba.\n\nWe consider standard backbones including ResNet‑18, MobileNet‑V3, and Transformers, across classification, regression, and reconstruction tasks to demonstrate the efficacy of learn2mix under diverse yet reproducible settings. The empirical results in Section 4 of the main text and Section B of the Appendix show that performance gains are consistent, indicating that learn2mix is largely architecture-agnostic. We note that learn2mix does not assume a particular model family. Thus, the U‑Nets used in diffusion models and sequence models such as Mamba can be trained with the same procedure. Exploring domain-specific architectures is an important extension of learn2mix that we plan to explore in future work.\n\n> The computational overhead of real-time error calculation for massive k is unquantified.\n\nWe note that the computational overhead of real-time error calculation is quantified in our empirical results for classification tasks in Section 4 of the main text, and in Section B of the Appendix. All reported results, including those for CIFAR-100 (where we consider a large value of $k=100$), are benchmarked in terms of _elapsed wall-clock training time_ on an NVIDIA GeForce RTX 3090 GPU, which accounts for all per-class error computations. As such, the overhead is fully integrated into our results.\n\n> The experiments are conducted on small-scale datasets, lacking validations on large-scale datasets, such as ImageNet-21k.\n\nFor the present study, we intentionally restricted ourselves to CIFAR-100-LT (using a logarithmically decaying imbalance factor to exhibit long-tail behavior), Imagenette (using a stepwise imbalance factor), and similar benchmarks because they are canonical in the imbalanced learning literature [1], admit controllable imbalance factors, and can be reproduced without access to large compute resources. Imagenette itself is a carefully chosen subset of ImageNet and reflects realistic high-dimensional image distributions. Additionally, our wall-clock results obtained on an NVIDIA GeForce RTX 3090 GPU confirm that learn2mix introduces no scaling bottlenecks beyond standard SGD.\n\n> Whether the proposed training strategy is applicable to other network architectures besides CNN and Transformer.\n\nWe note that learn2mix is architecture agnostic because it alters only the dataloader side of training, adapting class proportions within batches to accelerate convergence. We demonstrate the efficacy of learn2mix on a variety of neural network architectures apart from standard CNNs and transformers, including residual networks, mobile networks, and autoencoders.\n\n> Whether the sequence length in Transformers impacts adaptive batching.\n\nWe clarify that learn2mix is agnostic to sequence length (and more broadly to dataset dimensionality), since it adaptively modulates training based solely on the proportion of samples from each class, guided by instantaneous class-wise error rates. The sequence length of transformers pertains to input dimensionality rather than class proportions, and as demonstrated consistently across empirical evaluations, covering datasets of varying dimensions (e.g., CIFAR-100, Imagenette, IMDB), learn2mix’s adaptive batching operates independently of sequence length.\n\nThank you again for your feedback, which has been important in improving our paper.\n\n**References:**\n\n[1] Zhang, Yongshun, et al. \"Bag of tricks for long-tailed visual recognition with deep convolutional neural networks.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 35. No. 4. 2021."}, {"review_id": "AFMvexoLKb", "reviewer": "Reviewer_CXfQ", "summary": "Learn2mitx introduces an adaptive batching rule that increases the proportion of hard classes inside every mini-batch.", "questions": "see weaknesses", "limitations": "n/a", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 2, "significance": 1, "originality": 2, "strengths_and_weaknesses": "Weakness:\n1. Assumptions like every classwise loss being strongly convex and lipschitz in network parameters are too strong and do not apply to modern neural networks.\n2. The paper’s experiment section does not scale to a reasonably large dataset like ImageNet.\n3. The difficulty of a loss is judged solely by the loss at the step, classes with high label noise (wrong labels) could receive extra sampling and can dominate training. This could be happening for a real-world dataset while the dataset tested in the paper does not cover this issue.\n4. The appendix lists architectures and a single γ per dataset, but there is no sweep or variance report. For any other people who may want to try learn2mix, it is hard to determine which mixing rate to set.\n5. More analysis and visualization on how learn2mix affect per-class accuracy could be presented to understand the proposed method better.\n6. The proposed method can only be applied to classification tasks.", "was_revised": true, "final_justification": "Overall I acknowledge that this paper achieve good results within its scope (as pointed out by other reviewers), and should definitely be accepted if it were a NeurIPS 2020 submission. However, I find significant gap between this paper and deep learning in 2025. I will keep my current rating.", "rebuttal": "Dear Reviewer CXfQ,\n\n\nThank you for your helpful feedback. Below, we have provided responses to your comments.\n\n> Assumptions like every classwise loss being strongly convex and lipschitz in network parameters are too strong and do not apply to modern neural networks.\n\nWe note that assumptions like strong convexity are standard in the machine learning literature when deriving convergence rates for gradient-based optimization methods [1], [2]. These assumptions provide a manageable framework to obtain rigorous and provable insights into the behavior of optimization algorithms. While the strong convexity assumption may not hold strictly in practical neural network training, our theoretical findings offer valuable intuition about the convergence properties of learn2mix. Moreover, our empirical results demonstrate that learn2mix effectively accelerates convergence in real-world non-convex settings, reinforcing the practical relevance of our work.\n\n> The paper’s experiment section does not scale to a reasonably large dataset like ImageNet.\n\nFor the present study, we intentionally restricted ourselves to CIFAR-100-LT (using a logarithmically decaying imbalance factor to exhibit long-tail behavior), Imagenette (using a stepwise imbalance factor), and similar benchmarks because they are canonical in the imbalanced learning literature [1], admit controllable imbalance factors, and can be reproduced without access to large compute resources. Imagenette itself is a carefully chosen subset of ImageNet and reflects realistic high-dimensional image distributions. Additionally, our wall-clock results obtained on an NVIDIA GeForce RTX 3090 GPU confirm that learn2mix introduces no scaling bottlenecks beyond standard SGD.\n\n> The difficulty of a loss is judged solely by the loss at the step, classes with high label noise (wrong labels) could receive extra sampling and can dominate training. This could be happening for a real-world dataset while the dataset tested in the paper does not cover this issue.\n\nWe appreciate this point raised by the reviewer. However, we emphasize that this issue of incorrectly labeled (high-noise) classes dominating training is not specific to learn2mix, and is shared by all error-driven sampling methods in the imbalanced learning space, such as class-balanced loss functions (e.g., focal loss) and importance sampling. As stated previously, our present study focuses on a variety of canonical and standard benchmarks (e.g., CIFAR-100-LT [1]), since these datasets are widely used in the imbalance learning literature for comparisons under controlled and reproducible conditions. Extending learn2mix for applications in high label-noise scenarios remains an important direction for future work.\n\n> The appendix lists architectures and a single $\\gamma$ per dataset, but there is no sweep or variance report. For any other people who may want to try learn2mix, it is hard to determine which mixing rate to set. More analysis and visualization on how learn2mix affect per-class accuracy could be presented to understand the proposed method better.\n\nWe thank the reviewer for these suggestions. Regarding $\\gamma$, as we recommend in Section 4 of the main text, choosing $\\gamma \\in [0.01, 0.5]$ yields the best performance. We provide additional empirical results below for the worst _per-class_ classification accuracy on Imagenette and IMDB for $\\gamma \\in \\\\\\{ 0.05, 0.1, 0.5 \\\\\\}$ to demonstrate relative insensitivity to the choice of $\\gamma$. We will include this result in the revised manuscript.\n\n**Table 1: Worst-class test performance comparison for ResNet-18 on Imagenette** \n\n| Method | Worst-Class Accuracy (57.5 s) |  Worst-Class Accuracy (115.0 s) |  Worst-Class Accuracy (230.0 s) |\n|--------|-----------------------------------:|-----------------------------------:|-----------------------------------:|\n| L2M ($\\gamma = 0.05$)    | 0.0000 %         | 2.3866 %      | 14.1623 %                  |\n| L2M ($\\gamma = 0.1$)      | **1.3149 %**    | **10.9318 %**    | 14.1451 %                 |\n| L2M ($\\gamma = 0.5$)    | 0.2542 %         | 3.2815 %      | **15.6304 %**                  |\n| CL     | 0.7779 %                         | 0.6205 %                         | 0.6325 %                         |\n| FCL    | 0.0000 %                         | 0.0000 %                         | 4.8162 %                         |\n| SMOTE  | 0.0000 %                         | 0.0000 %                         | 8.9119 %                         |\n| IS     | 0.0000 %                         | 0.0000 %                         | 0.7924 %                         |\n| CURR   | 0.0000 %                         | 0.2482 %                         | 2.8140 %                         |\n\n**Table 2: Worst-class test performance comparison for Transformer on IMDB** \n\n| Method | Worst-Class Accuracy (38.0 s) |  Worst-Class Accuracy (75.0 s) |  Worst-Class Accuracy (150.0 s) |\n|--------|-----------------------------------:|-----------------------------------:|-----------------------------------:|\n| L2M ($\\gamma = 0.05$)    | 46.2067 %         | **77.4133 %**      | 79.2200 %                  |\n| L2M ($\\gamma = 0.1$)      | 40.2867 %         | 76.6267 %    | 80.7400 %                 |\n| L2M ($\\gamma = 0.5$)    | **60.3733 %**         | 74.4467 %      | **81.8533 %**                  |\n| CL     | 0.0000 %                         | 47.6400 %                         | 67.3933 %                         |\n| FCL    | 0.0000 %                         | 34.5333 %                         | 60.7200 %                         |\n| SMOTE  | 11.3200 %                         | 28.7867 %                         | 62.5400 %                         |\n| IS     | 24.8467 %                         | 57.7200 %                         | 69.6267 %                         |\n| CURR   | 0.0000 %                         | 48.4400 %                         | 66.0333 %                         |\n\n> The proposed method can only be applied to classification tasks.\n\nWe note that the reviewer’s claim is not correct; we explicitly detail how learn2mix can be applied to regression and reconstruction tasks (not just classification tasks) in Section 4 of the main text through its ability to adaptively handle different data distributions. We further provide _full empirical results_ in Section 4 on standard benchmarks for regression and image reconstruction, demonstrating that learn2mix maintains accelerated convergence over classical training.\n\nThank you again for your valuable feedback, which has been important in helping improve our paper.\n\n**References**:\n\n[1] Guanghui Wang, Shiyin Lu, Quan Cheng, Wei wei Tu, and Lijun Zhang. SAdam: A variant of adam for strongly convex functions. In International Conference on Learning Representations, 2020.\n\n[2] Arindam Banerjee, Pedro Cisneros-Velarde, Libin Zhu, and Misha Belkin. Restricted Strong Convexity of Deep Learning Models with Smooth Activations. In International Conference on Learning Representations, 2023."}, {"review_id": "qTGNJV0OK8", "reviewer": "Reviewer_43t3", "summary": "This paper proposes a novel training strategy meant to optimize training convergence by adaptively adjusting class proportions within batches during training. Specifically, this work is motivated by the challenge posed by imbalanced training datasets. This method is called Learn2Mix, and differs from existing methods that address imbalance by adjusting sample weights or by resampling, since it dynamically adjusts the composition of batches during training based on per-class error rates. The results show that convergence is faster with learn2mix (if conditions such as that class proportions converge), overfitting is prevented, and that this method works across several tasks (classification, regression, and reconstruction). Datasets include MNIST, Fashion-MNIST, CIFAR-10, Imagenette, CIFAR-100, IMDB, Wine Quality, and California Housing. Results for all tasks across all datasets indicate best performance for Learn2Mix, relative to several benchmark and SoTA methods.", "questions": "-\tIn some of the results, such as those of Fig 3 (c), the Learn2Mix training error remains relatively high compared to the Classical training error, whereas test error is improved relative to classical. Is there some insight to be learned from this? Why does train error remain higher?\n-\tAre there any downsides to your method relative to SoTA? This was not discussed. Could you imagine any types of scenarios in which your method would perform poorly or where existing methods would be preferable?\n-\tGiven that the notation was rather elaborate here, could you please describe in words how the class proportions are updated during training? Where does gamma come from? In Eq. (3), what is the meaning of the ‘1’ function in the denominator?", "limitations": "-\tAre there any downsides to your method relative to SoTA? This was not discussed. Could you imagine any types of scenarios in which your method would perform poorly or where existing methods would be preferable?\n-\tAdditionally, consider any societal impacts, perhaps related to biases in training datasets and whether these could be exacerbated.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Quality:\n\n-\tThis paper seems to be technically sound and claims seem to be supported, both through analysis (theory in Section 2) and through experiments (Section 4), though I was not able to verify all proofs. \n-\tThis seems to be a complete work, wherein evaluations are comprehensive.\n-\tThe authors do not comment on the weaknesses or limitations of their work.\n\nClarity:\n\n-\tI found this paper to be unusually heavy with notation. This meant that reading and understanding was encumbered, even though explanations were perhaps more precise than in the average paper. I would suggest reducing the heavy reliance on elaborate notation, and instead move these parts to the appendices. \n-\tSection 3 was also difficult to follow, with heavy reliance on elaborate notation. I would suggest describing in writing more details of how the class proportions are updated, since this remained somewhat unclear.\n-\tOtherwise, the paper is clear and has enough detail to reproduce results.\n-\tThe introduction section is well-written and clear, especially the listing of contributions, though background is rather lengthy. \n-\tIn Definition 2.2, it would seem that the loss for Learn2Mix is really the same, essentially; however, it is just that the mixing parameters are fixed for Classical training and are adaptive for Learn2Mix training. The only real piece of new information in this definition is the description, Eq. (3), of how the mixing parameters are updated. I would suggest revising this to emphasize the update step.\n-\tIn the paragraph starting on line 206, it is rather confusing reading about regression tasks with categorical variables. Is this not classification? Do you instead mean that the dataset is comprised of several distinct variables?\n\nSignificance:\n\n-\tThis work appears to be significant in that the method is novel and that it advances the field relative to all compared benchmark methods. \n-\tThis is a meaningful step towards the overall optimization of the training of deep neural networks.\n\nOriginality:\n\n-\tThe method is original and novel.", "was_revised": true, "final_justification": "I maintain my rating of 5-accept. Authors have addressed all my concerns, especially those surrounding improving clarity. I have not updated my ratings in my review (based on original submission) though my concerns were addressed in rebuttals. See my comments for details.", "rebuttal": "Dear Reviewer 43t3,\n\nThank you for your comprehensive feedback. We will improve the terminology and the presentation of the paper as suggested. Below, we have provided responses to your comments.\n\n> I found this paper to be unusually heavy with notation. This meant that reading and understanding was encumbered, even though explanations were perhaps more precise than in the average paper. I would suggest reducing the heavy reliance on elaborate notation, and instead move these parts to the appendices.  Section 3 was also difficult to follow, with heavy reliance on elaborate notation. I would suggest describing in writing more details of how the class proportions are updated, since this remained somewhat unclear. \n\nThank you for the constructive feedback. We will simplify our notation and expand the prose description of the class-proportion update in Section 3 of the revised manuscript.\n\n> In Definition 2.2, it would seem that the loss for Learn2Mix is really the same, essentially; however, it is just that the mixing parameters are fixed for Classical training and are adaptive for Learn2Mix training. The only real piece of new information in this definition is the description, Eq. (3), of how the mixing parameters are updated. I would suggest revising this to emphasize the update step.\n\nThank you for this suggestion. We will revise Definition 2.2 to streamline the loss formulation and emphasize the mixing parameter update.\n\n> In the paragraph starting on line 206, it is rather confusing reading about regression tasks with categorical variables. Is this not classification? Do you instead mean that the dataset is comprised of several distinct variables?\n\nRecalling Section 4, we write that “for regression tasks with a categorical variable taking $k$ distinct values, the samples from $J$ that correspond to each of the $k$ values can be aggregated to obtain each class-specific training dataset, $J_i$.” Here, “categorical variable” simply means any feature that can be used to partition the data into $k$ groups. Many regression benchmarks include such categorical features (e.g., California Housing), which can be leveraged to group samples by similarity. Learn2mix treats each group exactly like a “class”, wherein it maintains one mixing parameter, $\\alpha_i$, per group and adapts batch proportions based on each group’s error. No true class labels are required: any meaningful partition of the dataset suffices. We will clarify this grouping interpretation in the revised manuscript.\n\n> In some of the results, such as those of Fig 3 (c), the Learn2Mix training error remains relatively high compared to the Classical training error, whereas test error is improved relative to classical. Is there some insight to be learned from this? Why does train error remain higher?\n\nThis pattern reflects _reduced overfitting_ under learn2mix: as we note in the manuscript, learn2mix “has a tighter alignment between training and test errors versus classical training. This correspondence indicates reduced overfitting, as learn2mix inherently adjusts class proportions based on class-specific error rates, $L_i$. By biasing the optimization procedure away from the original class distribution and towards $L_i$, learn2mix achieves improved generalization.” In other words, learn2mix reallocates learning effort toward higher-error groups instead of exhaustively minimizing loss on easier ones, resulting in a modestly higher overall training error and improved test performance (the aforementioned “tighter alignment”).\n\n> Are there any downsides to your method relative to SoTA? This was not discussed. Could you imagine any types of scenarios in which your method would perform poorly or where existing methods would be preferable? Additionally, consider any societal impacts, perhaps related to biases in training datasets and whether these could be exacerbated.\n\nAs mentioned above, the application of learn2mix to regression and reconstruction tasks requires samples to be partitioned into groups by similarity (e.g., via a categorical variable taking $k$ values). Likewise, in settings with extreme, systematic label corruption inside a group, learn2mix (like other loss-driven reweighting methods such as focal loss or importance sampling) could over-allocate updates to this group. We will include these limitations in the updated manuscript.\n\n> Given that the notation was rather elaborate here, could you please describe in words how the class proportions are updated during training? Where does gamma come from? In Eq. (3), what is the meaning of the ‘1’ function in the denominator?\n\nAt the end of each epoch, we compute the average loss for each of the $k$ classes, forming the vector $L(\\theta^{t-1})=[L_1,\\dots,L_k]$. We then _normalize_ this vector into a probability distribution by dividing by its sum: $L(\\theta^{t-1})/(\\mathbf1_k^{\\top}L(\\theta^{t-1}))$, where $\\mathbf{1}_k$ is a length-$k$ vector of ones (the “1” in the denominator), which ensures the entries sum to one. Next we _nudge_ the current mixing vector toward that distribution:\n\n- $\\alpha^{t} = (1-\\gamma) \\alpha^{t-1} + \\gamma \\frac{L(\\theta^{t-1})}{\\mathbf1_k^{\\top}L(\\theta^{t-1})}$,\n\nwhereby classes with higher losses receive a larger share of samples in the next epoch. The scalar _mixing rate_, $\\gamma \\in (0,1)$, is a user-defined step size hyperparameter that controls how aggressively $\\alpha$ moves. In our experiments, we select $\\gamma\\in[0.01,0.5]$ via a small validation sweep. This update gradually reallocates batch proportions toward underperforming classes.\n\nThank you for your positive feedback and support of our paper. We sincerely appreciate your insightful comments, which has helped improve our work."}]

Author rebuttal:

Meta-review: This paper proposes an adaptive training strategy that dynamically adjusts class proportions in batches based on real-time class-wise error rates to accelerate convergence, with applications across classification, regression, and reconstruction tasks.  Key innovations include its dynamic batch composition (prioritizing high-error classes), architecture-agnostic design (validated on CNNs, Transformers, autoencoders), and integration of computational overhead into wall-clock metrics, ensuring practical efficiency.  

Reviewers raised concerns about theoretical assumptions (strong convexity not holding for neural networks), reliance on dated baselines, scalability to large datasets/modern architectures, and hyperparameter sensitivity. Authors addressed these effectively: clarifying theoretical assumptions as standard for convergence insights while emphasizing empirical validity in non-convex settings; justifying baselines via recent citations (2024 NeurIPS/ICML) using similar benchmarks; confirming no scaling bottlenecks and architecture generality.  

Initial ratings from reviewers were mostly "borderline accept" with one "borderline reject," but post-rebuttal, consensus shifted positively. Reviewers acknowledged resolved concerns, with 43t3 maintaining "accept," others recognizing practical value despite residual critiques.
Decision: Accept (poster)