ST$_k$: A Scalable Module for Solving Top-k Problems
Authors: Hanchen Xia
Weidong Liu
Xiaojun Mao
Venue: NeurIPS 2024 poster (2024)
Keywords: Average Top-k Loss
Smoothing ReLU
Ranking Problem
Primary area: other

Abstract: The cost of ranking becomes significant in the new stage of deep learning. We propose ST$_k$, a fully differentiable module with a single trainable parameter, designed to solve the Top-k problem without requiring additional time or GPU memory. Due to its fully differentiable nature, ST$_k$ can be embedded end-to-end into neural networks and optimize the Top-k problems within a unified computational graph. We apply ST$_k$ to the Average Top-k Loss (AT$_k$), which inherently faces a Top-k problem. The proposed ST$_k$ Loss outperforms AT$_k$ Loss and achieves the best average performance on multiple benchmarks, with the lowest standard deviation. With the assistance of ST$_k$ Loss, we surpass the state-of-the-art (SOTA) on both CIFAR-100-LT and Places-LT leaderboards.

E A Scalable Module for Solving Top-k Problems*
Hanchen Xia Tia Weidong Liuf+,, Xiaojun Mao 7.8
{ School of Mathematical Sciences, Ministry of Education Key Lab of Artificial Intelligence.
Ministry of Education Key Laboratory of Scientific and Engineering Computing]
Shanghai Jiao Tong University, Shanghai, China * RoyalFlush AI Research Institute, Hangzhou, China (x_hc_200 weidongl, [ao tu. edu.
Abstract
The cost of ranking becomes significant in the new stage of deep learning. We pro- pose STk, a fully differentiable module with single trainable parameter, designed to solve the Top-k problem without requiring additional time or GPU memory. Due its fully differentiable nature, ST& can be embedded end-to-end into neural networks and optimize the Top-k problems within a unified computational graph. We apply STę the Average Top-k Loss (ATg), which inherently faces Top-k problem. The proposed ST₁ Loss outperforms AT& Loss and achieves the best average performance on multiple benchmarks, with the lowest standard deviation. With the assistance of ST; Loss, we surpass the state-of-the-art (SOTA) on both CIFAR-100-L3 and Places-I.T leaderboards.
Introduction
The ranking problem is quite common in the field of AL. For imbalanced datasets, the Average Top-k (ATg) Loss is more suitable than the conventional Average Loss [Lyu et al., 2020]. In the context of ambiguous classification tasks, Top-k Learning allows the ground truth fall within the largest probabilities, enhancing the model's generalizability [Lapin et al., 2017, Berrada et al., 2018, Petersen al., 2022]. For language models, the Top-k sampling method helps the models select the top most probable words during text generation, producing more fluent and coherent sentences. Distributed learning systems employ Top-k sparsification with error compensation (Top-k SGD) to reduce communication traffic without noticeably impacting model accuracy [Chen et al., 2018, Lin al., 2018, Shi al., 2019]. However, deep learning models continue grow in size, the cost of the ranking process becomes increasingly significant. For example, on single NVIDIA H800 GPU, tuning Llama-80 8B model using the LoRA [Hu et al., 2022] method with batch size of 4096 and 1000 iterations takes 990.14 14 seconds. In particular, performing QuickSort on the individual losses consumes 86.33 seconds.
In this work, we propose ST (Smoothed Top-k), a scalable module for Top-k problems. By adding only single trainable parameter, ST₃ is able to solve the Top-k problem in O(n k)) steps. Due to the fully differentiable nature of ST&, it can be embedded end-to-end as layer. Experiments show that we can even add À the computational graph for unified optimization using Stochastic Gradient Descent (SGD), which means that we do not need to consider the time cost of solving the Top-k problems within the computational graph. However, we can still enjoy the performance improvements that Top-k optimization provides. The contributions of this work are summarized as follows:
"Weidong Liu and Xiaojun Mao the co-corresponding authors. Xiaojun Mao's research is supported by NSFC Grant No. 12422111 and 12371273, the Shanghai Rising-Stn Star Program 23QA 1404600 and Young Elite Scientists Sponsorship Program by CAST (2023QNRC001).
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

We propose uniformly convergent approximation of the ReLU function.
We propose an efficient and robust Smoothed Top-k module, STT-
We apply ST& Module to smooth AT Loss, resulting in performance improvement and refreshing the state-of-the-art methods on two long-tailed learning leaderboards.
We design imbalanced classification dataset with theoretical decision boundary.
For experiments, by applying the ST& Module, we smooth the AT& Loss into ST& Loss. The compu- tation time of S Loss is almost identical to that of the average aggregating method, significantly faster than the sorting- required AT& Loss, and it exhibits the best performance. Experiments on synthetic datasets demonstrate that models trained with ST& Loss most closely approximate the theoretical decision boundary. On benchmarks of imbalanced binary classification, models trained by ST& Loss exhibit the lowest average misclassification rate and the lowest standard deviation. On regression datasets, models trained with ST& Loss exhibit the lowest (RMSE). Experiments on large weal-world datasets demonstrate that ST& Loss, as an aggregating trick for individual losses, is a scalable technique that improves accuracy on long-tailed benchmarks. With the help of ST& Loss, we surpass the state-of-the-ard (SOTA) on both the CIFAR-100-LT and Places-LT leaderboards.
1.1 Related Work
In current research, although there are many studies on individual loss, the general characteristics of aggregate loss are often overlooked. In the existing machine learning literature, a related line of work is the data subset selection problem [Wei et al., 2015], which aims to select a subset from large training dataset for model training while minimizing average loss. Curriculum learning [Bengio et al., 2009] and self- self-paced [Kumar et al., 2010] learning are recent learning schemes. They organized the training process into several iterative stages, gradually including training data from easy difficult to learn, where the difficulty level is measured by individual loss. Therefore, each training session in these methods corresponds to the average aggregate loss in the selected subset. The difficulties encountered by the Average Loss when dealing with imbalanced data, as discussed in Shalev-Shwartz and Wexler, 2016, Huang et al., 2020], prompted the exploration of more robust aggregate losses. Among these, Lyu et al. [2020] introduced the AT& loss, which averages the k largest individual losses, and exhibits advanced performance on imbalanced datasets.
Many classification tasks in the real world have inherent label confusion, as mentioned in Berrada et al. [2018]. This confusion may arise from various factors, such as incorrect labels, incomplete annotations, some fundamental ambiguities that even confuse the true labels for human experts. Therefore, some works proposed the concept of Top-k Learning in the field of image classification to address the issues of multiple semantics and semantic confusion in images [Lapin et al., 2017].
Berrada et al. [2018] proposed method to partially smooth the Top-k Learning loss function, but did not completely solve the sorting problem in the loss function and introduced a sorting computation of Ck. Petersen et al. [2022] proposed a Split Selection Network (SSN) based on sorting networks, which made the Top-k process differentiable and achieved the state of the art on ImageNet-IK I at that time. However, the computation required by this method is cumbersome and multilayered. Sorting networks are similar to algorithms like QuickSort with time complexities of O(n log...
1.2 Notational Conventions
Let N denotes the set {l,..., n. and [[ denotes the indicator function (which is I when the proposition is true and 0 otherwise). Thus, the sign function can be defined as: sign(z) [(200 [==co)- The Hinge function can be defined as: [x] = max (0, We use ||>|]] || and || to represent the é, f2, and foo norms of x, respectively. For the set L= L1, L1, ..., En (1a denotes the k-th largest element, so we have fp] 2 f12] 2 ... 2 laai In supervised learning problems, our training set typically contains an input set and a target set, the input set coming from the input domain X, and the target set from the target domain ), and we use their joint domain Z = X' x ) to represent the range of the dataset. The training set S = 21. .... 2n is subset of Z, where 2; = (r,, yt). Our task is to find predictor ʃ X - ) from the function family H. that can predict the corresponding target y based on the new input x. To evaluate the effect of the predictor, we need to introduce an individual loss function e y x y -> IR+,, where {(f(x). v)
usually reflects some distance between the prediction 9 f(x) and the true value y. The training

---

process of the predictor can be described as using the gradient descent algorithm to optimize an objective function (usually minimizing the loss function). The objective function can generally be written as ((J,S) )) + ()()), where L(..) is the aggregated individual loss function, and ()(()) is regularization term (& or lg regularization term). The loss function L(f.S) is usually Average Loss, that is, Lavg (f)
Zi=1 ((f(x)) v:). However, recent works, have shown some drawbacks of the average loss adapting to imbalanced data distributions [Shalev-Shward and Wexler, 2016], and explored choices other than the average loss for the aggregate loss formed from individual losses, e.g., the maximum (aggregate) loss, Lman (S, S) = maxieN, ([((x))))))
2 & Architecture
Smoothed Top-k Module
o
9
Figure I: ST& Architecture, For any layer of neurons in a neural network, to solve the Top-k problem for its weights, insert S Module. The trainable parameter will gradually approximate the k-th largest element during the optimization process. And this À can be used to filter neurons.
Suppose we have a set of elements {e then the Top-k problem can be describe as:
1. find the k-th largest element [N)
2. find the sum of top-k largest elements Σ Li=1 Ejp]
This process can certainly be achieved through conventional sorting and summation. However, in the worst-case scenario, the cost of solving the Top-k problem can reach O(n²). Ogryczak and Tamir [2003] proposed an equivalent optimization form to solve the Top-k problem and proved its linear convergence.
EpA
min
Σ
Je 11 + +KA }
(I)
However, this surrogate objective function suffers from a non-differentiable point at e; X, which makes it challenging optimize. The key solving this problem lies in designing function approximating H++, which can be regarded as a rectified linear unit (ReLU) function. Here, we introduce the Smoothed ReLU (SReLU).
1
(
22
SReLU(x)
2 x +
(2)
where 8 is hyperparameter (usually we set 8 = 0.01). It can be observed from Figure 2 that as decreases, SReLU increasingly approximates the ReLU function. In fact, SReLU converges uniformly to ReLU as 5 + 0=; detailed proof of this uniform convergence is provided in Proposition 1 of Appendix A.1. With the help of SReLU, the objective function (1) can be smoothed as:
(ViCT-
ene RJ min
(e. A) +
+ +
tKAJ.
KA
82
1=1
the optimal ejk] which is the k-thh largest element. the following section, we will introduce an application scenario for ST:.

---

ReLU SReLU, 6-0.1
SReLU, 6=0.05
1.00
0.75
0.50
0.25
0.00
0.25
0.50
0.75
1.00
Figure 2: ReLU and SReLU with various smoothing coefficients 6.
3 From A Loss to ST& Loss
In the training process of neural network, first choose form of individual loss (e.g., logistic loss, hinge loss, mean square loss, or cross-entropy loss). Then, we aggregate all individual losses calculate their average, which is the most common aggregate Ioss function: Average Loss. The Average Loss is widely used in a myriad of deep learning tasks. This widespread application stems from the robust theoretical foundations [Bartlett et al., 2006, De Vito al., 2005]. However, Average Loss tends to overfit the training data, especially on imbalanced datasets [Shalev-Shwarr and Wexler, 2016, Huang al., 2020]. This has inspired the motivation find other forms of aggregate loss, such as the maximum value among individual losses (referred to as Maximum Loss). The Average Top-k (ATx)) Loss was introduced by Lyu et [2020]]
1
Cot
ppp
which represents the average of the largest l individual losses. According to the derivation in [Ogryczak and Tamir, 2003]. this ranking loss can be written in the following equivalent form:
1
+ k - A.
Lmat-k
[4 {]_
(5)
72 1=1
With the help of the ST; Module, Lmar- ((f, can be reconstructed as
(
(11 X)2
11
)1 A>
Lat-k
(Ls - A) +
(6)
2n
82
where E(Sarmas (xi), yi) the individual loss of sample i, and Wmodel represents the set of parameters of the predictor f. It is easy to verify that lims-snt Let-k Lmat-ks given Proposition

---

1 in Appendix A.I.
[e 11+
2 2
(e;
2) +
+1
71
n
(li - x)2
Vli 2)2 + 82 +
2n
1=1
1
8²
2n
V(e 2)3 + v (l, - 2)2
1m1
1
1
1
2n
(6 (1=-))2
(Li-2)2
1=1
82
8 2
The approximation error between the smoothed loss function and the original loss function can be uniformly bounded by 6/2, VY.
When l is convex, Equation (6) exhibits joint convexity with respect to the parameters (Wmodel: X). making the problem a special case of the non-linear multiple choice knapsack problem [Zemel, 1984]. which has at most g roots. These roots can be found in constant time, allowing the problem be solved in O(n In ((n) time when is fixed [Megiddo, 1984]. Therefore, it can be iteratively updated using relatively simple algorithms. For example, in the case batch learning, the block coordinate descent (BCD) method [Nocedal and Wright, 1999] can be employed, where W'medel and X are updated alternately after initialization.
BCD-STØ:
(1+1) + argmin Cst-ki
w (++1) f- argmin La
m
The convergence of the above coordinate descent algorithms can be found in Luo and Tseng [1992]. Saha and Tewari [2013], Tseng [2001].
Furthermore, empirical evidence suggests that we do not need to spend extra time optimizing separately, incorporating À into the computational graph of Windel for unified optimization using Stochastic Gradient Descent (SGD) [Bottou and Bousquet, 2008, Shamir, 2011, Srebro and Tewari, 2010], performance improvements can still be achieved.
SGD-ST
((+1) (11 17 CACoLLRi wp (++1) +- (t) V Whttt-
where 7) is the size of the update step, and when 1/2 Vp the stochastic gradient descent method can ensure convergence a local minimum of Equation (6) [Shamir, 2011, Srebro and Tewari, 2010]. By eliminating points where the gradients are discontinuous, the training process becomes more stable, and converges faster, experimentally demonstrated by the standard deviations reported in Tables 3, 4, and and the time cost reported in Table 2.
Synthetic Experiments
4.1 Time Cost
We first perform experiments to compare the time costs of two standard sorting algorithms, ATx, k. and STr, in calculating the ranking average. The experimental setup involves finding the Top-k (k=5) sum from 10,000 standard normally distributed samples. For both AT; and ST&, we iterate until the 50 experiments and record the average time taken.
emror is less than 1 For each algorithm,
5

---

Algorithm
Complexity
Average Time(s)
BubbleSort
O
20.42196 + 3.7015
HcapSort
On log(n))
0.1243 0.0446
AT&
O(n + k)
0.2167 0.1528
STx (Ours)
O + k)
0.0127 0.0020
Table 1: Performance comparison of different sorting algorithms and our method.
As shown in Table I, ST& indeed demonstrates linear time complexity while also exhibiting stable optimization process.
4.2 Gaussian Distributed Dataset
To illustrate the capability of ST& Loss in approximating the ideal decision boundary, we design daussian-distributte dataset. The dataset generated by a pre-set covariance matrix Σ E 12 _ddd and mean vectors 11 12 Rd corresponding two categories. We set 200, Σn 0.25 0.550.b----00
[41 O₂ and μ42 (1, 1,1,........ ..... 0, .... 0)T which has 10 ones. For the predictor, we Logistic Regression (LR) model fx) sigmoid(w> æ + 6). The detailed derivation process of the decision boundary can be found the Appendix A.2. Here, we present the conclusion directly. For two normal populations with given covariance matrix E, and means μn and 142, the LR model has theoretical decision boundary:
1 Mo
Pr(Y =1)
(8) Figure 3 provide example 2D plane, we [(,) = 0.815 141 = [0, OJ, u12 [2, 2] thus according to our derivation above, = 1.111], 00.36, where the black dashed line is the theoretical boundary.
w- = (H (o)
1o)
Σ
D"
Σ Mo + 11, E-'
+ ln
Pr(Y 0)
We ParaF, [Tian and Gu, 2017] measure the overlap of estimated supports and true supports:
precision recall
ParaF 2.
precision + rocall
Isupport (sspport(ww))) [[]]
where precision
Positive
Negelive
recall (espppoo" where @ is (support(w")
Boundary
the estimated value of the parameters obtained by the predictor, and w" the theoretical value of the parameters mentioned above. The opera- tor |- is used to find the cardinality of the set, which is the number of elements, and support()) the support set of the vector, which refers to the set of indices of its non-zero elements.
Other basic settings are as follows. The loss function we use is the binary cross-entropy loss. To obtain sparse solution, we add an é regular- ization term. The number of samples is 10,000 for the training set and 2,500 each for the vali- dation set and the test set. We implement early stopping of the iteration based on the accuracy curve in the validation set; that is, we break when the increase in validation accuracy within 200 steps less than 10-4 For each meta-experiment, we repeat it 50 times and take the average. Now we conduct two groups of experiments as follows.
Figure 3: A Synthetic Example 2D-Plain.
Aggregate Losses and ReLU Varients. In these experiments, we set the ratio of positive to negative samples at 8:2 for the training set, while the validation set and the test set remain 1:1. We also compare SReL.U with other variants of ReI.U. The smoothing coefficient for SReL.U set to 0.01, and the settings for the other ReLU variants remain at their defaults. We use Adam our optimizer,

---

setting the batch size to 512, while keeping the other hyperparameters as default. According to Equation (8), we could compute the theoretical value of w", which a vector with only the first 10 elements non-zero. Sensitive Analysis. In these experiments, we only adjust the positive-negative ratio and retain the rest of the settings from the previous experiment.
STe
Aggregate Loss
Average
Maximum
AT&
MAT
ELU
SoftPlus
Leaky-ReLLU
SReLU (Ours)
Accuracy (%)
72.864
65.448
73.013
73.180
72.988
70.968
73.864
76.104
ParaFi-Soed -Score
0.1904
0.0909
0.1986
0.2063
0.1925
0.1592
0.2441
0.3446
Time (s)
16.525
6.145
19.271
15.774
15.572
16.233
15.525
15.436
Table Accuracy and ParaF, 1-Sore on the synthetic dataset.
87.5
ST, Average
0.45
ST,
0.40
Average
Accuracy
Paraf Score
0.35
0.30
77.5
0.25
0.20
75.0
0.15
0.05
0.005
0:001
0.10
0.05
0.005
10
10
10
10
10
10
Negative Ratio
Negative Ratio
Figure 4: Accuracy and ParaFi-Sear vs Negative Sample Ratio.
Table 2 presents the Accuracy and ParaF score achieved when different aggregate loss functions are combined with the LR model and the cross-entropy loss, trained to convergence. With the help of SReLU, ST& Loss achieves performance that surpasses that of all other aggregate losses. Due its property of being real uniformly convergent approximation (not just similar in shape). SReLU outperforms other ReLU variants in this scenario. Figure shows the relationship between accuracy. ParaF,-scome and negative sample ratio (negative ratio). ST₃ is less sensitive to the imbalance ratio,
5
Real World Applications
5.1
Binary Classification
We select binary classification datasets from the KEEL2 and UCI databases; see Table 9.
Dataset
Average
Maximum
ATa
MAT
STu(Ours)
appendicitis
13.778++6.011
32.815 11.306
13.630+16.400
14.667+7.114
13.406-455491
australian
13.549+1.022.
46.89+18.799
14.428+1.005
14.324+ 1.058
13.358:10.982
german
2..029
45.832.15.465
26.30312..34
26.392-12.431
25.561+1.988
phoneme
20.82311..80
45.831
21.420
17.828.1.709
16.427++.494
spambase
6.972-11.519
45.777
6.955.11.535
6.687
6.610++1.333
titanic 22.613++1.22
48.065+11.338
22,468+1144 1.41
22.211+
21.801+00911
wisconsin
3.275:00.14
34.468+19.306
3.205+00.73
3.046+10.570
2.936:10.665
Table Misclassification Rate(%) and Standard Derivation of Various Aggregate Losses Combined with Individual Logistic Loss.
Next, we tested the performance different forms of aggregate loss on binary classification bench- marks. The individual loss function can generally be chosen as Logistic or Hinge, which are defined

---

Dataset
Average
Maximum
AT
MAT,
(Tur))
appendicitis
15.852+8.082
28.000+
16.148+77436
15,481+5.577
14.667-15.736
wisconsin
3.240+1.004
12.000+10.700
3.158+1.005
2.936+1.2112
2.889:11.099
australian
13.225-4:2.351
36.34749.576
13.896+22.671
14.243+22.48
13.746-
german
26.912.1...944
43.864-11..322
26.776.13.010
26.232.12.190
25.928:+2.973
titanic 22.152.11..62
11.691
22.316::1.490
22.116:11.567
11.96611.28
phoneme
2..543.11.358
44.6561116.055
20.733-1 83.986::8.955
17.335
17.152:+0.955
spambase
7.443
7.197=00665
6.645+t0.677
6.342::0.589
Table 4: Misclassification Rate (%) and Standard Derivation of Various Aggregate Losses Combined with Individual Hinge Loss.
as follows:
Logistic : +(f((z))) y) = log + exp((uf(x))) (x))++
Hinge E(f(z). y)
The prediction model is a two-layer MLP with 10 nodes in the hidden layer, activated by the ReLU function between the two fully connected layers. To increase the stability of the training process, we added an E, regularization term to the loss function ((w) = 22 w We divided the datasets into training, validation, and test sets in a 0.5 0.25 0.25 ratio. Nowadays, SGD generally replaced by the mini-batch method instead of stochastic gradient descent with single sample point, which is faster and more robust. To accommodate datasets of varying sizes in the article, we set the batch size to 16.
The hyperparameters in the experiment include k in MAT, and AT&, the coefficient of the regular- ization term C, the initial learning rate y), and the smoothing coefficient ó. These hyperparameters will be selected based on their convergence accuracy performance on the validation set (for each combination of typerparameters, we repeat the experiment fifty times and take the average of their prediction accuracy on the validation set as the basis for parameter selection). The search spaces for several hyperparameters are as follows: & E {1] U 00.1 0.1 1]: C (10", 10 10',,00 10?, 103, 100t, 10%); 7) E (0.1, 0, 01, 0.005, 0.001): 5 E f0.1, 0.001, 0.0001).
Since traditional gradient descent is too sensitive to the choice of step size, which is not conducive our pure comparison of the convergence speed and accuracy of the loss function, we use the AdaGrad algorithm to iteratively update the learning rate. During the learning process, to avoid overfitting the model, record the accuracy of the MI.P predictor the validation set after each iteration, and perform an early stop when the accuracy does not increase (the increase in the accuracy of the predictor on the validation set is less than 10 after 50 steps) and roll back the model to the checkpoint with the highest accuracy on the validation set during the entire training process; otherwise, we continue training until convergence. Tables 3 and 4, respectively, show the average probability of misclassification (%) of the models trained to convergence in 50 experiments in the test set under individual Logistic and Hinge loss (the standard deviation of the 50 experiments is listed in parentheses).
In the results shown in Tables and 4. Almost all the lowest misclassification rates appear in the STx trained models. Furthermore, models trained with ST₃ Loss exhibit the lowest standard deviation, which to some extent demonstrates the robustness of the training process.
5.2 Long-Tailed Classification
The interest long-tailed classification tasks has increased with the advent of large vasion-language models such as contrastive language- pre-training (CLIP) Radford et al. [2021]. Long-tailed versions of recognized datasets were built by the community. Using the Pareto distribution (a = 6), the ImageNet- KK and Places datasets can be sampled to create ImageNet-LT and Places-LT datasets, as described in [Liu et al., 2019]. Consider two types of imbalance [Cui et al., 2019, Buda et al., 2018), Cao et al. [2019] built CIFAR-10-1E (p 10) and CIFAR-100-LT (p 100), where p max; (m:)/ ming(n, (n.) represents the imbalance ratio, defined as the ratio between the sample sizes of the most frequent and least frequent class. Due to varying word frequencies, machine translation,
8

---

more generally, text generating, can inherently be considered as long-tailed classification tasks. We choose the WMT2017 and IWSLT2014 datasets for the experiments.
500
639
Number of Samples
674
112
127
400
408
819
200
6-
Head Classes
Tall Classes
Class
Figure ImageNet-LT.
Settings For visual classification tasks, we Parameter-Efffiient Long-Tailed (PEL) Recognition [Shi et al., 2023], which primarily leverages information from a text encoder to adjust classification probabilities. The text encoder in PEL is derived from the pre-trained CLIP model, and the backbone is Vision Transformer (ViT-Base) [Dosovitskiy et al., 2021] pre-trained in ImageNet-21K, We freeze the backbone and train the branch model of Parameter Efficient Fine-Tuning (PEFT) [Jia et al., 2022, Hu et al., 2022, Houlsby et al., 2019]. We simply replaced the Average aggregation of individual losses in the Logit Adjust (LA) [Menon al., 2021] loss with AT, or ST&. For k we set it at 0.9 batch size, and = 0.01. Then we report the accuracy of the balanced test sets. Visual classification experiments can be implemented on single L20 GPU, with varying durations ranging from 20 to 200 minutes.
For translation tasks, we use the training workflows provided by OpenNMT [Klein al., 2018] and FAIR-Seq" [Ott et al., 2019], which are very easy to reproduce. Models are essentially different sizes of Transformers (see the footnotes). In the training process of language models, the actual batch size, calculated as the sequence length multiplied by the number of sequences, is often dynamic. Consequently, we set k/batch size 0.96. We simply replaced the Average aggregation of individual losses in the Cross Entropy (CE) loss with AT or ST We calculate the bilingual evaluation understudy (BLEU) scores for the models. The BLEU for SWSLT2014-traad traiinedd models are derived from their test set, while for the WMT2017-traine, models, we evaluate their BLEU on the Newstesr2016 test set. Training these Transformer-ban based models would take about 10 hours on single RTX 4090.
Overall Results ST& Loss, aggregating trick for individual losses, is scalable technique that improves accuracy on long-tailed benchmarks. The models trained by ST& Loss surpass SOTA on the CIFAR-100-LT and Places-LT leaderboards.
bbttss//www.ssttnn..rrg//
"bupss/woorkkhop201iiiwtt...
ltg

---

CI0AR-I00-LT
ImageNet-LT Place-LT
IWSLT2014
WMT2017
Average
89.1*
78.3
52.2*
35.27
35,21
AT&
89.3
78.
52.3
35.45
35.05
STA
89.8
79.1
53.7
35.80
35.42
Table 5: Results on large long-tailed datasets are presented, where the first three columns indicate accuracy, and the last two columns show the BLEU scores. Values marked with an "*" represent fftt-off-hhe-ar (SOTA) on the leaderboard.
5.3 Regression Tasks
For descriptions of the four regression datasets, see Table 9. regression tasks, the individual loss is generally the distance (&, or distance) between the predicted value and the ground truth. Next, present the specific settings of the experiment. For the Housing, Abalone, and Cpusmall datasets, we normalize the output to between [0, 1]; for the Sine dataset, we first randomly sample 1000 points (1i, y:) from the function, where x4 E [-10,, 10). then we use the Radial Basis Function (RBF) kernel map I₁ into the kernel space. We select 10 RBF kernels, resulting 10-dimensional input = [k(z, ei).... k(z, c1o)], where k(2, cÿ)) = exp(- 2)) Furthermore, we add Gaussian noise (00, 0.23) to the target output
Square Individual Loss
Absolute Individual Loss
Datasels
Average
MAT;
ST₃ (Ours)
Average
MAT,
ST₃ (Ours)
Sinc
0.1603=000065
0.1591+0.0008
0.1446::0.0000
0.1686+00.006
0.1560=0.0066
0.1457=0.0043
Housing
0.0896=0.0083
0.0816=+0.0100
0.0810110.096
0.0781+0.0099
0.0670=000083
0.0626:10.0889
Abalone
0.0739100000
0.0739110.0018
0.0523:10.0014
0.074010.000
0,073310.0018
0.0517:0.0021
Cpusmall
0.0613:10.012
0.0313.10.0005
0.0296::.0005
0.0613:10.0012
0.03131100005
0.0244110.0006
Table 6: Comparison of Average RMSE and Standard Deviation for Different Aggregate Losses Combined with Square and Absolute Loss.
For individual losses, we choose the absolute value loss (f1) and the squared loss (42). The prediction model the same above, a two-layer MLP with 10 nodes in the hidden layer, activated by the ReLU function between the two fully connected layers. As in binary classification experiments, we randomly divide the dataset into training, validation, and sets, with hyperparameters chosen based on the performance in the validation set, repeated 50 times, and statistics on the test set are reported.
Tables show the average RMSE and standard deviation of 50 experiments in regression datasets with different aggregate losses combined with absolute individual loss and squared individual loss longer compare with Maximum Loss here),
(given the experience of the previous experiment, and S shows further performance improvement after smoothing.
Conclusion and Limitation
The proposed ST& Module can effectively solve the Top-k problem within neural networks, with no additional GPU memory ranking time. Due to its fully differentiable nature, the training process are stable. By applying ST, Module to the Average Top-k Loss, we achieve significant improvements across numerous benchmarks.
The limitation of this study is that we have not yet evaluated the ST; Module for additional application scenarios. The effectiveness of the ST₃ Module is only demonstrated in smoothing AT loss. We hope that future research will further explore the potential utility of the ST₃ Module.
References
Siwei Lyu, Yanbo Fan, Yiming Ying, and Bao-Gang Hu. Average top-k aggregate loss for supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(1):76 2020.
10

---

Maksim Lapin. Matthias Hein, and Bernt Schiele. Analysis and optimization of loss functions for multiclass, top-k, and multilabel classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(7):1533- 2017.
Leonard Berrada, Andrew Zisserman, and M. Pawan Kumar. Smooth loss functions for deep top-k classification. In International Conference on Learning Representations, 2018.
Felix Petersen, Hilde Kuehne, Christian Borgelt, and Oliver Deussen. Differentiable top-k classifi- cation learning. In International Conference on Machine Learning, pages 2022.
PMLR,
Chia- Yu Chen, Jungwook Choi, Daniel Brand, Ankur Agrawal, Wei Zhang, and Kailash Gopalakrish- nan. Adacomp: Adaptive residual gradient compression for data pataaaaaallel distributed training. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018.
Yujun Lin, Song Han, Huizi Mao, Yu Wang, and Bill Dally. Deep gradient compression: Reducing the communication bandwidth for distributed training. In International Conference on Learning Representations, 2018.
Shaohuai Shi, Xiaowen Chu, Ka Chun Cheung, and Simon See. Understanding top-k sparsification in distributed deep learning. arXiv preprint /9X 2019.
Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
Kai Wei, Rishabh Iyer, and Jeff Bilmes. Submodularity in data subset selection and active learning. In International Conference on Machine Learning, pages 1954-1963. PMLR, 2015.
Yoshua Bengio. Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual International Conference on Machine Learning, pages 41-48, 2009.
M Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. Advances in Neural Information Processing Systems, 23, 2010.
Shai Shalev-Shwartz and Yonatan Wexler. Minimizing the maximal loss: How and why. In International Conference on Machine Learning, pages 793-801, PMLR, 2016.
Lang Huang, Chao Zhang, and Hongyang Zhang. Self-adaptive training: beyond empirical risk 19376, 2020.
minimization. Advances in Neural Information Processing Systems,
Wlodzimierz Ogryczak and Arie Tamir. Minimizing the sum of the k largest functions in linear time. Information Processing Letters, 1573):117-122, 2003.
Peter L Bartlett, Michael Jordan, and Jon D McAuliffe, Convexity, classification, and risk bounds. Journal of the American Statistical Association, 101 1566, 2006.
Ernesto De Vito, Andrea Caponnetto. and Lorenzo Rosasco. Model selection for regularized least- squares algorithm in learning theory. Foundations of Computational Mathematics, 5:59-85, 2005.
Eitan Zemel. An (n) algorithm for the linear multiple choice knapsack problem and related problems. Information Processing Letters, 18(3):123-128, 1984.
Nimrod Megiddo. Linear programming in linear time when the dimension is fixed. Journal of the ACM (JACM), 31(1):114-12)) 1984.
Jorge Nocedal and Stephen J Wright. Numerical optimization. Springer, 1999.
Zhi-Quan Luo and Paul Tseng. On the convergence of the coordinate descent method for convex differentiable minimization. Journal of Optimization Theory and Applications, 72(1):7-35, 1992.
Ankan Saha and Ambuj Tewari. On the nonasymptotic convergence of cyclic coordinate descent methods. SIAM Journal on Optimization, 23(1):576-601, 2013.

---

Paul Tseng. Convergence of block coordinate descent method for nondifferentiable minimization. Journal of Optimization Theory and Applications. 109(3):475, 2001.
Bottou and O Bousquet. The tradeoffis of large scale learning, advances in neural information processing systems, 20, 2008.
Ohad Shamir. Making gradient descent optimal for strongly convex stochastic optimization. CoRR, abs/1109.5647, 2011.
Nathan Srebro and Ambuj Tewari. Stochastic optimization for machine learning. ICML Tutorial, 2010.
Lu Tian and Quanquan Gu. Communication--ffiiccne distributed sparse linear discriminant analysis. In Artificial Intelligence and Statistics, pages 118771187. PMLR, 2017.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 9748-8763. PMLR, 2021.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large- scale long-tailed recognition in open world. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2537-2546, 2019.
Yin Cui, Menglin Jia, Tsung-Y Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 9268-9277, 2019.
Mateusz Buda, Atsuto Maki, and Maciej A Mazurowski. A systematic study of the class imbalance problem in convolutional neural networks. Neural networks, (06:249-255) 2018
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with lobell-distrbuuion-wwwee margin loss. Advances in Neural Information Processing Systems, 32, 2019.
Jiang-Xin Shi, Tong Wei, Zhi Zhou, Xin- Han, Jie-Jing Shao, and Yu-Feng Li. Parameter-efficient long-tailed recognition. arXiv preprint arXiv:2309.0001, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Representations, 2021.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pages 709-727, Springer, 2022.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nip. In International Conference on Machine Learning, pages 2790-2799. PMLR, 2019.
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Loig-tail learning via logit adjustment. In International Conference on Learning Representations, 2021.
Guillaume Klein, Yoon Kim, Yuntian Deng, Vincent Nguyen, Jean Senellart, and Alexander M. Rush. Opennmt: Neural machine translation toolkit, 2018
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations, 2019.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
12

---

A Appendix: Proposition Proof
A.1 Uniform Convergence
[Proposition 1] We have that SReLU uniformly converges to ReLU as 8 + 0+, that is
limæ-so sup, (SReLUs(() - [-]+|= -
Proof.
ReLU(x)) [x]+ 2 (r + (x)):
SReLU(x) = 2 x + ( 82 +1 +
x2
(V +
Note that D(x) = 2(ReLU(x) SReLU(x)) = |K|
we have
D(x) = Vx²
Vx² 82
82
-
Vx2
2 82
For any x and 6, the form
therefore, for any >> 8 c, then Vo
2 17
+1
shall have D(x) <e. e.
A.2 Decision Boundary
[Proposition 2] For two normal populations with known (denote Σ) and equal covariance matrices, and means l1 and f22, the LR model f(x) sigmoid(w x + b) has a theoretical decision boundary:
w" (μ1 (4) T Σー
Pr(Y 1)
6" uD - uo + 1 I
m1
+ In
Pr(Y 0)
Proof. Usually, we estimate the parameters w and by maximizing the likelihood function
Σ In(Pr(y/)::))
This is to estimate parameters the "most data-fitting manner" when the distribution is unknown. Now know that two classes normally distributed, even know their mean and covariance matrix.
In Logistic Regression
Pr(Y 1|x)
1 e (w 2+b)*
w a0
Pr(Y = 0/z) + e e- x+6)
So we have
Pr(Y 1 [x)
T x + b.
(9)
Pr(Y = 0)z)
13

---

And by Bayes' theorem, we can derive
Pr(Y = 1x) Pr(Y 0
Pr(æ|Y = Pr(Y 1)
In
im
Pr(æ| = 0) Pr(Y = 0)
Here, since inherently know the data is normally distributed, have
Pr(Y =1/x) Pr(Y = 0|æ²]
Pr(Y 1
In
- mo) E
Po E pro + Mi E
p1
+ In
Pr(Y 0) (10)
Then, comparing Equation (10) with Equation (9), we'll have the answer.
A.3 More Experiments
We conduct ablation study "how STę complements other long-tailes learning algorithms. " Thee backbone here is a Vision Transformer (ViT) pre-trained by MAE / CLIP [He et al., 2022, Radford al., 2021]. "CS" represents cost-sensitive learning, and "PEL" refers to the method provided by [Shi et al., 2023]. The batch size was set to 2048 and the models were trained in 30,000 steps.
Pretrained By CS PEL STr
ImageNet-LT
CIFAR-100-LT
MAE
65.701
78.572
MAE
69.884
82,135
MAE
/
70.140
83.048
CLIP
78.296
89.103
CLIP
79.148
89.833
Table 7: Ablation Study on Long-Tailed Learning Algorithms.
The smoothing coefficient 6, 0.01, is grid search determined value for all datasets in Section 5.1 (see page 7) and was adopted in all other experiments. We conducted an ablation study on the five real-world datasets with all other settings remaining unchanged, except for 6.
CIFAR- 100-LT
ImageNet-LT Place- LT
IWSLT2014
WMT20177
0.1
88.78
76.11
51.06
33.12
32.07
0.05
89.44
78.96
53.80
34.82
35.28
0.01
89.83
79.09
53.69
35.80
35.42
0.005
89.80
79.16
53.75
35.67
35,25
Table 8: Sensitive Analysis of the Smoothing Coefficient d.
A.4
Benchmarks Detailed Information
Regression Datasets
Binary Classification Datasets
Sine Housing Abalone Cpusmall
Appendicitis Australian German Phoneme Spambuse Titanic Wisconsin
1000
506
4177
8192
106
690
1000
5404
4601
2201
683
10
12
7
24
9
2
2
2
2
2
2
Table 9: Statistics of Benchmarks.
In Table 9, we provide the statistical information for two benchmark sets discussed in Section 5, where c, n, are the number of classes, samples and features, respectively. The left part provides the information of four regression datasets in Section 5.3. The "Sinc" dataset is a synthetic dataset, sampled from the sine function y = sin(x))/x: The right part provides the information of seven binary classification datasets in Section 5.1.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and/or introduction clearly state the claims made.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations can be found Section 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
The proof can be found in Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
Every experiment can be easily reproduced, and the code will be available
soon.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification:
Main results can be reproduced by supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /niiss wwtppcccpppppeeeecccceeccccceeee /
seiisssssiiiiyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
The paper has specified all the training and test details.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We provide the Std of the results, where the proposed method exhibits the lowest Std and surpasses other methods in terms mean performance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The computational resource requirements of experiments on large real-world datasets can be found 5.2, while the remaining experiments can be produced on CPU.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: : http///eurrppp.peee http://uuurrs.ccppubeeeeeee httpp///ttt/ttthceccciieeeeellee
Answer: [Yes]
Justification: Sure about that.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We ensure that the creators or original owners of assets used the paper are properly credited, and the license and terms of use are explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi wwwode..cod...c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
19

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
20


Reviews:
[{"review_id": "7XGRpgvvVd", "reviewer": "Reviewer_8VJU", "summary": "Authors introduce a novel differentiable module (ST_k) for efficiently solving top-k problems. Their method relies on optimizing a differentiable form of an equivalent optimization problem, by proposing an approximation of ReLU differentiable everywhere. This equivalent optimization problem contains a single parameter \\lambda which is supposed to match the k-th largest element in the optimal case, and in practice is optimized over the dataset. Authors  experiment on synthetic data, showing that their loss objective results in a model that more closely matches the true decision boundary in this setting. Next, authors experiment with real-world applications on binary classification and long-tailed classification datasets, showing consistent improvement over other optimization losses.", "questions": "**Questions**\n- How do you tune the smoothing coefficient in practice? Is this a hyperparameter that needs to be sweeped over? That would somewhat deteriorate the efficiency of your module. I would like to see an ablation over the values of delta, i think it is important to see what/if any impact this choice has on model performance.\n- Did you experience any training instabilities with the two-step optimization scheme?", "limitations": "The authors mention as limitation the fact that they only applied ST_k to the average top-k loss. I think it would be good if they more explicitly mention future research directions that they are interested in/ that would be of value (there is space in the manuscript).", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Strengths**\n\n- Authors propose an efficient differentiable solution to the Top-K problem that integrates very easily into existing model architectures, and only has a single optimizable parameter.\n- Authors show consistent improvements both in synthetic and real-world tasks compared to sota methods.\n- The paper is well-written, authors provide ample motivation for their design choices.", "weaknesses": "**Weaknesses**\n\n- Although the improvements shown by the authors are consistent, they are also marginal - only improving slightly in for example CIFAR-100-LT classification and ImageNet-LT classification compared to average aggregation. \n- As authors mention, they only apply their module to a single Top-K Loss; ATK, making it unclear how ST_k could be applied in other scenarios and how well it generalizes to different settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Your positive feedback is very encouraging.\n\n**For your concerns:**\n\n> Although the improvements shown by the authors are consistent, they are also marginal - only improving slightly in for example CIFAR-100-LT classification and ImageNet-LT classification compared to average aggregation.\n\nWe acknowledge that the performance improvement brought by the smoothed Average Top-k Loss is not significant. \n\n* In the field of Computer Vision, classification accuracy is an extensively explored direction. In fact, any improvement is a further advancement beyond a high point.\n* By incorporating a single trainable parameter into the computation graph, ST$_k$ achieves performance enhancements without requiring additional computational resources or time, thereby making the proposed module energy-efficient.\n* Additionally, the adjustment to the aggregation method of the individual losses is compatible with almost all existing methods for handling imbalanced data, which is likely to attract widespread attention from the community focused on improving classification accuracy.\n\n\n> As authors mention, they only apply their module to a single Top-K Loss; ATK, making it unclear how ST_k could be applied in other scenarios and how well it generalizes to different settings.\n\nST$_k$ can be applied end-to-end to models in any scenario that involves Top-K problem (see page 3, lines 87-89).\n\n\n**For your questions:**\n\n> How do you tune the smoothing coefficient $\\delta$ in practice? Is this a hyperparameter that needs to be sweeped over? That would somewhat deteriorate the efficiency of your module. I would like to see an ablation over the values of delta, i think it is important to see what/if any impact this choice has on model performance.\n\n* The smoothing coefficient $\\delta$, 0.01 is a grid search determined value for all datasets in section 5.1 (see page 7, line 186) and was adopted in all the other experiments. \n* We believe that the ablation study is necessary and will include it in our final version. Thanks for the insightful suggestion.\n\n\n> Did you experience any training instabilities with the two-step optimization scheme?\n\n* As mentioned above (see page1, line 25), in the experiments conducted in this paper, ST$_k$ does not require a two-step optimization using block coordinate descent, but can synchronously update $\\lambda$ and model parameters using SGD/Adam. \n* Due to the elimination of points where gradients vanish, the training process is very stable, as evidenced by the standard deviations listed in Tables 2, 3, and 5.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FfseFBtt3j", "reviewer": "Reviewer_uCvC", "summary": "The authors proposed a differentiable layer to approximate the top-k loss in deep learning. The proposed layer is motivated from Eq. (1), replacing the ReLU with a smoothed ReLU (SReLU in Eq. (2)). The authors showed that the proposed layer is point-wise convergent to top-k loss. Numerical experiments validate the superiority of the proposed method.", "questions": "* In real-world problems, is it possible to design experiments to measure the top-k loss approximation error directly?\n* What is the theoretical advantage of $ST_k$ v.s. $AT_k$", "limitations": "N/A", "rating": 4, "confidence": 1, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The paper is over-all well written, easy to follow. Experiments on real-world datasets are provided, in addition to simulated results.\n* A fast, accurate approximate of top-k loss is important in deep ranking problems.\n* Experiments show that the proposed method is over-all better than baseline methods.", "weaknesses": "* The biggest concern is that most of the real-world experiments are done on small scale datasets. It would be more convincing if the authors could demonstrate their method on large-scale ranking problems, especially, large deep learning models.\n\n* The proposed method lacks strong theoretical guarantees. Point-wise convergence is somehow weak. As easy to expect, the convergence rate and error bound of the proposed method should depend on the data distribution, which is not discussed in the paper. It would be nice if the authors could show that under what conditions, the proposed method will converge faster than $AT_k$ and / or with smaller approximation error.\n\n* It is important to check how well the proposed method approximates top-k loss in real-world problems. However, most real-world experiments report model accuracy, which is an indirect metric of top-k loss approximation. Please consider to measure the top-k loss approximation error directly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback. \n\n\n**For your concerns:**\n\n> Limited dataset scale.\n\nWe hold a different view on this.\n\n* We conduct experiments on datasets such as ImageNet-LT and Places-LT, which, to the best of our knowledge, are the largest unbalanced visual classification datasets available. \n* Additionally, we conduct experiments on two machine translation tasks using the Transformer-Base model, which has 0.11B trainable parameters, approximates that of Large Language Models (LLMs).\n* In recent years, as model sizes have become increasingly larger, some works that contain large-scale experiments have become hard to follow for the community. We advocate for conclusions that can be validated at appropriate sizes without the need for validation at the level of LLMs.\n\n\n> Point-wise convergence is theoretically weak, expect more convincing experimental evidence.\n\n* In Table 1, ST$_k$ not only guides the model to achieve higher accuracy but also converges faster compared to AT$_k$.\n* we experiment with the time cost of a few common sorting algorithms, AT$_k$ and ST$k$ to calculate the ranking average. The specific scheme of the experiment is to find the Top-k (k=5) sum from 10,000 normally distributed samples. For AT$_k$ and ST$_k$, we iterate until the error is less than $10^{-2}$, and for each algorithm, we conduct 50 experiments and record the average time taken in the third column of the Table. \n\n| Algorithm    | Complexity         | Average Time(s) |\n|--------------|--------------------|-----------------|\n| BubbleSort | $\\mathcal{O}(n^2)$ | 20.42196$\\pm$3.7015        |\n| HeapSort    | $\\mathcal{O}(n\\log(n))$ | 0.1243$\\pm$0.0446   |\n| AT$_k$        | $\\mathcal{O}(n+k)$ | 0.2167$\\pm$0.1528     |\n| ST$_k$(Ours)| $\\mathcal{O}(n+k)$ | **0.0127**$\\pm$0.0020     |\n\nIn experiments, ST$_k$ is the fastest algorithm to solve the ranking average value, on the other hand, AT$_k$ lacks robustness in training due to the gradient discontinuity. \n\n\n> Consider measuring the top-k loss approximation error directly.\n\nUnlike synthetic datasets, real-world problems do not have an explicit theoretical decision boundary. Therefore, we cannot provide a measure like the ParaF1 score in Table 1, which assesses the overlap between theoretical decision boundary and its estimation.\n\nWe will consider including Precision, Recall, or F1-score in the final version, but these metrics would only serve as a supplement to Accuracy which cannot directly show the approximation performance.\n\n\n**For your Questions:**\n\n\n> In real-world problems, is it possible to design experiments to measure the top-k loss approximation error directly?\n\nHere comes a more fundamental question: in real-world classification problems, like ImageNet-classification, does an optimal model exist that minimizes the loss?\n\nEven if such an optimal model exists, it's likely that we cannot compute it with our finite resources, which is precisely why algorithms like Gradient Descent are so valuable. They allow us to approximate the optimal solution efficiently without the need to exhaustively search all possibilities.\n\n\n> What is the theoretical advantage of ST$_k$  v.s. AT$_k$\nST$_k$ converges faster theoretically.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nTGYDSo2du", "reviewer": "Reviewer_46Ey", "summary": "This paper proposes a differentiable module for solving the top-k problem. Specifically, the paper proposes to approximate the hinge function with a new differentiable function.   \nExperiments on binary classification, long-tailed classification, and regression tasks with $ST_k$ loss show some improvements over baselines, like Average loss, and $AT_k$ loss.", "questions": "(1) Explanation of why $ST_k$ outperforms $AT_k$.                  \n(2) The experiments on large-scale data, like ImageNet.    \n(3) Is the $ST_k$ loss complementary to current state-of-the-art long-tailed learning methods.    \n(4) Ablation for the optimization manner of model parameters and $\\lambda$.", "limitations": "Limitations are discussed.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "(1) The paper is written clearly and easy to follow.                 \n(2) The proposed new differentiable function in Eq.(2) can approximate the hinge function.            \n(3) The proposed method is flexible and is expected to be used in many scenarioes.", "weaknesses": "(1) For the implementation of $AT_k$, is it implemented by a sort algorithm or just Eq. (5)?\n     From my point of view, the performance with $ST_k$ loss should be bounded by the performance with $AT_k$ loss.\n     However, Tables 1,4 show that $ST_k$ outperforms $AT_k$.\n     Why is $ST_k$ loss better than $AT_k$ loss?\n\n(2) The proposed $ST_k$ loss is flexible and can be directly deployed on the traditional training paradigm.\n     It would be better to validate the effectiveness of $ST_k$ on large-scale data, like ImageNet, and iNaturalist.\n    \n(3) The authors claim that the average loss is easily overfitted on imbalanced data. With $ST_k$ loss and CLIP models, it achieves the state-of-the-art on ImageNet-LT and Places-LT.\n     Is the $ST_k$ loss complementary to other state-of-the-art long-tailed learning methods, like GPaCo[2] and BCL[1]?    \n     [1] Balanced Contrastive Learning for Long-Tailed Visual Recognition. CVPR 2022.     \n     [2] Generalized Parametric Contrastive Learning. TPAMI 2023. \n\n(4) The models trained with $ST_k$ loss could be optimized alternatively or simultaneously with SGD for model parameters and $\\lambda$. The ablation study is encouraged to be included.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and feedback. Below is a detailed point-by-point response addressing your main concerns and questions.\n\n\n> Explanation of why ST$_k$ outperforms AT$_k$.\n\n* ST$_k$ and AT$_k$ are not equivalent. Reference [1] proves the equivalence between AT$_k$ and MAT$_k$, as stated in Eq. (5). \n* The parameter $k$ is a hyper-parameter, usually we set $k$ to 0.9 $\\times$ batch size, which is a practical choice. However it's impossible to determine the optimal $k$ for the model.\n* During the optimization process, once $k$ is set, AT$_k$ strictly (and harshly) filters out all samples ranked beyond the ($k$+1)-th position. In contrast, ST$_k$, which uses SGD for optimization, does not always have $\\lambda$ as the k-th largest individual loss. Therefore, in the optimization process, $\\lambda$ also serves as a parameter aimed at minimizing the overall loss.\n\n[1] Ogryczak, Wlodzimierz, and Arie Tamir. \"Minimizing the sum of the k largest functions in linear time.\" Information Processing Letters 85.3 (2003): 117-122.\n\n\n> Lack of experiments on large-scale data, like ImageNet.\n\nImageNet is a highly balanced dataset, with nearly the same number of images in each of its 1,000 classes. It is not a long-tailed classification dataset. We conduct experiments on its long-tailed version, ImageNet-LT.\n\nIn addition to this, we also conduct experiments on 2 machine translation datasets. Due to the disparity in word frequencies, machine translation datasets are naturally long-tailed.\n\n\n> Is the ST$_k$ loss complementary to current state-of-the-art long-tailed learning methods.\n\nST$_k$ represents an improvement in the aggregation method of individual losses, making it complementary to almost all existing long-tailed learning methods.\n\nPaCo/GPaCo/BCL, after obtaining a pre-trained model through contrastive learning, can utilize ST$_k$ to aggregate cross-entropy individual losses during subsequent transfer learning.\n\n\n> Ablation for the optimization manner of model parameters and $\\lambda$\n\nIn this work, we use Adam optimizer. We will consider including SGD or BCD (Block Coordinate Descent) as ablation studies.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E3Cim5uNS2", "reviewer": "Reviewer_SwHE", "summary": "This paper addresses the Top-K problem by introducing a new loss function. Building on the Average Top-K Loss, the authors incorporate a smoothed ReLU function to create the $ST_k$ loss, which is fully differentiable. Through experiments on various datasets, they demonstrate the effectiveness of their approach.", "questions": "1. In lines 20-21, the authors mention that \"the cost of the ranking process can no longer be ignored.\" I am curious about how significant this ranking cost is in the context of the full forward pass of a deep learning model. Given that larger models require more computational resources for their intermediate layers, this ranking cost might be negligible in most cases. Additionally, there is no detailed comparison of time costs between the Average Top-K Loss $AT_k$ and the proposed $ST_k$ loss, apart from on the toy synthetic dataset.", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is clearly written and easy to follow.\n2. The authors conduct extensive experiments to evaluate the proposed $ST_k$ loss, using both synthetic and real-world datasets.", "weaknesses": "1. The core weakness lies in the lack of significant innovation. The main technical contribution is the replacement of the ReLU-like operation in an existing optimization method for the Top-K problem with a smoothed ReLU function, controlled by a hyperparameter for smoothness. Similar concepts already exist in various ReLU function variants, as seen in related works such as [1,2].\n2. The only large-scale experiment is a long-tailed classification task involving parameter-efficient fine-tuning of a CLIP model pretrained on ImageNet-21K. Given that this model already has substantial pre-trained knowledge, the results are less convincing. Evaluating the method by training a model from scratch on these tasks would better demonstrate its effectiveness in practical scenarios.\n3. The paper does not include an ablation study on the hyperparameter $\\delta$ to study its impact on the performance.\n\n[1]. Biswas, Koushik, et al. \"SMU: smooth activation function for deep networks using smoothing maximum technique.\" arXiv preprint arXiv:2111.04682 (2021).\n\n[2]. Hendrycks, Dan, and Kevin Gimpel. \"Gaussian error linear units (gelus).\" arXiv preprint arXiv:1606.08415 (2016).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and feedback. Below is a detailed point-by-point response addressing your main concerns and questions.\n\n\n**For your concerns:**\n\n\n> lack of significant innovation\n\nWe hold a different view on this.\n\n* The sorting optimization algorithm proposed by [1] has been around for two decades and has not been widely applied, largely due to the instability caused by the non-differentiable nature of the optimization function.\n* This paper introduces a new SReLU that possesses point-wise convergence to the original ReLU, a feature not present in other shape-similar ReLU variants. Our experiments demonstrate that SReLU surpasses other ReLU variants at least in solving the Top-K problem.\n* We have adapted the improved version of [1] for training deep learning models, and our experiments have shown that the trainable parameter $\\lambda$ and model parameters from [1] do not require a two-step optimization using BCD but can be integrated into the computation graph and optimized uniformly with SGD.\n* Utilizing ST$_k$, we refresh the state-of-the-art (SOTA) records for two long-tailed classification leaderboards without additional computational resources. Moreover, ST$_k$ is almost complementary to any existing long-tailed learning methods. This is likely to attract widespread attention from the relevant communities.\n\n[1] Ogryczak, Wlodzimierz, and Arie Tamir. \"Minimizing the sum of the k largest functions in linear time.\" Information Processing Letters 85.3 (2003): 117-122.\n\n> Evaluating the method by training a model from scratch\n\nIn this work, we trained models from scratch on two machine translation datasets, and the experimental results show that ST$_k$ improves model performance. We will consider including results from training ImageNet-LT from scratch,  in our final version.\n\n\n> An ablation study on hyperparameter $\\delta$.\n\n* The smoothing coefficient $\\delta$, 0.01 is a grid search determined value for all datasets in section 5.1 (see page 7, line 186) and was adopted in all the other experiments. \n* We believe that the ablation study is necessary, and here we provide an ablation study on the CIFAR-100-LT dataset. All other settings remain the same as those in the paper, with only $\\delta$ being changed. \n\n| $\\delta$   | 0.1 | 0.05 | 0.01 | 0.005 |\n|--------------|-------|--------|-------|--------|\n| Accuracy | 88.8 | 89.4 | 89.8 | 89.8 |\n\n* We will include extensive ablation studies in our final version, thanks again for the insightful suggestion.\n\n\n**For your question:**\n\n> Is the ranking cost negligible？\n\nWe experiment with the time cost of a few common sorting algorithms, AT$_k$ and ST$k$ to calculate the ranking average. The specific scheme of the experiment is to find the Top-k (k=5) sum from 10,000 normally distributed samples. For AT$_k$ and ST$_k$, we iterate until the error is less than $10^{-2}$, and for each algorithm, we conduct 50 experiments and record the average time taken in the third column of the Table. \n\n| Algorithm    | Complexity         | Average Time(s) |\n|--------------|--------------------|-----------------|\n| BubbleSort | $\\mathcal{O}(n^2)$ | 20.42196$\\pm$3.7015        |\n| HeapSort    | $\\mathcal{O}(n\\log(n))$ | 0.1243$\\pm$0.0446   |\n| AT$_k$        | $\\mathcal{O}(n+k)$ | 0.2167$\\pm$0.1528     |\n| ST$_k$(Ours)| $\\mathcal{O}(n+k)$ | **0.0127**$\\pm$0.0020     |\n\nIn experiments, ST$_k$ is the fastest algorithm to solve the ranking average value, on the other hand, AT$_k$ lacks robustness in training due to the gradient discontinuity. \n\nIn addition, we conduct time cost experiment by training a Large Language Model. On a single H800, we tune a Llama-8B using LoRA method. After iterating 1000 steps, the average time cost is 1.9824s per step, while the performing the quick sort on the individual losses cost 0.0317s per step (# of tokens per batch = 2048).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate all your comments.

Here is a summary of the strengths and a general response to the concerns received:

**Strengths:**

Firstly, we appreciate that all reviewers think our paper to be clearly written and easy to follow.
- Reviewer 8VJU noted that our framework is efficient and integrates very easily into existing model architectures, and believed that our motivation was ample.
- Reviewer uCvC found our method to be fast and overall better than baseline methods.
- Reviewer 46Ey thought our method was flexible and is expected to be used in many scenarios.
- Reviewer SwHE observed that we conducted extensive experiments, using both synthetic and real-world datasets.

**Concerns:**

> The limited scale of the dataset:

* We conduct experiments on datasets such as ImageNet-LT and Places-LT, which, to our knowledge, are among the largest imbalanced visual classification datasets available.
* Additionally, we have conducted experiments on two machine translation tasks using the Transformer-Base model, which has 0.11B trainable parameters.
* In recent years, as model sizes have become increasingly larger, some works that contain large-scale experiments have become hard to follow for the community. We advocate for conclusions that can be validated at appropriate sizes without the need for validation at the level of LLMs.

> To add more ablation studies and time-cost experiments:

We add 2 additional experiments during rebuttal and will consider including more experiments in our final version.

Meta-review: The paper received the following ratings:  R1 - 7 (accept), R2 - 4 (borderline reject, no reaction to rebuttal), R3 - 4 (borderline reject), R4 - 6 (weak accept).

Overall reviewers highlighted that paper is clearly written and easy to follow, that the proposed top-k loss is an important problem, and that the experiments are clear and show the improvement by the provided loss. 

The main weaknesses listed are that the evaluation might be a bit limited and that more theoretical would be helpful. The authors provided a solid rebuttal which provided additional information. 
While I share the concerns of the reviewers for the original paper, I think that the paper can significantly improve if all comments and additional experiments from the rebuttal would be incorporated in the main paper and/or the supplementary. Given the rebuttal, I'm also confident that the authors are interested in improving the paper and will do this. 

I therefore recommend accepting the paper.
Decision: Accept (poster)