Title: ST$_k$: A Scalable Module for Solving Top-k Problems
Venue: NeurIPS 2024
Accepted: yes
Keywords: Average Top-k Loss

Smoothing ReLU

Ranking Problem
Abstract: The cost of ranking becomes significant in the new stage of deep learning. We propose ST$_k$, a fully differentiable module with a single trainable parameter, designed to solve the Top-k problem without requiring additional time or GPU memory. Due to its fully differentiable nature, ST$_k$ can be embedded end-to-end into neural networks and optimize the Top-k problems within a unified computational graph. We apply ST$_k$ to the Average Top-k Loss (AT$_k$), which inherently faces a Top-k problem. The proposed ST$_k$ Loss outperforms AT$_k$ Loss and achieves the best average performance on multiple benchmarks, with the lowest standard deviation. With the assistance of ST$_k$ Loss, we surpass the state-of-the-art (SOTA) on both CIFAR-100-LT and Places-LT leaderboards.

ST k : A Scalable Module for Solving Top-k Problems ∗

Abstract

The cost of ranking becomes significant in the new stage of deep learning. We propose ST k , a fully differentiable module with a single trainable parameter, designed to solve the Top-k problem without requiring additional time or GPU memory. Due to its fully differentiable nature, ST k can be embedded end-to-end into neural networks and optimize the Top-k problems within a unified computational graph. We apply ST k to the Average Top-k Loss (AT k ), which inherently faces a Top-k problem. The proposed ST k Loss outperforms AT k Loss and achieves the best average performance on multiple benchmarks, with the lowest standard deviation. With the assistance of ST k Loss, we surpass the state-of-the-art (SOTA) on both CIFAR-100-LT and Places-LT leaderboards.

1 Introduction

The ranking problem is quite common in the field of AI. For imbalanced datasets, the Average Top-k (AT k ) Loss is more suitable than the conventional Average Loss [Lyu et al., 2020]. In the context of ambiguous classification tasks, Top-k Learning allows the ground truth to fall within the largest k probabilities, enhancing the model's generalizability [Lapin et al., 2017, Berrada et al., 2018, Petersen et al., 2022]. For language models, the Top-k sampling method helps the models select the top k most probable words during text generation, producing more fluent and coherent sentences. Distributed learning systems employ Top-k sparsification with error compensation (Top-k SGD) to reduce communication traffic without noticeably impacting model accuracy [Chen et al., 2018, Lin et al., 2018, Shi et al., 2019]. However, as deep learning models continue to grow in size, the cost of the ranking process becomes increasingly significant . For example, on a single NVIDIA H800 GPU, tuning a Llama-8B model using the LoRA [Hu et al., 2022] method with a batch size of 4096 and 1000 iterations takes 990.14 seconds. In particular, performing QuickSort on the individual losses consumes 86.33 seconds.

In this work, we propose ST k (Smoothed Top-k), a scalable module for Top-k problems. By adding only a single trainable parameter, ST k is able to solve the Top-k problem in O ( n + k ) steps. Due to the fully differentiable nature of ST k , it can be embedded end-to-end as a layer. Experiments show that we can even add λ to the computational graph for unified optimization using Stochastic Gradient Descent (SGD), which means that we do not need to consider the time cost of solving the Top-k problems within the computational graph. However, we can still enjoy the performance improvements that Top-k optimization provides. The contributions of this work are summarized as follows:

We propose a uniformly convergent approximation of the ReLU function.

We propose an efficient and robust Smoothed Top-k module, ST k .

We apply ST k Module to smooth AT k Loss, resulting in performance improvement and refreshing the state-of-the-art methods on two long-tailed learning leaderboards.

We design an imbalanced classification dataset with a theoretical decision boundary.

For experiments, by applying the ST k Module, we smooth the AT k Loss into ST k Loss. The computation time of ST k Loss is almost identical to that of the average aggregating method, significantly faster than the sorting-required AT k Loss, and it exhibits the best performance. Experiments on synthetic datasets demonstrate that models trained with ST k Loss most closely approximate the theoretical decision boundary. On benchmarks of imbalanced binary classification, models trained by ST k Loss exhibit the lowest average misclassification rate and the lowest standard deviation. On regression datasets, models trained with ST k Loss exhibit the lowest (RMSE). Experiments on large real-world datasets demonstrate that ST k Loss, as an aggregating trick for individual losses, is a scalable technique that improves accuracy on long-tailed benchmarks. With the help of ST k Loss, we surpass the state-of-the-art (SOTA) on both the CIFAR-100-LT and Places-LT leaderboards.

1.1 Related Work

In current research, although there are many studies on individual loss, the general characteristics of aggregate loss are often overlooked. In the existing machine learning literature, a related line of work is the data subset selection problem [Wei et al., 2015], which aims to select a subset from a large training dataset for model training while minimizing average loss. Curriculum learning [Bengio et al., 2009] and self-paced [Kumar et al., 2010] learning are two recent learning schemes. They organized the training process into several iterative stages, gradually including training data from easy to difficult to learn, where the difficulty level is measured by individual loss. Therefore, each training session in these methods corresponds to the average aggregate loss in the selected subset. The difficulties encountered by the Average Loss when dealing with imbalanced data, as discussed in [Shalev-Shwartz and Wexler, 2016, Huang et al., 2020], prompted the exploration of more robust aggregate losses. Among these, Lyu et al. [2020] introduced the AT k loss, which averages the k largest individual losses, and exhibits advanced performance on imbalanced datasets.

Many classification tasks in the real world have inherent label confusion, as mentioned in Berrada et al. [2018]. This confusion may arise from various factors, such as incorrect labels, incomplete annotations, or some fundamental ambiguities that even confuse the true labels for human experts. Therefore, some works proposed the concept of Top-k Learning in the field of image classification to address the issues of multiple semantics and semantic confusion in images [Lapin et al., 2017].

Berrada et al. [2018] proposed a method to partially smooth the Top-k Learning loss function, but did not completely solve the sorting problem in the loss function and introduced a sorting computation of C k n . Petersen et al. [2022] proposed a Split Selection Network (SSN) based on sorting networks, which made the Top-k process differentiable and achieved the state of the art on ImageNet-1K at that time. However, the computation required by this method is cumbersome and multilayered. Sorting networks are similar to algorithms like QuickSort with time complexities of O ( n log n ) .

1.2 Notational Conventions

Let N n denotes the set { 1 , ..., n } and I { a } denotes the indicator function (which is 1 when the proposition a is true and 0 otherwise). Thus, the sign function can be defined as: sign ( x ) = I { x> 0 } -I { x< 0 } . The Hinge function can be defined as: [ x ] + = max { 0 , x } . We use ∥ x ∥ 1 , ∥ x ∥ 2 , and ∥ x ∥ ∞ to represent the ℓ 1 , ℓ 2 , and ℓ ∞ norms of x , respectively. For the set L = ℓ 1 , ℓ 2 , ..., ℓ n , ℓ [ k ] denotes the k -th largest element, so we have ℓ [1] ≥ ℓ [2] ≥ ... ≥ ℓ [ n ] . In supervised learning problems, our training set typically contains an input set and a target set, the input set coming from the input domain X , and the target set from the target domain Y , and we use their joint domain Z = X × Y to represent the range of the dataset. The training set S = z 1 , z 2 , ..., z n is a subset of Z , where z i = ( x i , y i ) . Our task is to find a predictor f : X → Y from the function family H that can predict the corresponding target y based on the new input x . To evaluate the effect of the predictor, we need to introduce an individual loss function ℓ : Y × Y → R + , where ℓ = ℓ ( f ( x ) , y ) usually reflects some distance between the prediction ˆ y = f ( x ) and the true value y . The training

process of the predictor can be described as using the gradient descent algorithm to optimize an objective function (usually minimizing the loss function). The objective function can generally be written as L ( f, S ) + Ω( f ) , where L ( f, S ) is the aggregated individual loss function, and Ω( f ) is a regularization term ( ℓ 1 or ℓ 2 regularization term). The loss function L ( f, S ) is usually Average Loss, that is, L avg ( f, S ) = 1 n ∑ n i =1 ℓ ( f ( x i ) , y i ) . However, recent works, have shown some drawbacks of the average loss in adapting to imbalanced data distributions [Shalev-Shwartz and Wexler, 2016], and explored choices other than the average loss for the aggregate loss formed from individual losses, e.g., the maximum (aggregate) loss, L max ( f, S ) = max i ∈ N n ℓ ( f ( x i ) , y i ) .

2 ST k Architecture

Smoothed Top-k Module

Suppose we have a set of elements { e i } n i =1 , then the Top-k problem can be describe as:

find the k -th largest element e [ k ] ;

find the sum of top-k largest elements ∑ k i =1 e [ k ] .

This process can certainly be achieved through conventional sorting and summation. However, in the worst-case scenario, the cost of solving the Top-k problem can reach O ( n 2 ) . Ogryczak and Tamir [2003] proposed an equivalent optimization form to solve the Top-k problem and proved its linear convergence.

However, this surrogate objective function suffers from a non-differentiable point at e i = λ , which makes it challenging to optimize. The key to solving this problem lies in designing a function approximating [ · ] + , which can be regarded as a rectified linear unit (ReLU) function. Here, we introduce the Smoothed ReLU (SReLU).

where δ is a hyperparameter (usually we set δ = 0 . 01 ). It can be observed from Figure 2 that as δ decreases, SReLU increasingly approximates the ReLU function. In fact, SReLU converges uniformly to ReLU as δ → 0 + ; a detailed proof of this uniform convergence is provided in Proposition 1 of Appendix A.1. With the help of SReLU, the objective function (1) can be smoothed as:

the optimal λ ∗ = e [ k ] which is the k -th largest element. In the following section, we will introduce an application scenario for ST k .

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

0.0

0.2

0.4

0.6

0.8

1.0

ReLU

SReLU,

=0.1

SReLU,  =0.05

3 From AT k Loss to ST k Loss

In the training process of a neural network, we first choose a form of individual loss (e.g., logistic loss, hinge loss, mean square loss, or cross-entropy loss). Then, we aggregate all individual losses to calculate their average, which is the most common aggregate loss function: Average Loss. The Average Loss is widely used in a myriad of deep learning tasks. This widespread application stems from the robust theoretical foundations [Bartlett et al., 2006, De Vito et al., 2005]. However, Average Loss tends to overfit the training data, especially on imbalanced datasets [Shalev-Shwartz and Wexler, 2016, Huang et al., 2020]. This has inspired the motivation to find other forms of aggregate loss, such as the maximum value among individual losses (referred to as Maximum Loss). The Average Top-k (AT k ) Loss was introduced by Lyu et al. [2020]:

which represents the average of the largest k individual losses. According to the derivation in [Ogryczak and Tamir, 2003], this ranking loss can be written in the following equivalent form:

With the help of the ST k Module, L mat -k ( f, S ) can be reconstructed as

where ℓ i = ℓ ( f w model ( x i ) , y i ) is the individual loss of sample i , and w model represents the set of parameters of the predictor f . It is easy to verify that lim δ → 0 + L st -k = L mat -k , given Proposition

1 in Appendix A.1.

The approximation error between the smoothed loss function and the original loss function can be uniformly bounded by δ/ 2 , ∀ ℓ .

When ℓ is convex, Equation (6) exhibits joint convexity with respect to the parameters ( w model , λ ) , making the problem a special case of the non-linear multiple choice knapsack problem [Zemel, 1984], which has at most q = 2 roots. These roots can be found in constant time, allowing the problem to be solved in O ( n · ln q ) = O ( n ) time when q is fixed [Megiddo, 1984]. Therefore, it can be iteratively updated using relatively simple algorithms. For example, in the case of batch learning, the block coordinate descent (BCD) method [Nocedal and Wright, 1999] can be employed, where w model and λ are updated alternately after initialization.

BCD-ST k :

The convergence of the above coordinate descent algorithms can be found in Luo and Tseng [1992], Saha and Tewari [2013], Tseng [2001].

Furthermore, empirical evidence suggests that we do not need to spend extra time optimizing λ separately, incorporating λ into the computational graph of w model for unified optimization using Stochastic Gradient Descent (SGD) [Bottou and Bousquet, 2008, Shamir, 2011, Srebro and Tewari, 2010], performance improvements can still be achieved.

SGD-ST k :

where η t is the size of the update step, and when η t ∼ 1 √ t ′ , the stochastic gradient descent method can ensure convergence to a local minimum of Equation (6) [Shamir, 2011, Srebro and Tewari, 2010]. By eliminating points where the gradients are discontinuous, the training process becomes more stable, and converges faster, as experimentally demonstrated by the standard deviations reported in Tables 3, 4, and 9, and the time cost reported in Table 2.

4 Synthetic Experiments

4.1 Time Cost

We first perform experiments to compare the time costs of two standard sorting algorithms, AT k , and ST k , in calculating the ranking average. The experimental setup involves finding the Top-k (k=5) sum from 10,000 standard normally distributed samples. For both AT k and ST k , we iterate until the error is less than 10 -2 . For each algorithm, we run 50 experiments and record the average time taken.

As shown in Table 1, ST k indeed demonstrates linear time complexity while also exhibiting a stable optimization process.

4.2 Gaussian Distributed Dataset

To illustrate the capability of ST k Loss in approximating the ideal decision boundary, we design a Gaussian-distributed dataset. The dataset is generated by a pre-set covariance matrix Σ ∈ R d × d , and mean vectors µ 1 , µ 2 ∈ R d corresponding to two categories. We set d = 200 , Σ jk = 0 . 25 0 . 5+ | j -k | , µ 1 = 0 d and µ 2 = (1 , 1 , ..., 1 , 0 , 0 , ..., 0) ⊤ , which has 10 ones. For the predictor, we use a Logistic Regression (LR) model f ( x ) = sigmoid ( ω ⊤ x + b ) . The detailed derivation process of the decision boundary can be found in the Appendix A.2. Here, we present the conclusion directly. For two normal populations with a given covariance matrix Σ , and means µ 1 and µ 2 , the LR model has a theoretical decision boundary:

Figure 3 provide an example on 2D plane, we set Σ ( j,k ) = 0 . 8 | j -k | , µ 1 = [0 , 0] ⊤ , µ 2 = [2 , 2] ⊤ , thus according to our derivation above, ω ∗ = [1 . 111 , 1 . 111] ⊤ , b ∗ = -0 . 836 , where the black dashed line is the theoretical boundary.

We use ParaF 1 [Tian and Gu, 2017] to measure the overlap of estimated supports and true supports:

where precision = | support(ˆ ω ) ∩ support( ω ∗ ) | | support(ˆ ω ) | , recall = | support(ˆ ω ) ∩ support( ω ∗ ) | | support( ω ∗ ) | , where ˆ ω is the estimated value of the parameters obtained by the predictor, and ω ∗ is the theoretical value of the parameters mentioned above. The operator | · | is used to find the cardinality of the set, which is the number of elements, and support( · ) is the support set of the vector, which refers to the set of indices of its non-zero elements.

Other basic settings are as follows. The loss function we use is the binary cross-entropy loss. To obtain a sparse solution, we add an ℓ 1 regularization term. The number of samples is 10,000 for the training set and 2,500 each for the validation set and the test set. We implement early stopping of the iteration based on the accuracy

2

0

2

4

4

2

0

2

4

6

Positive

Negative

Decision Boundary

curve in the validation set; that is, we break when the increase in validation accuracy within 200 steps is less than 10 -4 . For each meta-experiment, we repeat it 50 times and take the average. Now we conduct two groups of experiments as follows.

Aggregate Losses and ReLU Varients. In these experiments, we set the ratio of positive to negative samples at 8:2 for the training set, while the validation set and the test set remain 1:1. We also compare SReLU with other variants of ReLU. The smoothing coefficient for SReLU is set to 0.01, and the settings for the other ReLU variants remain at their defaults. We use Adam as our optimizer,

setting the batch size to 512, while keeping the other hyperparameters as default. According to Equation (8), we could compute the theoretical value of ω ∗ , which is a vector with only the first 10 elements non-zero. Sensitive Analysis. In these experiments, we only adjust the positive-negative ratio and retain the rest of the settings from the previous experiment.

10

3

10

2

10

1

Negative Ratio

72.5

75.0

77.5

80.0

82.5

85.0

87.5

0.5

0.1

0.05

0.01

0.005

0.001

ST

k

AT

k

Average

10

3

10

2

10

1

Negative Ratio

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

ParaF1-score

0.5

0.1

0.05

0.01

0.005

0.001

ST

k

AT

k

Average

Table 2 presents the Accuracy and ParaF 1 score achieved when different aggregate loss functions are combined with the LR model and the cross-entropy loss, trained to convergence. With the help of SReLU, ST k Loss achieves performance that surpasses that of all other aggregate losses. Due to its property of being a real uniformly convergent approximation (not just similar in shape), SReLU outperforms other ReLU variants in this scenario. Figure 4 shows the relationship between accuracy, ParaF 1 -score, and negative sample ratio (negative ratio). ST k is less sensitive to the imbalance ratio,

5 Real World Applications

5.1 Binary Classification

We select binary classification datasets from the KEEL 2 and UCI 3 databases; see Table 9.

Next, we tested the performance of different forms of aggregate loss on binary classification benchmarks. The individual loss function can generally be chosen as Logistic or Hinge, which are defined

as follows:

Logistic :

ℓ ( f ( x ) , y ) = log(1 + exp( - yf ( x )));

Hinge :

ℓ ( f ( x ) , y ) = [1 - yf ( x )] + .

The prediction model is a two-layer MLP with 10 nodes in the hidden layer, activated by the ReLU function between the two fully connected layers. To increase the stability of the training process, we added an ℓ 2 regularization term to the loss function Ω( w ) = 1 2 C ∥ w ∥ 2 2 . We divided the datasets into training, validation, and test sets in a 0.5 : 0.25 : 0.25 ratio. Nowadays, SGD is generally replaced by the mini-batch method instead of stochastic gradient descent with a single sample point, which is faster and more robust. To accommodate datasets of varying sizes in the article, we set the batch size to 16.

The hyperparameters in the experiment include k in MAT k and AT k , the coefficient of the regularization term C , the initial learning rate η , and the smoothing coefficient δ . These hyperparameters will be selected based on their convergence accuracy performance on the validation set (for each combination of hyperparameters, we repeat the experiment fifty times and take the average of their prediction accuracy on the validation set as the basis for parameter selection). The search spaces for several hyperparameters are as follows: k ∈ { 1 }∪ [0 . 1 : 0 . 1 : 1] ; C ∈ { 10 0 , 10 1 , 10 2 , 10 3 , 10 4 , 10 5 } ; η ∈ { 0 . 1 , 0 . 05 , 0 , 01 , 0 . 005 , 0 . 001 } ; δ ∈ { 0 . 1 , 0 . 01 , 0 . 001 , 0 . 0001 } .

Since traditional gradient descent is too sensitive to the choice of step size, which is not conducive to our pure comparison of the convergence speed and accuracy of the loss function, we use the AdaGrad algorithm to iteratively update the learning rate. During the learning process, to avoid overfitting the model, we record the accuracy of the MLP predictor on the validation set after each iteration, and perform an early stop when the accuracy does not increase (the increase in the accuracy of the predictor on the validation set is less than 10 -6 after 50 steps) and roll back the model to the checkpoint with the highest accuracy on the validation set during the entire training process; otherwise, we continue training until convergence. Tables 3 and 4, respectively, show the average probability of misclassification (%) of the models trained to convergence in 50 experiments in the test set under individual Logistic and Hinge loss (the standard deviation of the 50 experiments is listed in parentheses).

In the results shown in Tables 3 and 4. Almost all the lowest misclassification rates appear in the ST k trained models. Furthermore, models trained with ST k Loss exhibit the lowest standard deviation, which to some extent demonstrates the robustness of the training process.

5.2 Long-Tailed Classification

The interest in long-tailed classification tasks has increased with the advent of large vision-language models such as contrastive language-image pre-training (CLIP) Radford et al. [2021]. Long-tailed versions of recognized datasets were built by the community. Using the Pareto distribution ( α = 6 ), the ImageNet-1K and Places datasets can be sampled to create ImageNet-LT and Places-LT datasets, as described in [Liu et al., 2019]. Consider two types of imbalance [Cui et al., 2019, Buda et al., 2018], Cao et al. [2019] built CIFAR-10-LT ( ρ = 10 ) and CIFAR-100-LT ( ρ = 100 ), where ρ = max i { n i } / min i { n i } represents the imbalance ratio, defined as the ratio between the sample sizes of the most frequent and least frequent class. Due to varying word frequencies, machine translation, or

more generally, text generating, can inherently be considered as long-tailed classification tasks. We choose the WMT2017 4 and IWSLT2014 5 datasets for the experiments.

800-

639

600

Number of Samples

674

112

127

400

408

819

200-

D

HeadClasses

TailClasses

Class

Settings For visual classification tasks, we use Parameter-Efficient Long-Tailed (PEL) Recognition [Shi et al., 2023], which primarily leverages information from a text encoder to adjust classification probabilities. The text encoder in PEL is derived from the pre-trained CLIP model, and the backbone is a Vision Transformer (ViT-Base) [Dosovitskiy et al., 2021] pre-trained in ImageNet-21K. We freeze the backbone and train the branch model of Parameter Efficient Fine-Tuning (PEFT) [Jia et al., 2022, Hu et al., 2022, Houlsby et al., 2019]. We simply replaced the Average aggregation of individual losses in the Logit Adjust (LA) [Menon et al., 2021] loss with AT k or ST k . For k , we set it at 0 . 9 × batch size, and δ = 0 . 01 . Then we report the accuracy of the balanced test sets. Visual classification experiments can be implemented on a single L20 GPU, with varying durations ranging from 20 to 200 minutes.

For translation tasks, we use the training workflows provided by OpenNMT 6 [Klein et al., 2018] and FAIR-Seq 7 [Ott et al., 2019], which are very easy to reproduce. Models are essentially different sizes of Transformers (see the footnotes). In the training process of language models, the actual batch size, calculated as the sequence length multiplied by the number of sequences, is often dynamic. Consequently, we set k/ batch size = 0 . 96 . We simply replaced the Average aggregation of individual losses in the Cross Entropy (CE) loss with AT k or ST k . We calculate the bilingual evaluation understudy (BLEU) scores for the models. The BLEU for IWSLT2014-trained models are derived from their test set, while for the WMT2017-trained models, we evaluate their BLEU on the Newstest2016 test set. Training these Transformer-based models would take about 10 hours on a single RTX 4090.

Overall Results ST k Loss, as an aggregating trick for individual losses, is a scalable technique that improves accuracy on long-tailed benchmarks. The models trained by ST k Loss surpass SOTA on the CIFAR-100-LT and Places-LT leaderboards.

5.3 Regression Tasks

For descriptions of the four regression datasets, see Table 9. In regression tasks, the individual loss is generally the distance ( ℓ 1 or ℓ 2 distance) between the predicted value and the ground truth. Next, we present the specific settings of the experiment. For the Housing, Abalone, and Cpusmall datasets, we normalize the output to between [0 , 1] ; for the Sinc dataset, we first randomly sample 1000 points ( x i , y i ) from the function, where x i ∈ [ -10 , 10] , then we use the Radial Basis Function (RBF) kernel to map x i into the kernel space. We select 10 RBF kernels, resulting in a 10-dimensional input x = [ k ( x, c 1 ) , ..., k ( x, c 10 )] , where k ( x, c j ) = exp ( -( x -c j ) 2 ) . Furthermore, we add Gaussian noise N (0 , 0 . 2 2 ) to the target output y .

For individual losses, we choose the absolute value loss ( ℓ 1 ) and the squared loss ( ℓ 2 ). The prediction model is the same as above, a two-layer MLP with 10 nodes in the hidden layer, activated by the ReLU function between the two fully connected layers. As in binary classification experiments, we randomly divide the dataset into training, validation, and test sets, with hyperparameters chosen based on the performance in the validation set, repeated 50 times, and statistics on the test set are reported.

Tables 6 show the average RMSE and standard deviation of 50 experiments in regression datasets with different aggregate losses combined with absolute individual loss and squared individual loss (given the experience of the previous experiment, we no longer compare with Maximum Loss here), and ST k shows further performance improvement after smoothing.

6 Conclusion and Limitation

The proposed ST k Module can effectively solve the Top-k problem within neural networks, with no additional GPU memory or ranking time. Due to its fully differentiable nature, the training process are stable. By applying ST k Module to the Average Top-k Loss, we achieve significant improvements across numerous benchmarks.

The limitation of this study is that we have not yet evaluated the ST k Module for additional application scenarios. The effectiveness of the ST k Module is only demonstrated in smoothing AT k loss. We hope that future research will further explore the potential utility of the ST k Module.

References

Siwei Lyu, Yanbo Fan, Yiming Ying, and Bao-Gang Hu. Average top-k aggregate loss for supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(1):76-86, 2020.

Maksim Lapin, Matthias Hein, and Bernt Schiele. Analysis and optimization of loss functions for multiclass, top-k, and multilabel classification. IEEE Transactions on Pattern Analysis and Machine Intelligence , 40(7):1533-1554, 2017.

Leonard Berrada, Andrew Zisserman, and M. Pawan Kumar. Smooth loss functions for deep top-k classification. In International Conference on Learning Representations , 2018.

Felix Petersen, Hilde Kuehne, Christian Borgelt, and Oliver Deussen. Differentiable top-k classification learning. In International Conference on Machine Learning , pages 17656-17668. PMLR, 2022.

Chia-Yu Chen, Jungwook Choi, Daniel Brand, Ankur Agrawal, Wei Zhang, and Kailash Gopalakrishnan. Adacomp: Adaptive residual gradient compression for data-parallel distributed training. In Proceedings of the AAAI conference on artificial intelligence , volume 32, 2018.

Yujun Lin, Song Han, Huizi Mao, Yu Wang, and Bill Dally. Deep gradient compression: Reducing the communication bandwidth for distributed training. In International Conference on Learning Representations , 2018.

Shaohuai Shi, Xiaowen Chu, Ka Chun Cheung, and Simon See. Understanding top-k sparsification in distributed deep learning. arXiv preprint arXiv:1911.08772 , 2019.

Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations , 2022.

Kai Wei, Rishabh Iyer, and Jeff Bilmes. Submodularity in data subset selection and active learning. In International Conference on Machine Learning , pages 1954-1963. PMLR, 2015.

Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual International Conference on Machine Learning , pages 41-48, 2009.

M Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. Advances in Neural Information Processing Systems , 23, 2010.

Shai Shalev-Shwartz and Yonatan Wexler. Minimizing the maximal loss: How and why. In International Conference on Machine Learning , pages 793-801. PMLR, 2016.

Lang Huang, Chao Zhang, and Hongyang Zhang. Self-adaptive training: beyond empirical risk minimization. Advances in Neural Information Processing Systems , 33:19365-19376, 2020.

Wlodzimierz Ogryczak and Arie Tamir. Minimizing the sum of the k largest functions in linear time. Information Processing Letters , 85(3):117-122, 2003.

Peter L Bartlett, Michael I Jordan, and Jon D McAuliffe. Convexity, classification, and risk bounds. Journal of the American Statistical Association , 101(473):138-156, 2006.

Ernesto De Vito, Andrea Caponnetto, and Lorenzo Rosasco. Model selection for regularized leastsquares algorithm in learning theory. Foundations of Computational Mathematics , 5:59-85, 2005.

Eitan Zemel. An o (n) algorithm for the linear multiple choice knapsack problem and related problems. Information Processing Letters , 18(3):123-128, 1984.

Nimrod Megiddo. Linear programming in linear time when the dimension is fixed. Journal of the ACM (JACM) , 31(1):114-127, 1984.

Jorge Nocedal and Stephen J Wright. Numerical optimization . Springer, 1999.

Zhi-Quan Luo and Paul Tseng. On the convergence of the coordinate descent method for convex differentiable minimization. Journal of Optimization Theory and Applications , 72(1):7-35, 1992.

Ankan Saha and Ambuj Tewari. On the nonasymptotic convergence of cyclic coordinate descent methods. SIAM Journal on Optimization , 23(1):576-601, 2013.

Paul Tseng. Convergence of a block coordinate descent method for nondifferentiable minimization. Journal of Optimization Theory and Applications , 109(3):475, 2001.

L Bottou and O Bousquet. The tradeoffs of large scale learning, advances in neural information processing systems, 20, 2008.

Ohad Shamir. Making gradient descent optimal for strongly convex stochastic optimization. CoRR, abs/1109.5647 , 2011.

Nathan Srebro and Ambuj Tewari. Stochastic optimization for machine learning. ICML Tutorial , 2010.

Lu Tian and Quanquan Gu. Communication-efficient distributed sparse linear discriminant analysis. In Artificial Intelligence and Statistics , pages 1178-1187. PMLR, 2017.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning , pages 8748-8763. PMLR, 2021.

Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Largescale long-tailed recognition in an open world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2537-2546, 2019.

Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9268-9277, 2019.

Mateusz Buda, Atsuto Maki, and Maciej A Mazurowski. A systematic study of the class imbalance problem in convolutional neural networks. Neural networks , 106:249-259, 2018.

Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution-aware margin loss. Advances in Neural Information Processing Systems , 32, 2019.

Jiang-Xin Shi, Tong Wei, Zhi Zhou, Xin-Yan Han, Jie-Jing Shao, and Yu-Feng Li. Parameter-efficient long-tailed recognition. arXiv preprint arXiv:2309.10019 , 2023.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2021.

Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision , pages 709-727. Springer, 2022.

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning , pages 2790-2799. PMLR, 2019.

Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. In International Conference on Learning Representations , 2021.

Guillaume Klein, Yoon Kim, Yuntian Deng, Vincent Nguyen, Jean Senellart, and Alexander M. Rush. Opennmt: Neural machine translation toolkit, 2018.

Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations , 2019.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16000-16009, 2022.

A Appendix: Proposition Proof

A.1 Uniform Convergence

[Proposition 1] We have that SReLU uniformly converges to ReLU as δ → 0 + , that is, lim δ → 0 + sup x | SReLU δ ( x ) -[ x ] + | = 0 .

Proof.

Note that D( x ) = 2(ReLU( x ) -SReLU( x )) = | x | -δ ( √ x 2 δ 2 +1 -1 ) , we have

For any x and δ , the form 0 < 1 -1 √ x 2 δ 2 + √ x 2 δ 2 +1 < 1 , therefore, for any ϵ > 0 , if δ < ϵ , then ∀ x we shall have D( x ) < ϵ .

A.2 Decision Boundary

[Proposition 2] For two normal populations with known (denote as Σ ) and equal covariance matrices, and means µ 1 and µ 2 , the LR model f ( x ) = sigmoid ( ω ⊤ x + b ) has a theoretical decision boundary:

Proof. Usually, we estimate the parameters ω and b by maximizing the likelihood function

This is to estimate parameters in the 'most data-fitting manner' when the distribution is unknown. Now we know that two classes are normally distributed, we even know their mean and covariance matrix.

In Logistic Regression

So we have

And by Bayes' theorem, we can derive

Here, since we inherently know the data is normally distributed, we have

Then, comparing Equation (10) with Equation (9), we'll have the answer.

A.3 More Experiments

We conduct an ablation study on "how ST k complements other long-tailed learning algorithms." The backbone here is a Vision Transformer (ViT) pre-trained by MAE / CLIP [He et al., 2022, Radford et al., 2021]. "CS" represents cost-sensitive learning, and 'PEL' refers to the method provided by [Shi et al., 2023]. The batch size was set to 2048 and the models were trained in 30,000 steps.

The smoothing coefficient δ , 0.01, is a grid search determined value for all datasets in Section 5.1 (see page 7) and was adopted in all other experiments. We conducted an ablation study on the five real-world datasets with all other settings remaining unchanged, except for δ .

A.4 Benchmarks Detailed Information

In Table 9, we provide the statistical information for two benchmark sets discussed in Section 5, where c, n, d are the number of classes, samples and features, respectively. The left part provides the information of four regression datasets in Section 5.3. The 'Sinc' dataset is a synthetic dataset, sampled from the sinc function y = sin( x ) /x . The right part provides the information of seven binary classification datasets in Section 5.1.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and/or introduction clearly state the claims made.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations can be found in Section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The proof can be found in Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Every experiment can be easily reproduced, and the code will be available soon.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Main results can be reproduced by supplementary material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The paper has specified all the training and test details.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We provide the Std of the results, where the proposed method exhibits the lowest Std and surpasses other methods in terms of mean performance.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The computational resource requirements of experiments on large real-world datasets can be found in 5.2, while the remaining experiments can be produced on CPU.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Sure about that.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We ensure that the creators or original owners of assets used in the paper are properly credited, and the license and terms of use are explicitly mentioned and properly respected.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the Top-K problem by introducing a new loss function. Building on the Average Top-K Loss, the authors incorporate a smoothed ReLU function to create the $ST_k$ loss, which is fully differentiable. Through experiments on various datasets, they demonstrate the effectiveness of their approach.\n\nSTRENGTHS:\n1. The paper is clearly written and easy to follow.\n2. The authors conduct extensive experiments to evaluate the proposed $ST_k$ loss, using both synthetic and real-world datasets.\n\nWEAKNESSES:\n1. The core weakness lies in the lack of significant innovation. The main technical contribution is the replacement of the ReLU-like operation in an existing optimization method for the Top-K problem with a smoothed ReLU function, controlled by a hyperparameter for smoothness. Similar concepts already exist in various ReLU function variants, as seen in related works such as [1,2].\n2. The only large-scale experiment is a long-tailed classification task involving parameter-efficient fine-tuning of a CLIP model pretrained on ImageNet-21K. Given that this model already has substantial pre-trained knowledge, the results are less convincing. Evaluating the method by training a model from scratch on these tasks would better demonstrate its effectiveness in practical scenarios.\n3. The paper does not include an ablation study on the hyperparameter $\\delta$ to study its impact on the performance.\n\n[1]. Biswas, Koushik, et al. \"SMU: smooth activation function for deep networks using smoothing maximum technique.\" arXiv preprint arXiv:2111.04682 (2021).\n\n[2]. Hendrycks, Dan, and Kevin Gimpel. \"Gaussian error linear units (gelus).\" arXiv preprint arXiv:1606.08415 (2016).\n\nQUESTIONS:\n1. In lines 20-21, the authors mention that \"the cost of the ranking process can no longer be ignored.\" I am curious about how significant this ranking cost is in the context of the full forward pass of a deep learning model. Given that larger models require more computational resources for their intermediate layers, this ranking cost might be negligible in most cases. Additionally, there is no detailed comparison of time costs between the Average Top-K Loss $AT_k$ and the proposed $ST_k$ loss, apart from on the toy synthetic dataset.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a differentiable module for solving the top-k problem. Specifically, the paper proposes to approximate the hinge function with a new differentiable function.   \nExperiments on binary classification, long-tailed classification, and regression tasks with $ST_k$ loss show some improvements over baselines, like Average loss, and $AT_k$ loss.\n\nSTRENGTHS:\n(1) The paper is written clearly and easy to follow.                 \n(2) The proposed new differentiable function in Eq.(2) can approximate the hinge function.            \n(3) The proposed method is flexible and is expected to be used in many scenarioes.\n\nWEAKNESSES:\n(1) For the implementation of $AT_k$, is it implemented by a sort algorithm or just Eq. (5)?\n     From my point of view, the performance with $ST_k$ loss should be bounded by the performance with $AT_k$ loss.\n     However, Tables 1,4 show that $ST_k$ outperforms $AT_k$.\n     Why is $ST_k$ loss better than $AT_k$ loss?\n\n(2) The proposed $ST_k$ loss is flexible and can be directly deployed on the traditional training paradigm.\n     It would be better to validate the effectiveness of $ST_k$ on large-scale data, like ImageNet, and iNaturalist.\n    \n(3) The authors claim that the average loss is easily overfitted on imbalanced data. With $ST_k$ loss and CLIP models, it achieves the state-of-the-art on ImageNet-LT and Places-LT.\n     Is the $ST_k$ loss complementary to other state-of-the-art long-tailed learning methods, like GPaCo[2] and BCL[1]?    \n     [1] Balanced Contrastive Learning for Long-Tailed Visual Recognition. CVPR 2022.     \n     [2] Generalized Parametric Contrastive Learning. TPAMI 2023. \n\n(4) The models trained with $ST_k$ loss could be optimized alternatively or simultaneously with SGD for model parameters and $\\lambda$. The ablation study is encouraged to be included.\n\nQUESTIONS:\n(1) Explanation of why $ST_k$ outperforms $AT_k$.                  \n(2) The experiments on large-scale data, like ImageNet.    \n(3) Is the $ST_k$ loss complementary to current state-of-the-art long-tailed learning methods.    \n(4) Ablation for the optimization manner of model parameters and $\\lambda$.\n\nLIMITATIONS:\nLimitations are discussed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors proposed a differentiable layer to approximate the top-k loss in deep learning. The proposed layer is motivated from Eq. (1), replacing the ReLU with a smoothed ReLU (SReLU in Eq. (2)). The authors showed that the proposed layer is point-wise convergent to top-k loss. Numerical experiments validate the superiority of the proposed method.\n\nSTRENGTHS:\n* The paper is over-all well written, easy to follow. Experiments on real-world datasets are provided, in addition to simulated results.\n* A fast, accurate approximate of top-k loss is important in deep ranking problems.\n* Experiments show that the proposed method is over-all better than baseline methods.\n\nWEAKNESSES:\n* The biggest concern is that most of the real-world experiments are done on small scale datasets. It would be more convincing if the authors could demonstrate their method on large-scale ranking problems, especially, large deep learning models.\n\n* The proposed method lacks strong theoretical guarantees. Point-wise convergence is somehow weak. As easy to expect, the convergence rate and error bound of the proposed method should depend on the data distribution, which is not discussed in the paper. It would be nice if the authors could show that under what conditions, the proposed method will converge faster than $AT_k$ and / or with smaller approximation error.\n\n* It is important to check how well the proposed method approximates top-k loss in real-world problems. However, most real-world experiments report model accuracy, which is an indirect metric of top-k loss approximation. Please consider to measure the top-k loss approximation error directly.\n\nQUESTIONS:\n* In real-world problems, is it possible to design experiments to measure the top-k loss approximation error directly?\n* What is the theoretical advantage of $ST_k$ v.s. $AT_k$\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors introduce a novel differentiable module (ST_k) for efficiently solving top-k problems. Their method relies on optimizing a differentiable form of an equivalent optimization problem, by proposing an approximation of ReLU differentiable everywhere. This equivalent optimization problem contains a single parameter \\lambda which is supposed to match the k-th largest element in the optimal case, and in practice is optimized over the dataset. Authors  experiment on synthetic data, showing that their loss objective results in a model that more closely matches the true decision boundary in this setting. Next, authors experiment with real-world applications on binary classification and long-tailed classification datasets, showing consistent improvement over other optimization losses.\n\nSTRENGTHS:\n**Strengths**\n\n- Authors propose an efficient differentiable solution to the Top-K problem that integrates very easily into existing model architectures, and only has a single optimizable parameter.\n- Authors show consistent improvements both in synthetic and real-world tasks compared to sota methods.\n- The paper is well-written, authors provide ample motivation for their design choices.\n\nWEAKNESSES:\n**Weaknesses**\n\n- Although the improvements shown by the authors are consistent, they are also marginal - only improving slightly in for example CIFAR-100-LT classification and ImageNet-LT classification compared to average aggregation. \n- As authors mention, they only apply their module to a single Top-K Loss; ATK, making it unclear how ST_k could be applied in other scenarios and how well it generalizes to different settings.\n\nQUESTIONS:\n**Questions**\n- How do you tune the smoothing coefficient in practice? Is this a hyperparameter that needs to be sweeped over? That would somewhat deteriorate the efficiency of your module. I would like to see an ablation over the values of delta, i think it is important to see what/if any impact this choice has on model performance.\n- Did you experience any training instabilities with the two-step optimization scheme?\n\nLIMITATIONS:\nThe authors mention as limitation the fact that they only applied ST_k to the average top-k loss. I think it would be good if they more explicitly mention future research directions that they are interested in/ that would be of value (there is space in the manuscript).", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}