Adaptive Sparse Softmax: An Effective and Efficient Softmax Variant for Text Classification
Authors: Qi Lv
Lei Geng
Ziqiang Cao
Min Cao
Sujian Li
Wenjie Li
Guohong Fu
Venue: Submitted to ICLR 2023 (2023)

Abstract: Softmax with the cross entropy loss is the standard configuration for current neural text classification models. The gold score for a target class is supposed to be 1, but it is never reachable under the softmax schema. Such a problem makes the training process continue forever and leads to overfitting. Moreover, the “target-approach-1” training goal forces the model to continuously learn all samples, leading to a waste of time in handling some samples which have already been classified correctly with high confidence, while the test goal simply requires the target class of each sample to hold the maximum score. To solve the above weaknesses, we propose the \textbf{A}daptive \textbf{S}parse softmax (AS-Softmax) which designs a reasonable and test-matching transformation on top of softmax. For more purposeful learning, we discard the classes with far smaller scores compared with the actual class during training. Then the model could focus on learning to distinguish the target class from its strong opponents, which is also the great challenge in test. In addition, since the training losses of easy samples will gradually drop to 0 in AS-Softmax, we develop an adaptive gradient accumulation strategy based on the masked sample ratio to speed up training. We verify proposed AS-Softmax on a variety of multi-class, multi-label and token classification tasks with class sizes ranging from 5 to 5000+. The results show that AS-Softmax consistently outperforms softmax and its variants, and the loss of AS-Softmax is remarkably correlated with classification performance in validation. Furthermore, adaptive gradient accumulation strategy can bring about 1.2× training speedup comparing with the standard softmax while maintaining classification effectiveness.

Under review as a conference paper at ICLR 2023
FORMATTING INSTRUCTIONS FOR ICLR 2023 CONFERENCE SUBMISSIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Softmax with the cross entropy loss is the standard configuration for current neural text classification models. The gold score for target class is supposed to be 1, but it never reachable under the softmax schema. Such a problem makes the training process continue forever and leads overfitting. Moreover, the "target-approach- I" training goal forces the model to continuously learn all samples, leading to a waste of time in handling some samples which have already been classified cor- rectly with high confidence, while the test goal simply requires the target class of each sample to hold the maximum score. To solve the above weaknesses, we pro- pose the Adaptive Sparse softmax (AS-Softmax) which designs a reasonable and test-matching transformation on top of softmax. For more purposeful learning, we discard the classes with far smaller scores compared with the actual class during training. Then the model could focus on learning to distinguish the target class from its strong opponents, which is also the great challenge in test. In addition, since the training losses of easy samples will gradually drop to 0 in AS-Softmax, we develop an adaptive gradient accumulation strategy based on the masked sam- ple ratio to speed up training. We verify proposed AS-Softmax on a variety multi-class, multi-label and token classification tasks with class sizes ranging from 5 to 5000+. The results show that AS-Softmax consistently outperforms softmax and its variants, and the loss of AS-Sofumax is remarkably correlated with classi- fication performance in validation. Furthermore, adaptive gradient accumulation strategy can bring about 1.2x training speedup comparing with the standard soft- max while maintaining classification effectiveness.
INTRODUCTION
Softmax is widely used as the last-layer activation function of neural classification model. It nor- malizes the output of a network to a probability distribution over predicted output classes. Softmax is extremely appealing in both academies and industries it is simple to evaluate and differentiate. Many research efforts are devoted to refining softmax. On one hand, several works (Shim et al.) 2017 Liu et al.) 2017) have explored accelerating the training process by outputting subset of classes to reduce computational complexity. On the other hand, improving training effectiveness also an important direction. For example. Liu al. (2016) and [iang al. (2017) enlarged the margin of intra-class and inter-class in softmax, while Szegedy et al. (2016) used label smoothing to prevent the neural networks from excessively trusting gold labels.
Despite the probability form, each output score of softmax is within the range of (0. 1), unlike the sparse probability distribution real classification tasks. Consequently, the weights of all the classes have to be updated endlessly, which wastes a lot of time and leads to overfitting (Sun et al., 2021). In addition, the training goal of softmax with the cross entropy loss to make the target score approach to 1, while in test we expect the target score could be superior to scores of other classes. Thus there is an obvious gap between the two objectives. Take the following two prediction cases an example: It is 5-class classification task and Class is the gold label.
Case A: (0.4. 0.15, 0.15, 0.15, 0.15)
Case B: (0.49, 0.5, 0.004, 0.003, 0.003)

---

Under review as a conference paper at ICLR 2023
We can find that in Case A, the score of Class 1 is far larger than all the other classes, leading successful prediction. By contrast, there is a strong negative class in Case B and its classification result is wrong. Unfortunately, in the training period, since the target score in Case B is much higher than that in Case A, the corresponding cross entropy loss would make the model biased towards improving Case A, although Case B is hard sample in practice. Our experiments show that sometimes the correlation coefficient between the cross entropy loss of softmax and the classification accuracy is even close to 0 in validation (refer to Table 4
To address the above-mentioned deficiency, we propose the Adaptive Spare softmax (AS-Softmax) which involves a reasonable and test-matching transformation on top of softmax. Specifically, when computing the softmax activation function, we exclude the classes whose scores are smaller than the actual class to a given margin. In this way, the model focuses on learning to distinguish the target class from its strong opponents, matching the use in test. It is also noted that the loss of AS- Softmax can drop to 0 as long as the score of the target class much larger than all other classes. In other words, with the progress of training, more and more easy samples will be dropped from back propagation. Thus the learning process tends to find the useful hard training instances, which makes learning more effective. In practice, it is not always the case that increasing the training sample helps the final performance of the model. Yao et al.] (2022) and Wang et al. (2017) claimed that training the model with representative instances is more efficient. Easy samples can be discarded by the cascade structure at early stages (Yang et al.] 2019) for learning better classification models. Moreover, the increasing removed samples also guide us in developing a novel adaptive gradient accumulation strategy to make the training process faster and faster.
We test AS-Softmax on variety of text classification tasks, including multi-class classification et al. (2019), multi-label classification Chalkidis et al. (2021); Kowsari et al. (2017) and token classification (Sang & De Meulder (2003); Tseng et al. (2015). All the bench- marks are publicly available and the class sizes range from 5 to 5000+. Experiments show that AS-Softmax consistently improves the performance of text classification by 1% to 2% and allevi-
Socher et al. (2013):
ates overfitting to large extent. In addition, AS-Softmax with the adaptive gradient accumulation strategy can bring about 1.2x training speedup while maintaining the performance.
To conclude, AS-Softmax solves the endless training and train-test objective mismatching problems of the original softmax function. Its advantages can be summarized as follows:
With AS-Softmax, easy samples will be dropped from back propagation gradually, mod- erating overfitting and making training more effective.
The loss of AS-Softmax is highly correlated with the final classification performance.
Its training can be acceleraeed via the proposed adaptive gradient accumulation strategy.
AS-Softmax is fairly easy to implement. We have made it as an standard module in Py- torch
2 RELATED WORK
Softmax loss function is widely used as an output activation function for modeling categorical prob- ability distributions. There are lot of variants of softmax proposed in the literature (De Brebisson & Vincent 2015 Martins& & Astudillo 2016 Yang et al. [017 Kanai et al., 2018 Gimpel & Smith, 2010). We will briefly introduce them in terms of efficiency and effectiveness.
Training a model using the full softmax loss becomes prohibitively expen- sive in the settings where large number of classes are involved. One important kind of efficient training is to reduce its output dimension to reduce computational complexity. For example, hicrar- chical softmax (Morin & Bengio 2005) partitioned the classes into tree based on class similarities. Meanwhile, many works explored efficient subsets instead of all output classes. We define these methods as sparse softmax family uniformly. Jeaa al. (2014); |Rawat et al. (2019) and [Blanc & Rendle (2018) preserves a small number of negative classes in training by importance sampling and kernel based sampling, respectively. Martins & Astudillo (0066) suggested new softmax variant
Enhancing Efficiency
Coode will he released in the final version.
2

---

Under review as a conference paper at ICLR 2023
which could generate sparse posterior distributions. Some studies also tried to approximate timation to simplify the calculation of softmax. [evlin et al. (2014) and Vaswani et al. (2013) used seff-normalization mechanism and noise contrastive estimation separately simplifed the complex- ity of calculation. Spherical softmax (Vincent et al. 2014 De Brebisson & Vincent 2015) adopted quadratic function instead of the original the exponential function, making its parameter update independent of the output size while Shim et al. (2017) introduced the singular value decomposition method to restrict the probability distribution.
Boosting Effectiveness Softmax usually lacks the accurate discrimination of similar output classes. Thus, intuitive idea is to enlarge the inter-class margin and compress the intra-class distribution. Liang et al. (2017) designed a soft distant margin while Liu et al. (2016) incorporated angular margin to obtain
discriminative decision boundary. The features learned by softmax
loss have intrinsic angular distribution Liu et al. (2017). Follow-up studies (Deng et al. 2019 Choi
al. 2020 Wang al., 2018d Ranjan et al. 2017 Liu et al. 2017; [Wang et al. (018c) made more effort on the angular constraints and normalizate
Additionally, variants in the sparse softmax family like Chen et al. (202T) and [Sun et al. (202I) are also favor of improving the effectiveness of models as they reduce the risk of overfitting, Besides, to prevent the network from becoming over-confident in the current labels, SSegedy et al. (2016) introduced label smoothing while Huang et al. (2021) proposed Noise- Doise-Aware-weee -wtte-witt-----eeeeee weet--ithhheee to distinguish hard samples from noisy labels. More- over, Lee et al. (2018) overlay binary masking variables over class output probabilities by dropout to anput-adaptively learned via variational inference.
METHODS
3.1 BACKGROUND OF SOFTMAX
Suppose the final output of the neural classifier is o E R". Here denotes the number of classes. The prediction score of a class p; in the original softmax is computed as follows:
e
ps softmax(o,)
Σn e
Classifiers usually adopt cross entropy combined with softmax their loss function. For multi-class classification, there is only one actual target class t. Then the expression of the loss function and its backward propagation are as follows:
- -lo(pp))
(2)
log(
E) - -0+
j=1
0c
Pj 1,
if
VL
0aj
Pp.
otherwise.
The cross entropy loss is composed of exponential and logarithmic functions, which is convenient for computation of the loss function and its back propagation. However, mentioned Section I softmax is trapped in the endless training and train-test goal mismatching problems. In addition, the smaller the cross entropy loss, the stricter the constraints between the target and non-target classes. (un et al. (202T) proved that in order to make sure the loss L can be reduced to log 2, the output should satisfy the following inequality:
-min 2 log(n - 1),
(4)
where ot Oniin respectively means the target (maximal) logit and the minimal logit. When dealing with the high-dimensional classification problems, log n is a relatively massive but unnecessary margin, which increases the overfitting risk.
In order to mitigate disadvantages aforementioned, we propose an alternative variant of softmax named Adaptive Sparse softmax (AS-Softmax).
3

---

Under review as a conference paper at ICLR 2023
3.2 ADAPTIVE SPARSE SOFTMAX
The training goal of softmax is to make the score of the target class as large as possible. This practice does not match the test objective encourage the target probability greater than that of any other category. Motivated by Sparse-softmax (Sun et al. (02T), we propose new training objective which encourages the probability of target class Pt to exceed the probability of non-target class Pige by specific margin d:
Pt Pizt 2 6,
(5)
where 8 [0, 1] is hapereparameter.
To achieve this goal, we discard the classes which have already satisfied Eq. 5 in training. Specifi- cally, we adopt binary factor Zz:
0,
if Eq satisfied and +
2,
(6)
1, otherwise.
Then the modified probability of a class p4 is computed as follows:
2
p: = AS-Softmax((n)
(7)
Σド=ı2ee
It be that AS-Softmax is extremely easy implement. Given the output of softmax, AS- Softmax only needs a simple linear screening step while the back propagation process remains the same. With the import of 2,, we find that the losses of more and more training samples will reduce to zero, that is to say, these samples are masked, as shown in Figure Therefore, the classification model with AS- AS-Softmax tends to discard easy samples and focus on the rest hard cases, making learning more effective.
In the mathematical forms, AS-Softmax is close Sparse-Softmat which preserves the fixed top negative classes to speed up training. However, the loss of Sparse-Softman is still always larger than O and it does not contrast the representations of positive and negative classes.
We also examine a more direct training criterion expressed as:
(8) Nevertheless, take any value, namely upper bound of 8', which makes it hard to find the optimal value. More comparisons of the above two strategies are provided in Appendix
log(pt) - log(P( = Or Offt 2 8'.
A.I A.l
3.2. ALGORITIM DISCUSSION
We can prove that our training object enables the output o to obey the following condition:
0₂ Omin
log (nō + 1).
(9)
Proof. From our criterion in Eq.
5 we can get:
e Omin
Pt Pmin
Σj= eos
Ej= Bog
2 8.
(10)
This formula can be further converted into:
12
2t Omin
> 8 25 > noee.nir
(11)
1=1
Namely:
et 2 (nd + 1
(12)
With a log transformation, we finally conclude that:
Or Omin 2 log (nd + 1).
(13)
O
Recall the requirement of softmax in Eq. 4 the criterion of AS-Softmax is much easier to meet, as quite small in practice.
4

---

Under review as a conference paper at ICLR 2023
3.2.2 EXTENSION TO MULTI-LABEL CLASSIHICATION
Softmax typically used in multi-class classification and token classification tasks. However, [Su (2020) demonstrated that softmax could also be applied to multi-label classification tasks with the following loss function:
C log1 + Σ e + log(1 +
Ee 1E0
(14)
iEineg
where Spos stands for the set of target classes while Qneg denotes the set of non-target classes. The training objective of this method is to raise scores of target classes to exceed zero and decrease scores of non-target classes to be lower than zero. Thus in test, the classes with the scores larger than 0 can be regarded the prediction output. Compared with the sigmoid based method, this method can mitigate the sparse problem when the number of entire output classes is greatly larger than the number of target classes.
Likewise, we extend AS-Softman to multi-label classification, aiming to encourage probabilities of entire target classes to exceed probabilities of non-target classes by specific margin 6. To formulate, our additional goal is:
Pi 2 6, Pt-
Pr
(15)
Pi maz 2 6,
(16)
where Pt min represents the smallest probability probability of all non-target classes. To achieve this goal, define the following binary factors z and 2, for multi-label classification tasks:
target classes and p! means the largest
0, if Eq. [5 is satisfied,
2t = 0, if Eq. 16 is satisfied,
2,
= [1, otherwise,
(17)
1, otherwise.
Consequently, the final loss function combined with AS-Softmax can be obtained as follows:
L= l=o[[1 + Σ
2e")++
+ log(1 +
Σ
)t.
(18)
iERneg
TE
Note, if pm p. 2 is satisfied, the whole loss will reduce to 0, namely the sample masked.
3.3 ADAPTIVE GRADIENT ACCUMULATION STRATEGY
0.9
0.8
a 5:40
0.7
Progortooni0
0.6
0.5
0.4
0.3
0.2
0.1
0.0
2000
Training Steps
Figure 1: Variation of masked samples ratio on Clinc oos.
In theory, the model with the training objective of AS-Softmax will gradually remove easy samples during training. As shown in Figure experimental result shows taht the effective sample size in a batch of AS-Softmax is much smaller than that of softmax. Such few effective samples in batch would waste computing resources such as the GPU memory. In addition, the hyper-parameters the training optimizer such as the learning rate do not well match the training process in softmax. There- fore, we accumulate the training steps of AS-Softmax to make the effective sample size approach that of softmax. Specifically, we propose an adaptive gradient accumulation method according to the masked samples ratio:
Nall
stepsaccum = A Nail Nmasked
(19)
5

---

Under review as a conference paper at ICLR 2023
where À is a hyper-parameter controlling the accelerate magnitude and Nan//Nmasked denotes the number of all/masked samples. is necessary as stepsaceum is a prediction of the successive batches.
In addition, we attach the following restrictions to the accelerate strategy to guarantee its effective- ness: i). making sure stepsaccum is increasing monotonically; ii). ensuring the difference between two adjacent step8 does not exceed I; and iii). setting an upper limitation to steP For convenience, we call the acceleration algorithm of AS-Softmax as AS-Speed.
4 EXPERIMENTS
4.1 DATASETS AND EVALUATION METRICS
To evaluate our approach extensively, we conduct experiments on 6 different text classification datasets, including SST5 (Socher al., 2013), Clinc (Larson al. 2019). Conll2003 (Sang & De Meulder 2003). SIGHAN2015 (enn et al. 2013), Eurlex (Chalkidis et al. 2019) and WOS- 46985 (Kowsari et al. 2017). These datasets belong multi-class classification, token classification and multi-label classification tasks respectively. We separately adopt following evaluation metrics: accuracy, fl, macro/micro-fl
measure multi-class classification, token classification and multi- label classification tasks. The basic information of the datasets and evaluation metrics is listed in See Appendix A.2 for more details.
Table]] I
Dataset
Task
Train/DewwTest
# classes
Model
Metries
SST5 Clinc_oos
multi-class classification
8544/1101/2210
5
BERT BERT
accuracy accuracy
15250/3100/5000
151
Con112003
14042/3251/3454
9
BERT
fl
token classification
SIGHAN2015
277786/11000 100
5201
Roberta
Eurlex
55000/5000/5000
127
BERT BERT
macro/micro-ff macro/micro-fl
multi-label classification
WOS-46985
30070/7518/9397
141
Table 1: Statisties of experimental datasets.
4.2 BASELINES
We implement baselines including Softmax and six variants. We first compare our method with the cepperaturrebased softmax (T-Softmax) which has hyper-parameter. Then, for sparse based algorithms, we choose Sparse-Softmax (Sun et al. 202I), Sparsemax (Martins & Astudillo! 2016) and Entmax (Peters et al. 2019) our comparison methods. From the perspective of the reachable training objective, we introduce Label-Smoothing (Szegedy et al. (016) method for comparison. In addition, 8 plays the role of margin in AS-Softmaa Thus we compare AS-Softmax with the additive margin softmax (AM-Softmax (Wang et al., 2018c)). Most shared hyper-parameten refers to example while the specific hyper-parameters of each model are tuned on the validation set. Details of these models and their hyper-paramete, settings are shown in Appendix A.3
4.3 SETTINGS
Most experiments are conducted based on huggingface's 's pytorch implementation of transform- ers. The base pretrained models (Devlin et al. 2018 Liu et al. 2019) are initialized by BERT-base- cased and Roberta-wwm-exx wwm-ext respectively. We utilize Adam (Loshchilov & Hutter 2017) as the
optimizer. Due to the little change of AS-Softmax to the original softmax code, it can be reproduced easily. Our experiments are mainly carried out on RTX A5000 with a memory size of 24G except that the experiment related to SIGHAN2015 is conducted on V100 with 32G memory. For more specific parameter settings of the tasks we used, please refer to the Appendix A.4

---

Under review as a conference paper at ICLR 2023
SST5
Clinc_oos accuracy
Conl12003
SIGHAN2015 f1
Eurlex
WOS-46985
accuracy
f1
macro-f1
micro-f1
macro-f1
micro-f1
Softmax
$1.90
88.60 60
90.56
70.80
46.87
68.73
80.40
86.00
T-Softmax
53.12
88.65
90.63
70.80
46.87
68.73
80.76
86.36
Sparse- Softmax
52.99
89.02
90.93
71.40
47.77
68,99
80.94
86.65
Sparsemax
51.49
88.
90.57
71.49
Entmax
51.90
88.33
90.62
72.81
Smoothino
52.85
88.64
90.64
68.10
AM-Softmax
52.17
89.05
90.66
68.57
AS-Softmax
53.12
89.07
91.03
72.81
48.30
68.99
80.94
86.39
AS-Speed
52.53
88.78
90.58
71.40
48.14
68.70
81.07
86.71
Table 2: Comparison of experimental results on various text datasets with different baselines ("-""
means available since their papers codes do not provide the corresponding method for multi- label classification tasks).
SST5
Clinc_oos
Con112003
SIGHAN2015
Eurlex
WOS-46985
Softmax
288
533
625
10905
5944
5644
Sparse-Softmax
-7%
+2%
1
-2%
-1%
-1%
Sparsemax
-2%
0
0
+8%
Entmax
0
0
-2%
+10%
Label-Smooe Smoothing
+1%
-3%
.4%
AM-Softmax
0
+1%
0
-3%
AS-Softmax
-1%
-1%
-1%
-2%
-1%
-2%
AS-Speed -22%
-15%
-21 %
-7 %
-13%
-5% %
Table 3: Training time of different methods (in seconds). The training time of T-Softmax is not listed because it is the same as that Softmax in theory.
4.4
MAIN RESULTS
The performance and training time on experimental tasks are presented separately in Table 2 and Table Overall, the proposed AS- AS-Softmax and AS AS-Speed have achieved improvements terms of classification performance and efficiency in almost all the tasks.
From Table 2 we can observe that AS-Softmax consistently outperforms other methods. For multi- class classification tasks and token classification tasks, we find that AS-Softmax could bring more performance enhancement the case of difficult tasks than simple tasks. worth noting that the fl score of AS-Softmax SIGHAN2015 is about 2 points higher than that of softmax. In terms of multi-label classification tasks, there a slight difference between the training objective of AS- Softmax and the test goal. In testing, will choose the classes with scores greater than 0 as the prediction results. However, in training period, AS-Softmax only encourages the probabilities of non-target classes be lower than the minimum probabilities of target classes by specific margin Despite the impact of this gap, the results of AS-Softmax also surpass softmax results on such datasets. More details will be analyzed following sections.
Table shows that all compared algorithms except AS-Speed perform fairly in terms of the training time. Notably, when it comes high-dimensional output classes (i.e. SIGHAN2015), it may cost more time for those methods which requires O (nlogn) computation complexity, such as Entmax and Sparsemax. Compared with softmax, SpeSpeed could increase the training speed by about 1.2x on the premise of maintaining the classification accuracy. Too some extent, the acceleration effect depends on the number classification categories. AS-Speed has achieved greater speed improvement SST5 and Conll2003 which have more than 10 output classes. Mcanwhile, AS- Softmax also has more than 1.1 acceleration performance over softmax although there are 150+ output classes such Cline Loo. For SIGHAN2015 and WOS-46685. AS AS-Speed has significant effect. One important reason is that there are too many output classes and the proportion masked samples is unstable, leading the maximum accumulated steps only be 2.
In additional, we conduct experiment on some classification tasks of GLUE benchmark Wang et al. 2018a). The result are presented in Appendix A.6

---

Under review as a conference paper at ICLR 2023
4.5 RESULT ANALYSIS
Impact of d The 8 value can significantly affect the accuracy of the classifier. If ó is too large, AS-Softmax would not mask such easy samples, making the model overlear a lot of useless information. In contrast, AS-Softmax would cast away some potential useful knowledge when is rather small, which could result insufficient learning. our experiments, we attempt different values of 6 ranging from 0.05 to 0.5 and choose the final value which achieves the highest classifi- cation performance on the development set. According to such experiments, we suggest setting to moderate value 0.3. Then we investigate the influence of ó in the case of relative easy and hard tasks. As Figure 2 shown, the predictions on SSTS represents a difficult situation while the result on Conl12003 denotes simple case. It can be seen that the model performs best SST5 when the value is 0.2 (relative smaller). By contrast, a slightly larger ó more helpful to its classification effect Con112003. In conclusion,
believe that it could be better set relative larger value for simple tasks and set 5 to smaller value for difficult tasks.
0.9150
SSTS
Con112003
0.9125
0.535
0.9100
Accuracy
0.530
0.9075
0.525
0.9050
0.520
0.9025
0.515
0.9000
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.10
0.15
0.20
0.25
0.30
0.35
0.40
Figure 2: Accuracy/FI metrics on SST5 and Conl12003 with different
SST5
Clinc o0s
Conl12003
SIGHAN2015
Eurlex
WOS-46985
Softmax
0.038
-0.943
-0.975
-0.947
-0.337
-0.902
Sparse-Softmax
0.088
-0.914
-0.981
-0.907
-0.114
-0.901
Sparsemax
0.050
-0.976
-0.968
-0.681
Entmax
0.080
-0.986
-0.983
-0.617
Label- Sabbo-Snmothing
-0.087
-0.952
-0.998
-0.989
AM-Softmax
0.057
-0.915
-0.974
-0.982
AS AS-Softmax
-0.952
-0.949
-0.996
-0.986
-0.909
-0.904
Table Results Pearson correlation coefficient between loss and accuracy.
Correlation between Loss and Classification Performance To verify that our training objective can alleviate the train-test goal mismatching problem, we introduce the Pearson correlation coeffi- cient which measures the relationship between the loss and the classification ability in validation. The correlation coefficient is close to or ( indicating the loss is linear or has no significant corre- lation with the classification performance. the correlation coefficient is close I, demonstrates that the learning direction of the model is obviously opposite. In theory, the final classification per- formance inversely correlated with the loss. Table illustrates that the correlation between loss of all methods and their classification performance is linear on datasets with high identical accuracy such as Conll2003 and SIGHAN2015. However, when comes to hard datasets (i.e. SST5 and Eurlex), the correlation coefficients of compared baselines are not good enough and some corre- lations could be even the opposite. By comparison, the loss of AS-Softmax and its classification performance is highly relevant across whole experimental datasets, which verified the design of AS-Softmax training objective.
To intensively explore the correlation between the loss and classification performance, we record the loss and accuracy on development set of SST5. The left/right subfigure of FreureBrse its classification performance and normalized loss, respectively. It can be seen that other models tend to overfit as their accuracy is still fluctuating while the loss has turned to increase, after 1000+ training steps. By contrast, the loss of AS AS-Sofumax does not increase significantly the end of training
8

---

Under review as a conference paper at ICLR 2023
process, which also reveals that it could moderate the overfitting problem. To verify the discarded samples are easy samples we expected, we do more experiment. See details in Appendix A.5
Scftmax
AS Softmax
0.8
Entrnax
Sreching
0.45
Softmer
Accuracy
0.6
Scarse Sottaax
0.40
Loss
Starsemaa
Softmax A5-Sehmax
0.4
0.35
Entmax
Label Smeothing
0.2
0.30
500
1000
1500
2000 2500
3000 3500
500
1000
1500
2000
2500
3000
3500
Steps
Steps
Figure 3: Comparison of development performance on SST5.
Probability Margin of Positive and Negative Classes The AS-Softmax training goal caleu- lated according to target probability Pt and non-target probability Pişt- To check this objective, we design metric Pmargin = Pt Pney where Pneg indicates the largest probability in non-target classes. Pmargin is greater less than denoting that the classification result is correct wrong. Then we draw the distribution diagram of Pmargin on SST5 in Figure 4 where 5 in AS-Softmax set to 0.3. Generally speaking, Softmax result is widely distributed while the distribution of AS- Softmax result is very concentrated. On one hand, when Pmargin is less than O, AS-Softmax pushes Pmargin close 0. indicates that even if the true label is not identified correctly, its top candi- dates in the prediction result still have some reference value. By comparison, there are many strong negative classes in the softmax result, which makes the result completely unavailable. On the other hand, when Pmargin. is greater than 0, the overall distribution of Pmargin is squeezed to be lower than or close 8. suggests that AS-SSoftmx could prevent the model from being optimized to- wards overlearning easy samples and the model could pay more attention to those hard samples. To conclude, the probability margin distribution in accordance with the proposed training objective which encourages the target class exceed other non-target classes by specific margin 8.
15
142
Softmax
11.7
AS Softmax
Proportion(%
10
10.2
9.8
822
8.3
6.2
6.2
5.0
6.4
6.9
4.8
8
00.-~0.88
-08-
0.5-0.5
0.5
0.4 04--0.
0.2
0.2--0.1
0.1
0.0.0.1
0.1-0.2
0.2.0.3
0.3.0.4
0.4-0.5
0.5-0.6
0.6.0.7
0.7-0.8
0.8-0.9
0.7
0
0
0.3
Margin
Figure 4: Distribution diagram Pmargin on the SST5 test set with 0.3. We add a dotted line
stand for Pmorgin 0, which splits the wrongly (left) and correctly (right) predicted cases.
5 CONCLUSION
In this paper, we propose simple yet effective softmax variant, namely adaptive sparse softmax L.S-Softmax), which discards easy training samples at the aim of a more reasonable and test- matching learning objective. We further develop an adaptive gradient accumulation strategy based on the masked sample ratio to accelerate the training process AS-Softmax. Experimental results on text classification datasets show that AS-Softmax consistently surpasses the original softmax and its variants. We believe our work can be extended in many aspects. On the one hand, instead of the current fixed value of 6, plan investigate an improvement strategy which decides ó adap- tively. On the other hand, we are curious about the performance AS- AS-Sofmax on the generation tasks and other input modals such images audio.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Guy Blanc and Steffen Rendle. Adaptive sampled softmax with kernel based sampling. In Interna- tional Conference on Machine Learning, pp. 590-599. PMLR, 2018.
Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, and lon Androutsopoulos. Large-scale multi-label text classification on eu legislation. arXiv preprint arXiv: arXiv:/906.000 /90..099669 2019.
Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Mar- tin Katz, and Nikolaos Aletras. Lexglue: A benchmark dataset for legal language understanding in english. arXiv preprint arXiv:2110.00976. 2021.
Phil Chen, Mikhal Itkina, Ransalu Senanayake, and Mykel J Kochenderfer. Evidential softmax for sparse multimodal distributions in deep generative models. Advances in Neural Information Processing Systems, 34:1 15111116116666 2021.
Xingyi Cheng, Weidi Xu, Kunlong Chen, Shaohua Jiang, Feng Wang, Taifeng Wang, Wei Chu, and Yuan Qi. Spellgen: Incorporating phonological and visual similarities into language models for chinese spelling check. arXiv preprint arXiv:2004. 14166, 2020.
Hongjun Choi, Anirudh Som, and Pavan Turaga. Ame-loss: Angular margin contrastive loss for improved explainability in image classification. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition workshops, pp. 838-839, 2020.
Alexandre De Brebisson and Pascal Vincent. An exploration of softmax alternatives belonging to the spherical loss family. arXiv preprint arXiv: 1.05042, 2015.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos 7afeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4690-4699, 2019.
Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas Lamar, Richard Schwartz, and John Makhoul. Fast and robust neural network joint models for statistical machine translation. In proceedings of the 52nd annual meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pp. 1370-1380, 2014.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: /8 10.04005 2018.
Kevin Gimpel and Noah A Smith. coftmax-margin crfs: Training log-linear models with cost func- tions. In Human Language Technologies: The 2010 Anmual Conference of the North American Chapter of the Association for Computational Linguistics, pp. 733-736, 2010.
Xiusheng Huang, Yubo Chen, Shun Wu, Jun Zhao, Yuantao Xie, and Weijian Sun. Named entity recognition via noise aware training mechanism with data filter. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 4791-4803, 2021.
Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. On using very large target vocabulary for neural machine translation. arXiv preprint arXivv:14222
2014.
Sckitoshi Kanai, Yasuhiro Fujiwara, Yuki Yamanaka, and Shuichi Adachi. Sigsoftmax: reanalysis of the softmax bottleneck. neural information processing systems, 2018.
Kamran Kowsari, Donald E Brown, Mojtaba Heidarysafa, Kiana Jafari Meimandi, Matthew Gerber, and Laura Barnes. Hdltex: Hierarchical deep learning for text classification. In Machine Learning and Applications (ICMLA), 2017 16th IEEE International Conference on. IEEE, 2017.
Stefan Larson, Anish Mahendran, Joseph J Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K Kummerfeld, Kevin Leach, Michael A Laurenzano, Lingjia Tang, al. An evaluation dataset for intent classification and out-of-scope prediction. arXiv preprint arXiv: arXiv:1909 1909.02077 2019.
Hae Beom Lee, Juho Lee, Sachoon Kim, Eunho Yang, and Sung Ju Hwang. Dropmax: Adaptive variationial softmax. In NewrIPS, 2018.
10

---

Under review as a conference paper at ICLR 2023
Chong Li, Cenyuan Zhang. Xiaoqing Zheng, and Xuanjing Huang. Exploration and exploitation: Two ways to improve chinese spelling correction models, arXiv preprint arXiv: 14813, 2021.
Xuezhi Liang, Xiaobo Wang, Zhen Lei, Shengeai Liao, and Stan Z Li. maft-margin softmax for deep classification. In International Conference on Neural Information Processing, pp. 413-421. Springer, 2017.
Weiyang Liu, Yandong Wen, Zhiding Yu, and Meng Yang. Large-margin softmax loss for convolu- tional neural networks. international conference on machine learning, 2016.
Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 212-220, 2017.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. 1692, 2019.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 17X.v:1711.05101, 2017.
André F. 'T. Martins and Ramón Fernandez Astudillo. From softmax to spansemax: A sparse model of attention and multi-label classification. arXiv: Computation and Language, 2016.
Frederic Morin and Yoshua Bengio. Hierarchical probabilistic neural network language model. international conference on artificial intelligence and statistics, 2005.
Ben Peters, Vlad Niculae, and André FT Martins. Sparse sequence-to-ssqqeence models. arXiv preprint arXiv:1905.057000 2019.
Rajeev Ranjan, Carlos D Castillo, and Rama Chellappa. L2-constrained softmax loss for discrimi- native face verification. arXiv preprint arXiv: 09507, 2017.
Ankit Singh Rawat, Jiccao Chen, Felix X. Yu, Ananda Theertha Suresh, and Sanjiv Kumar. Sampled softmax with random fourier features. neural information processing systems, 2019.
Erik Sang and Fien De Meulder. Introduction to the con1l-2003 shared task: Language- independent named entity recognition. arXiv preprint cs/0306050, 2003.
Kyuhong Shim, Minjae Lee, Iksoo Choi, Yoonho Boo, and Wonyong Sung. Svd-softmax: Fast soft- max approximation on large vocabulary neural networks. neural information processing systems, 2017.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language pro- cessing, pp. 1631-1642, 2013.
Jianlin Su. Extending the softmax combining with cross entropy to multi label classification prob- lem. Apr 2020.
Shaoshi Sun, Zhenyuan Zhang, BoCheng Huang, Pengbin Lei, Jianlin Su, Shengfeng Pan, and Jiarun Cao. Sparse-softmax: A simpler and faster alternative softmax transformation. arXiv preprint 1rXiv:2112.124333 2021.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2818-2826, 2016.
Yuen-Hsien Tseng, Lung- Hao Lee, Li-Ping Chang, and Hsin-Hsi Chen. Introduction to sighan 2015 bake-off for chinese spelling check. In Proceedings of the Eighth SIGHAN Workshop on Chinese Language Processing, pp. 32-37, 2015.
11

---

Under review as a conference paper at ICLR 2023
Ashish Vaswani, Yinggong Zhao, Victoria Fossum, and David Chiang. Decoding with large-scale neural language models improves translation. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1387-1392 2013.
Pascal Vincent, Alexandre de Brébisson, and Xavier Bouthillier. Efficient exact gradient update for training deep networks with very large sparse targets. arXiv: Neural and Evolutionary Computing, 2014.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: 00440. 2018a.
Dingmin Wang. Yan Song, Jing Li, Jialong Han, and Haisong Zhang. A hybrid approach to auto- matic corpus generation for chinese spelling check. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 2517-2521211 2018b.
Feng Wang, Jian Cheng, Weiyang Liu, and Haijun Liu. Additive margin softmax for face verifica- tion. IEEE Signal Processing Letters, 25(7):926-955) &c.
Hao Wang. Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5265-5274, 2018d.
Min Wang, Fan Min, Zhi-Heng Zhang, and Yan-Xue Wu. Active learning through density clustering. Expert systems with applications, 85:305-317, 2017.
Zihan Wang, Peiyi Wang, Lianzhe Huang, Xin Sun, and Houfeng Wang. Incorporating hierar- chy into text encoder: a contrastive learning approach for hierarchical text classification. arXiv preprint arXiv:2203.03885, 2022.
Shih-Hung Wu, Chao-Lin Liu, and Lung-Hao Lee. Chinese spelling check evaluation sighan bake- off 2013. In Proceedings of the Seventh SIGHIAN Workshop on Chinese Language Processing, pp. 35-42, 2013.
Dongming Yang, YueXian Zou, Jian Zhang, and Ge Li. C-rpns: Promoting object detection in real world via cascade structure of region proposal networks. Neurocomputiing 367:20 30, 2019.
Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W. Cohen. Breaking the softmax bottleneck: A high-rank mnn language model. arXiv: Computation and Language, 2017.
Xingcheng Yao, Yanan Zheng, Xiaocong Yang, and Zhilin Yang. Nlp from scratch without large- scale pretraining: A simple and efficient framework. In International Conference Machine Learning, pp. 25438-25451. PMIR, 2022.
Liang-Chih Yu, Lao Lee, Yuen-Hsien Tseng, and Hsin-Hsi Chen. Overview of sighan 2014 bake-off for chinese spelling check. In Proceedings of The Third CIPS-SIGHAN Join! Conference on Chinese Language Processing, pp. 126-132, 2014.
A APPENDIX
A.1 AS-VARIANT
AS-Softmax is to mask the weak negative class features after obtaining the probabilities of classes. One feature of AS-Softaax is 8 E [0, 1]. is convenient to find the optimal ó value of AS-Softmax. We also examine simple training criterion. That is, apply 8' to the final output of the neural classifier. Similar to the principle of AS-Softmax, is a margin used to open the gap between the scores of target categories and that of other categories. Nevertheless, no upper bound of 8', which makes it hard to find the optimal value. We call the above method AS- We carry out the experiment of AS-variant on SST5.
take any value, namely
In the Figure 5 the gray dotted line represents the prediction results of original softmax while the experimental results of AofSofaaax indicated by the red dotted line. The blue one represents
12

---

Under review as a conference paper at ICLR 2023
0.535
0.530
Accuracy
0.525
0.520
0.515
Softmax
Softmax
0.510
5
Figure 5: Accuracy of AS-viniant with different 8' on SST5 test set.
that the results of AS-variant change with 8'. We take the value of 8' from 0 to 7. As can be seen from this figure, when is greater than 6, AS-variant equivalent to softmax. can seen that the prediction results fluctuate up and down around the baseline. Therefore, where to obtain the optimal 5' under this algorithm as easily as AS-Softmax. Moreover, the results not good AS-Softmax.
not determine
A.2 DATASETS AND METRICS
In this section, we describe the experimental datasets and metrics in details.
SST5. This dataset (Socher al. 2013) is a sentence level classification dataset for sentiment analysis. It contains 5 output classes (from very negative to very positive). It is made up of 8544 train, 1101 validation, 2200 test samples. We adopts pretrained language model BERT initialized with bast-base-cased parameters. The evaluation metric for this dataset is accuracy.
Clinc_oos, This dataset (Larson et al., 2019) is for intent classification, comprising of 151 classes over 10 domains. Its train, validation, test dataset contains 15250, 3100, 5500 instances. We use BERT as our backbone. Accuracy is also used as evaluation metrics.
Conl12003. This dataset (Sang & De Meulder 2003) is for named entity recognition (NER) task. consists of four types of name entities: persons, locations, organizations and others. There are nine categories in its output. The sizes of train, validation, test dataset are 14042, 3251, 3454, respectively. We use fl score to measure this dataset.
SIGHAN2015. This dataset (Tseng al. 2013) is designed for the Chinese Spelling Check (CSC) task. Following previous works (Li et al. 2021 [Cheng et al.) 2020), we formula CSC as token classification task whose candidates are tokens in vocab. The training dataset with the size of 277786 is comprising of the automatic generated dataset (Wang et al. 2018b) and all SIGHAN training sets 2015). The test set the SIGHAN2015 test [Tseng et al. (015) with a size of I100. Considering the tokens we focus are Chinese characters, we cut original vocab according
(Wu et al. 2013 Yu et al. 2014: Tseng et
the training data. Finally there are 5201 characters left. We follow previous works (Cheng et al., 2020 Li et al.) (02T) and choose the metric of sentence level correction-fl which could more strietly represent the classifier competence.
This dataset (Chalkidis et al., 2021 is for legal lulti-label classification task. All EU laws are annotated by the EU Publications Office with multiple concepts from the eurovoc dictionary, multilingual dictionary maintained by the Publications Office. Given document, the task is to predict its Euro Voc labels (concepts). It contains 55000, 5000, 5000 instances for training, validation and test. There 127 labels in this dataset, and each sample has multiple labels. We use BERT as our base model and macro-fl f1mm//ro-ffl to evaluate the classifier performance.
Eurlex.
WOS-46985. We choose WOS-46985 (Kowsari al. 2017) the dataset of multi- labell classifica- tion task. Obviously, the total size this dataset is 46985. Following Wang et al. (2022), we split the dataset into training set, validation and test according to 6.4:1.6:2 split the dataset into
13

---

Under review as a conference paper at ICLR 2023
training set and test set. The dataset consists of three types of labels: target label, parent label and child label. We combined the parent label with the child label as the output classes following Wang et al. (2022). Finally, the number of output classes in this dataset is 141. Notably, there two corresponding labels for each sample and we only select the largest two classes as the result in test. We also BERT our backhone and macro- faero-fl/mmiroof1 the evaluation metrics.
A.3 BASELINES
The comparison algorithms used in our experiment are described in details here.
T-Softmax. T-Softmax simple variant of introducing the temperature parameter 7 into Soft- max. Divide the model output by 7. The larger is, the smoother the result is. The smaller the corresponding 7 is, the sharper the probability distribution is. The prediction score class p; in T-Sofimax is computed as follows:
P: = T-Softmax(o:)
(20)
Ej= e
Label-Smoothing. The authors of Label-Smoothing (Szegedy et al. 2016) argue that these non- target classes need be given probability greater than zero so that the model can fully learn all classes' features. Considering that the dataset is not completely correct, the original softmax loss function is even less applicable. Label-Smoothing has a parameter c. In the softmax loss function, the calculated weight of non-target classes is O. However, in the label smoothing algorithm, assuming that dataset has labels, the weight of non-target class is - The weight of the target class e. is a prrra-pprrrettr of label smoothing.
Sparse-Sofftax. Spfrre-Softmax (Sun et al. 2021) means that softmax results are sparse. Sparse effective because avoids overlearning of the model. This algorithm only preserves the features of the top & classes, and the remaining features are directly assigned to 0 for mask. Obviously, k is a Hyper-parametee
AM-Softmax Inspired by algorithms such as large-margin, additive margin softmax (AM-Softmax) WWang et al. 2018c) aims to minimize the intra-class variation. And the loss function and back propagation process of large-margin softmax are too complicated due to the introduction of angle. AM-Softmax not only simplifies the calculation process, but also speeds up the training speed. Its
loss function is as follows:
L --
- log cs((oc-m) + 2j=1.jz1 C.
(21)
It has two parameters, and margin m. m is used to increase the margin between categories. has the effect of acceleration. When s or m is too large, NAN may occur in the value of the loss function. Therefore, and m vary greatly with different datasets. This is process that needs to be adjusted slowly.
Sparsemax (Martins & Astudillo 2016) returns sparse posterior distributions by signing zero probability to its output variables with small scores.
Sparsemax.
Entmax. Entmax algorithm (Peters al., 2019) tends to produce sparse probability distributions, yielding a function family continuously interpolating between softmax and Sparsemax. There are two algorithms including a method based on sorting and disection-based method. We do compar- ative experiment with the 1.5-entmax proposed by author (Peters et al. 2019). It is based on sorting when a is equal to 1.5 and more accurate and less costly.
A.4 EXPERIMENTAL DETAILS
In our experiment, additionally adopt special warm-up strategy Clinc_oos and Conll2003. At the beginning of training period, AS AS- Softmax may discard some potential useful samples due to the poor classifier competence. Therefore, For AS-Speed algorithm, there are two more aaper--arammters: which controls the accelerate magnitude while denotes maximum accumulated steps. In addition,
keep å equal to 1 in the first percent of training steps.
and s. a parameter
14

---

Under review as a conference paper at ICLR 2023
implement Entmax, Sparsemax and AM-Softmax according to the source code general and specific hyper-parameten respectively Table 5 and Table 6
We record the
SST5
Clinc_oos
Conl12003
SIGHAN2015
Eurlex
WOS-46985
epoch batch size
10
20
15
20
10
16
128
12
learning rate
2e-5
2e-5
2e-5
5e-5
3e-5
3e-5
Table 5: Information of universal hyper-parameters.
SST5
Clinc_oos
Conl12003
SIGHAN2015
Eurlex
WOS-46985
T-Softmax (T)
5
0.5
0.5
0.5
1
0.5
Sparse-Softmar (k)
4
10
5
20
50
100
Entmax (a)
1.5
1.5
1.5
1.5
Label-Smoothing
0.30
0.20
0.10
0.10
AM-Softmax m)
10/0.30
8/0.35
5/0.30
1/0.10
AS AS-Softmax (6/r)
0.30/0.00
0.30/0.15
0.35/0.25
0.30/0.00 0.5/2
0.05/0.00
0.35/0.00
AS-Speed (N/)
1.5/4
0.5/5
0.5/S
0.5/2
1.0/2
Table 6: Parameter setting algorithm. after AM-Softmax means scale. s after AS-Speed means max accumulation steps.
A.5 SAMPLE ANALYSIS
To verify the "easy" samples are not selected by chances, we construct the datasets with hard samples left at the end of the AS-Softmax training period SST5 (approximately 1 11% of all) and the same number of random selected samples. Then, we train the classifier with softmax and the result is shown in the T The performance illustrates that these hard samples more difficult than the random selected samples for the model to learn.
SST5
Random samples
45.52
Hard samples
38.64
Table 7: Result of standard softmax method on random sample dataset and hard sample dataset.
A.6 RESULTS ON GLUE DATASET
We further evaluate our proposed AS-Softmax on the classification tasks of GLUE benchmark [Wang el al.) 2018a), including MNLI, MRPC, QNLI, QQP, RTE, SST-2 and CoLA, The shared hyper-parameters are following Liu (019)) while the margin AS- AS-Softmax adjusted in [0.05, 0.35] for searching. Table 8 presents the experimental result. It can be seen that the result of AS-Softmax surpasses that of Softmax, which verifies its effectiveness.
MNLI:3
MRPC:2
QNLI:3
QQP:2
RTE:3
SST-2:2
CoLA:2
Avg
Softmax 84.18
89.41 90.72
90.49 91.12
91.00
68. 68.23
92.54
62.44 64.55
82.61 83.44
AS-Softmax
84.19
90.95
69.31
93.23
Table 8: Result of GLUE benchmark (better results are in bold), with the number of output class attached behind the task name.
15


Reviews:
[{"review_id": "TIoS3PeS2g", "reviewer": "Reviewer_UMbA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a weighting variant of the softmax function. The weights are binary and are a function of the difference between class probabilities subject to a threshold.\n\nThe authors evaluate their model in six datasets and show that it achieves a small improvement over other variants. They also show that their model is faster in training and the speed-up is noticeable in some cases.", "summary_of_the_review": "The idea is intuitive and easy to implement, but the arguments in the paper are weak, the related work section is insufficient, the authors have missed to consider curriculum learning, the improvements are not noticeable, and finally the presentation is horrible.", "strength_and_weaknesses": "Reasons to accept:\n\nThe idea is intuitive and is easy to implement.\n\nReasons to reject:\n\n- I found the arguments discussed in the paper weak and incomplete.\n  - Example 1: Page 1, second paragraph, the first three lines, the authors state: “the weights of all the classes have to be updated endlessly, which wastes a lot of time and leads to overfitting”.\n\n    In my opinion, this is an incorrect depiction of the issue. The classes in neural nets share their weights. The authors show that their model speeds up the training. The cause of this acceleration is not omitting the classes, but it is dropping the easy examples.\n  - Example 2: Page 1, second paragraph, Lines 4-5, the authors state: “the training goal of softmax with the cross entropy loss is to make the target score approach to 1, while in test we expect the target score could be superior to scores of other classes.”\n\n    This is incorrect. The accurate output of softmax function has many applications in Bayesian learning, in active learning, in model explainability, etc.\n\n- The authors have done a poor job in contrasting their work against other similar studies. In the related work section they simply list the references, without explicitly discussing their distinctions. They have done the same in Section 4.2. Given this, I would say their proposed model is a small extension of the “Sparse-Softmax”, and is not worth publishing as a full paper at a top conference.\n\n- I believe the closest area of study to the proposed model is curriculum learning, which the authors have not discussed it (or aren’t aware of it) at all. No discussion, no related work, and no baselines on the subject.\n\n- You would need a magnifier to see the improvements, they are not noticeable to publish.\n\n- The writing is very poor, I won’t even bother giving examples, starting from the title (specifically the title in the pdf file) up to the end.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SnoSfroB7TB", "reviewer": "Reviewer_bnPp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides an alternative to the softmax + cross entropy loss widely used in text classification tasks. The proposed method, named adaptive sparse softmax (AS-Softmax), is inspired by the sparse-softmax (Sun et al., 2021) and changes the training objective by including a binary term that, when multiplied by a model’s output, excludes from the loss computation the classes that already satisfy a given margin (using an hyperparameter $\\lambda$). They further propose an adaptive gradient accumulation strategy to make the training procedure faster. Finally, they perform experiments on multi-class, multi-label, and token classification tasks, showing slightly better results over the baselines. ", "summary_of_the_review": "Although the work is well motivated and the problem they try to address is relevant, I don’t think the paper is ready for publication. The main reasons are explained in the Strengths/Weaknesses section.", "strength_and_weaknesses": "Strengths:\n\n* The motivation is clear.\n* AS-softmax seems to be an interesting  and easy-to-implement way to address some of the problems with softmax + cross entropy models.\n\nWeaknesses:\n\n* The related work section is too short and needs to be reviewed. Although you are already citing several works that attempt to solve some of the problems you are trying to tackle, the discussion is very superficial, making it hard to understand the difference between each of the baselines you consider (e.g., sparse-softmax vs sparsemax vs entmax). In particular, sparse-softmax should be clearly discussed here, since your proposal is highly influenced by this method. Crucially, some of the functions that you are using as a baseline in $\\S4$, are not even included in the discussion in $\\S2$. Although I understand that they are presented in App. A.3 (some of them are not very detailed), I think the paper could benefit a lot from this discussion at an earlier stage. Besides, you used the term “softmax loss” several times, which does not sound accurate to me – softmax is an activation function, not a loss. \n* The experimental results are not very convincing. Although AS-Softmax tends to perform generally better than the baselines, there is a drop in the performance when using the AS-Speed. Can you elaborate on why you think this happens? \n* I found some inconsistencies/typos (listed below), please revise them.  \n\nOther questions: \n* Can you please provide further information on how AS-Softmax works at test time? I don’t think the current version of the paper discusses that. \n* Can you please explain how you tuned the hyperparameters for all the methods presented in Table 2? I may have missed something but I could not find this information for all the methods you experimented with. \n\nMinor comments:\n\n* Please update the title\n* “Then the model could focus on learning to distinguish the target class from its strong opponents, which is also the great challenge in test.” -> I don’t think this sentence is clear. \n* Typo (*sparse*) in “we propose the Adaptive Spare softmax (AS-Softmax)”\n* Typo (remove whitespace before the comma) in “will be dropped from back propagation gradually ,” \n* Typo (*preserve*) in “Jean et al. (2014); Rawat et al. (2019) and Blanc & Rendle (2018) preserves”\n* Not using the right citation format in “The features learned by softmax loss have intrinsic angular distribution Liu et al. (2017).”\n* Typo (*a*) in “We have made it as an standard”\n* Typo in “of the original the exponential function”\n* Typo (*that*) in “As shown in Figure 1, experimental result shows taht”\n* What do you mean by “Most shared hyper-parameters refers to examples”?\n* Typo in “In additional, we conduct experiment on some (...) The result are presented”.\n* Typo (*shows*) in “As Figure 2 shown”\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PgdSKMnk1CM", "reviewer": "Reviewer_EExx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Authors believe there are problems with naive softmax function used in ML training. The design of softmax will lead to 2 potential problems: overfitting and mismatch between softmax loss and prediction goal. They proposed AS-SOFTMAX to address so. Authors addressed how these 2 problems could affect the final performance. But a more rigorous analysis is not presented. Experimental results found some interesting problems such as low correlation between accuracy and loss for certain datasets.", "summary_of_the_review": "Overall, I think this research question is interesting and the proposed method is certainly a doable solution. However, I do think the analysis is not in depth enough to convince me the efficacy. To encourage authors to justify their method, I am more leaning toward acceptance at this point of time, and hope authors resolve all of my confusions mentioned above. ", "strength_and_weaknesses": "Strength:\n1. An interesting problem.\n2. Reasonable method to work on it.\n3. Moderately enough experiments.\n\nWeakness:\n1. Presentation is confusing. Motivating example is not demonstrating the point.\n2. Not enough experiments to justify the articulated deficiencies is the root cause.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "OMhSRtTKaP", "reviewer": "Reviewer_FqyY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper argued that Softmax cross entropy loss function may lead to overfitting for large-output space problems. To resolve such issue, the author propose Adaptive Sparse Ssoftmax (AS-Softmax) that masked out a non-target logit when the target logit exceed the non-target logit by a specific margin. The author also extends AS-Softmax to the multi-label setting. Experiments are conducted on standard text classification datasets, where AS-Softmax demonstrated marginal improvement over other Softmax variants. ", "summary_of_the_review": "This paper has rather limited novelty, lacks theoretical justification (Weakness-1), and rather preliminary experiment results (Weakness-2). Hence, I am not inclined accepting this paper. ", "strength_and_weaknesses": "**Strength**\n- The paper writing is clear to follow\n\n**Weakness**\n- The proposed method lack theoretical justification. Several theoretical analysis are missing. For example, (1) How does it resolve the overfitting issue? Can you show AS-Softmax has better generalization error? (2) What metric is AS-Softmax optimizing? Is it a consist estimator?  \n- AS-Softmax introduced additional hyper-parameter delta, which requires extra hyper-parameter tuning\n- The experiment results are somehow weak with room to improve. For example, (1) What's the performance on the multi-label problem with extreme large output space, such as Wiki-500K and Amazon-3M? (2) What's the performance on image classification such as ImageNet-1K, which also has large number of classes?\n ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject