Effectively Clarify Confusion via Visualized Aggregation and Separation of Deep Representation
Authors: Yi Lin
Jingchi Jiang
Dongxin Chen
Zhaoyang Ma
Yi Guan
Xiguang Liu
Haiyan You
Jing Yang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Data Insufficiency
Class Imbalance
Evidience Unclearity
Confusion
Representation Learning

Abstract: Clarifying confusion is the most critical issue for improving classification performance. Confusion occurs with almost all classification models but tends to be ignored in excellent-performing models. The current mainstream research mainly focuses on solving the confusion in a specific case, such as data insufficiency and class imbalance. We believe that mining the commonalities of the same class and the gaps among different classes will effectively clarify the confusion. In this paper, we propose a novel, simple and intuitive Aggregation Separation Loss (ASLoss), as an adjunct for classification loss to clarify the confusion in some common cases. The ASLoss aggregates the representations of the same class samples as near as possible and separates the representations of different classes as far as possible.
We use two image classification tasks with three simultaneous confounding characteristics i.e. data insufficiency, class imbalance, and unclear class evidence to demonstrate ASLoss. Representation visualization, confusion comparison and detailed comparison experiments are conducted. The results show that representations in deep spaces extracted by ASLoss are sufficiently clear and distinguishable, the confusion among different classes is significantly clarified and the optimal network using ASLoss reaches the state-of-the-art level.

Under review as a conference paper at ICLR 2023
EFFECTIVELY CLARIFY CONFUSION VIA VISUALIZED AGGREGATION AND SEPARATION OF DEEP REPRE- SENTATION
Anonymous authors Paper under double-bling review
ABSTRACT
Clarifying confusion is the critical issue for improving classification perfor- mance. The current mainstream research mainly focuses on solving the confusion in a specific case, such as data insufficiency and class imbalance. In this paper, we propose a novel, simple and intuitive Aggregation Separation Loss (ASLoss), as an adjunct for classification loss to clarify the confusion in some common cases. The ASLoss aggregates the representations of the same class samples as near possible and separates the representations of different classes as far as possible. We use two image classification tasks with three simultaneous confounding char- acteristics i.e. data insufficiency, class imbalance, and unclear class evidence to demonstrate ASLoss. Representation visualization, confusion comparison and de- tailed comparison experiments are conducted. The results show that representa- tions in deep spaces extracted by ASLoss are sufficiently clear and distinguishable, the confusion among different classes is significantly clarified and the optimal net- work using ASLoss reaches the state-of-the-ar level.
INTRODUCTION
Clarifying confusion is the most crit- ical issue for improving classifica- tion performance. In fact, all pre- diction mistakes in classification are confusion, i.e., the model incorrectly considers samples of class "A" as class "B". Confusion occurs with almost all classification models but tends to be ignored in excellent- performing models because the main- artificially
Aggregating
Agtregating
Separated
Aggregated
Augregated
Separating
stream datasets structed to be nearly perfect [Tor- ralba et al. 2008); RRussakvvsyy al. (2015): Lin et al. (2014). However, the sample distribution in practical problems is often not as perfect as artificial datasets. First, collecting enough samples is difficult in some situations, such the medical field due privacy invasion LinioonLLnnn et al. (202T). High-performing models are extremely hard to obtain with insufficient data. Second, samples of different categories generally do not follow uniform distribution but class imbalance Liu (2118). The vast majority of samples belong to small number of categories. Samples from unim- pressive categories will be confused by the model into categories Cui et al. (2019). In addition, the classification boundaries of some problems confuse samples into similar classes. More seriously, many real-world problems simultancously have more than case confusing the models.
Figure 1: Schematic of Aggregation Separation Loss.
(2019); Buda et al.)
not clear Lin et al. (2022) Models will always
Many methods have been proposed to clarify confusion, but almost focus on insufficient data or class imbalance. For insufficient data, few-shot learning is mainstream, including meta-learning Antoniou et al.] (2019); J Jamal et al. (2019); Lifchitz et al.] (2019); Metz et al.] (2019); Rajeswaram
al. (2019) and metric-based /Zhang et al. 2020) Li al. (2019); /Sung al. (2018) methods.

---

Under review as a conference paper at ICLR 2023
These methods have made great progress but the lack of data in few-shot learning is too extreme while its true generalization performance is still unclear. For class imbalance, resampling Chawla et al. (2022); Wang et al.] (2019c); Zhou et al. (2020) and reweighting Cao et al.] 2 Cui et al. (2019) are two mainstreams. All these methods balance input sample frequency or loss of different classes but ignore the characteristic difference among categories.
In view of previous research, we believe that focusing on the class-level representation of samples to mine the commonalities of the same class and the gaps among different classes can clarify the confusion in these common fusion cases. As long as deep learning models can construct suffi- ciently clear representations during extracting features, the confusion under common conditions will be clarified. We propose an assistant to optimize the representations into distinguishable in the geometric space of deep features to efficiently clarify the confusion.
In this paper, we propose novel, simple and intuitive method called Aggregation Separation (ASLoss), as an adjunct for classifi- cation loss. This loss can be adopted on any linear feature extraction layers shown in Figure 2 constructing distinguishable representations in geometric spaces of deep features to clarify the confusion in common cases as shown in [ It aggregates the representations of the same class samples as near as possible and separates the representations of different classes as far as possi- ble to mine the commonalities of the same class and the gaps among different classes. To interpret its effect, the distinguishable representations can be visualized by condensing the representation layers into two dimensions.
Representation Extraction Layers
Loss
Clasifiiation
Deep Network
Classification Layer
Apprepaiion Soparation LOwN
Network Output Layer
Figure 2: Schematic Using Aggregation Separation Loss.
We validate our method using two image classification tasks that simultaneously have three easily confusion-caused common conditions: data insufficiency, class imbalance, and unclear evidence. The experimental results show that the representations in deep geometric spaces are sufficiently clearer, the performances of various deep networks are efficiently improved and the optimal network achieves stae-off-the-at The code for this work is available on GitHut
2 RELATED WORK
Contrastive learning is a series of unsupervised methods that use an agent task to minimize the distances among varieties of the same sample and maximize the distances among different sample varieties Cheie al. (2020); ; Gao al. (202T): ); Wang al. (202Tb); Bachman et al. 2011). These ex- cellent methods using unsupervised pretrain improve downstream classification by finetuning. The key is using the agent task which transforms a sample into many varieties. Our method is plug-and- play without pretraining and transforming. And ASloss directly pulls and pushes the representations of samples but not varieties.
Metric learning also optimizes sample distances Wang et al. (2018b) Wang al. (201 2018|| 2017a);
Liu et al. (0017); : Zhou et al. (2019); Wang et al. (201 9a : Xu et al. 2011 Zheng et al. (2019) Wang Some of them propose better activation methods and others use linear transtormation Our method is similar to the linear transformation series but uses linear transformation on multiple layers, leading to more separable representations. Furthermore, our method does not set the class interval but sets a scope to make the distance between classes as large as possible, which is more straightforward
ct (201 (2019b)
Triple loss sets triple of (anchor, positine, negotive) to pull the same class samples and push other class samples Yuan al. (2020): Schroff et al. (2015), ASLoss is more flexible, calculating all
'The link will be open if work can be accepted.
2

---

Under review as a conference paper at ICLR 2023
samples directly. Similarly, ASLoss does not set the interval but sets the optimization scope that classes are as far apart and evenly distributed as possible.
DEFINITION OF CONFUSION
For any sample x X where X is category, if person model considers belong to another category Y. this misperception is confusion. Confusion often occurs when the samples are very similar, the perceiver is inexperienced. and the class boundaries are clear. For an example of similar samples, people often misidentify twins. For an example of inexperience, children who have never seen a tiger will consider a young tiger as cat. The insufficient data and imbalanced class in machine learning belong to this case. For an example of the unclear boundary, it is difficult to determine whether person is middle-aged or young based on only appearance.
AGGREGATION SEPARATION LOSS
The ASLoss is simple and intuitive method including three parts: inner aggregation, outer separa- tion and boundary constraint as shown in Figure 2
4.1 INNER AGGREGATION
To mine the commonalities of the same class, the inner aggregation narrowing deep representations of the same class near as possible is computed as:
discnner
E uc== (()r..)
(1)
Nermey
N N I>(==o) 1=1==
Najez
(2)
where N is the total number of samples, c; denotes the class of sample i, Ne.myy the total number of c, Cj and r; refers to the representation in a deep geometric space output from a certain linear layer of sample i. II(4 = C₁) is indicator function whose value is when G; Cj. otherwise the value is O. P(r,,ry) is a distance metric of r; and ry such as Euclidean distance etc. disinner represents the average intra-class distance all samples. Minimizing dis will aggregate the representations of the same class together during optimizing networks.
4.2 OUTER SEPARATION
To mine the gaps among different classes, the outer separation pushing the representations of differ- ent classes away as far as possible is calculated as:
N N
diSouter Noctte im1
ΣΣ
lle # c,))()))))
(3)
N
N
Nejtoz
Σ
ΣΣ T(c # (4) i=
where is the total number of samples, e; denotes the class of sample i, Ne the total number of c; A çj and r; refers to the representation in deep geometric space output from a certain linear layer of sample i. I[c; * c;) is an indicator function whose value when c * Cj. otherwise the value is O. Drr,,r,) is distance metric of r, and r,. disputer represents the average inter-class distance all samples. Maximizing dis will separate the representations of different classes farther during optimizing networks.
3

---

Under review as a conference paper at ICLR 2023
ACNE04 Dataset
Subset CelebA Dataset
700
434
600
Total Train Test
400
352
348
Total
Train
Test
500
350
410
250
200
200
146
103
150
125
100
100
100
72
Mild(1-5)
Moderate(6
2215re(21550)
Very Severe(>50)
NOBNS
NOBS
OBNS
0BS
The total, trainset and testsct sample numbers for each category of the ACNE04 and CelebA subset. (NO: mouth not open, NS: not smile, mouth open and smile)
Figure 3:
4.3 BOUNDARY CONSTRAINT
Because the farthest distance in a space can be infinity, The optimization of equation 3 will not converge. So use the boundary constraint limit the representation locations to ensure training convergence as:
N N Σ D(r,, constraint||a
dishoundary
where is the total number of samples and r; refers to the representation deep geometric space output from certain linear layer of sample i. (rr,, 0) is distance metric of and origin of the space. ||- is L2 norm. constraint is a parameter used limit the distance of representations to the origin. dishoundary represents the average Euclidean distance of all samples to the boundary. When the number of categories is large, disboundary can be set larger, so that different categories find suitable spatial positions. However, if diSboundory is too large, the convergence will be slow. Minimizing diShoundary ensures that the representations will not infinitely far from the origin, then the training process can converge.
4.4 OVERALL OPTIMIZATION
The optimization of ASLoss is calculated as follows:
ASLoss AnnnerdiSiines AonterdiSonter AboundarydiSbounder
where Ainner, Aouter and Aboundory are three parameters. To ensure overall convergence, Aboundary should be slightly larger than Nouter.
The final overall optimization of the whole network is computed as:
LoSSowerall
AossAS + Los
where Aast and Acts are two parameters and Los8ds refers to the classification loss such as cross entropy loss.
EXPERIMENT SETTINGS
5.1 TASK DESIGN, DATASETS AND EVALUATION METRICS
Task Design and datasets: We conduct experiments two classification tasks that are extremely prone to confusion, i.e. acne severity grading and facial expression recognition verify the effec- tiveness of the proposed method. Both tasks simultancously have three confounding characteristies: data insufficiency, class imbalance, and vague evidence.
Acne Severity Grading: We use the ACNE04 dataset Wu et al. (2019) for acne severity grading. It is an open image dataset for facial acne severity grading. This dataset was collected from outpatients, its number of class samples dataset totally contains 1457 images. According to the annotations of the ACNE04 dataset, there four levels (four categories): mild, moderate, severe and very severe. For each category, 80% of
reflect the true distribution of the severity of acne patients. The

---

Under review as a conference paper at ICLR 2023
the samples are randomly divided into the training set and the rest are the test set. The class sample distribution is shown in Figure 3 First, the data insufficient for deep learning where only 633 samples in the fattest class. Second, the class imbalance is also serious. The sample size of the largest class is almost five times than that of the smallest class. Next, the evidence for distinguishing among the four categories is unclear. The severity depends on the number of very fine-grained lesions on the whole face where I to 5 is mild, 6 to 20 is moderate, 21 to 50 is severe, and more than 50 is very severe. For models, inputs are large faces but the classification evidence the number of extremely small objects which is very difficult to mine. This blurs the lines among the different categories. Some examples can be seen in Figure
Facial Expression Recognition: We use a sub- set of the CelebA dataset [iu et al. (2015) for this task. CelebA is an open dataset containing 202,599 face images of 10,177 celebrity identi- ties. Its annotations contain 40 attributes, such as eyebrows, eyes, nose shape, and more. We selected two attributes to form four categories: NO&NS (no mouth open and no smile), NO&S (no mouth open and smile), O&NS (mouth open and no smile), and O&S (mouth open and smile). Then, we randomly selected 1000 sam- ples according to the class distribution of the ACNE04 dataset and divided 80% as the train- set and the rest as the test set as shown in Fig-
ACNE04
Mild
Moderate
Severe
Very Severe
CelebA Subset
ure 3 Similarly, this subset is also insufficient, imbalanced and the evidence for distinguishing the four classes is also unclear because differ- not significant. The inputs are similar faces but the identification should be focused only the mouth.
No mouth open No mouth open No smile
Mouth open No smile
Mouth open Smile
Smile
Figure 4: Examples for each category of the ACNE04 and CelebA subset.
ences among categories
Evaluation Metrics:
We use the accuracy, precision, recall and fl f1-score commonly used in classi-
fication tasks the evaluation metrics.
Implementation Details All models in the experiment use the same implementation settings. For the parameters of the ASLoss, the Ainner and Aouter are both set as 0.001 and the Aboundary is set as 0.01. For the parameters of the overall optimization, the Xast and the Acts are both set as 0.5. For training different networks, the learning rate is set as 1e-3, the batch size is set 16 and the maximum training epoch is 400. Adam is chosen as the optimizer employed for training all models. All programs are coded using Python with PoTorch All the experiments are conducted on a Linux Ubuntu with 32G RAM and an NVIDIA Geforce RTX 3090 GPU of 24G VRAM.
EXPERIMENT RESULTS, ANALYSIS AND DISCUSSION
6.1 EFFECTIVE OF AGGREGATION SEPARATION Loss
To verify the effectiveness of ASLoss, we use ASLoss with different distance metrics on several mainstream deep vision networks. The mainstream vision networks include convolution-basel VG-
GNet Simonyan & Zisserman (2015), ResNet He et al. (2016). EfficientNet Tan & Le (2019) and transtormer-based VIT Dosovitskiy et al. (2021a) and T2T- Vit Yuan et al. (021). For each network, we add linear mapping layers to the output backend of the original network to gradually reduce the number of features into two di- mensions and then add classification layer. The schematic diagram is shown in Figure 2 Next, we use ASL.oss Loss of three distance measures, L.1 distance, 1.2 distance and cosine similarity, respectively, on the last two-dimensional feature layer. The comparison results on the two tasks of acne severity grading and facial expression recognition are shown Table
al. (2021). Transtrrantrrmer [Lu et al. (2021). WVTTWang
The performances of all the networks have been improved after using ASLoss, no matter the convolution- based or transformer- based network. For the severity grading task, the ASLoss with LI distance metric improves the VGG16 network the most. The ASLoss with L2 distance brings the most improvement to the RES50, EFFICIENT B3, Swin-Transformat and PVT networks. The ASLoss with cosine similarity improves the ViT and T2T-ViT the most significantly. Among all the
5

---

Under review as a conference paper at ICLR 2023
Table 1:
Comparison results of mainstream deep vision methods not using and using ASLoss of different distance metrics on acne severity grading (left) and facial expression recognition (right). ACC: Accuracy, PRE: Precision, REC: Recall and Fl: Fl-score. Ll: LI distance, L2: L2 distance
and COS: Cosine Similarity. ASL: ASLoss
Tasks
Acne Severity Grading
Facial Expression Recognition
Methods
ACC1
PRET
RECT
Flf
ACC1
PRE1
REC1
FI
VGG16
85.62
85.98
84.00
84.69
80.81
79.64
80.10
79.86
VGGI6+ASL(LI)
85.96
86.73
85.27
85.59
82.83
82.59
82.97
82.46
VGGI6+ASL(L2)
86.64
86,56
84.49
85.04
82.83
80.03
79.66
79.65
VGGI6+ASL(COS)
85.96
84.54
82.71
83.30
82.83
81.82
79.29
80.05
RES50
83.22
82.88
81.26
81.76
82.32
78.80
78.44
78.55
RESSO+
84.25
84.98
81.19
82.86
83.84
82.85
85,38
83.93
RES50+ASL((12)
86.64
86.56
84.49
85.04
85.35
85.00
82.40
83.59
RESSO+ASI<COOS)
83.90
83.56
82.64
82.27
83.84
81.33
81.16
81.05
EFFICIENT _B3
82.53
83.60
80.74
81.80
79.80
78.81
78.74
78.77
EFFICIENT B3+ASL((I))
86.30
86.19
83.90
84.88
83.84
83.50
84.13
83.73
EFFICIENT B3+ASL(L2)
87.33
89.17
88.46
88.78
83.33
82.78
83.78
83.10
EFFICIENT B3+ASL(COSS)
85.62
86,40
85.08
85.44
83.33
80.88
81.81
81.30
VIT ViT+ASL(LI)
65.75
66.93
64.94
65,4
66.67
57.30
47.30
47.57
82.88
83.04
81.19
81.09
82.32
81.59
80.32
80.90
ViT+ASL(L2)
82.53
82,63
79.42
80.12
83.84
81.37
79.51
80.31
ViT+ASL(COS)
82.53
83.75
82.99
83.29
81.82
81.17
82.74
81.01
Soin-Transfomer
84.93
85,20
82.61
83.04
77.78
76.54
74.77
74.95
Wwin-Transfomeer--ALLLL
85.96
85.92
82.84
83.83
80.30
75.95
79.43
77.20
Swin- wwin-Transfommer---SLL12
86.99
85.08
85.06
84,90
84.34
81.49
82.25
81.57
Swin- Transoomer--SSSSCCOS
83.22
82,81
78.11
80.14
81.31
80.62
76.10
78.08
PVT
80.48
80.71
78.83
79.71
75.25
72.20
68.80
70.10
PVT+ASIALI)
83.90
82.04
81.34
81,46
76.26
78.67
70.57
73.65
PVT+ASL(L2)
83.90
82.35
81.34
81.66
77.78
79.06
73.73
75.99
PVT+ASL(COS)
83.56
82.38
79.84
80.80
79.29
79.39
7.663
74.48
T2T-VIT
82.53
82.14
79.64
79.75
79.80
78.97
76.10
76.83
T2S-VITASL(LII T2T- (LL))))) T2T-VI++ASL(COS)
83.22
81.67
82.26
81.83
78.79
81.10
76.74
78.68
83.90
84.74
83.19
83.67
81.82
85.25
76.62
79.86
84.25
83.42
84.79
84.08
80.30
79.13
76.32
77.25
networks, ASLoss brings the most obvious improvement ViT with almost all evaluation metrics improved by about 15%. Meanwhile, the best performing model is EFFICIENT.DE .B3 using ASLoss of L2 distance, with evaluation results over 87%, surpassing the performances of all the other networks. For the facial expression recognition task, the ASLoss with L1 distance improves the VGG16 and EFFICIENT B3 mostly. The ASLoss with L2 distance improves the Res50, Swin- Transformer and TIT-VIT most significantly. The ASLoss with cosine similarity brings the most obvious improvement to ViT and PVT. Among them, ASLoss brings the most obvious improvement ViT, some evaluation metrics are even improved by about 20%. The best performing model is RES50 using ASLoss with L2 distance, with all evaluation results around 83%, surpassing most of the other network performances. Overall, ASLoss with L2 distance brings significant improvements on most networks.
First, the performance these networks on the acne severity grading is slightly better than that on the facial expression recognition. On the one hand, the data volume for the acne severity grading slightly larger than that of the facial expression recognition. ing and smiling are more easily confused, compared to the number of small lesions. So the net- works are more easily to obtain effective generalization on the acne severity grading task. Next, the transformer-basee ViT-series networks do not perform significantly better than convolution-basee networks. We believe that it is difficult for the ViT series of methods to converge at a good enough performance when the data is not sufficient for both tasks. In addition, the distance metries that improve significantly on most networks are the LI and L2 distances, although the ASLoss of all three distance metrics can improve the performance of different networks. In fact, when using L.1
the other hand, mouth open-

---

Under review as a conference paper at ICLR 2023
Table 2: Comparison results of using ASLoss multiple linear layers in front of the classification layer on acne severity grading (left) and facial expression recognition (right). ACC: Accuracy, PRE: Precision, REC: Recall and Fl: Fl-score. L2: L2 distance. ASL: ASLoss
Tasks
Acne Severity Grading
Facial Expression Recognition
Base Models
EFFICIENT + ASL(L2)
Res50
ASL(L2)
Layers
ACC1
PRET
REC1
Flf
ACC1
PRE1
REC1
FI
Base model
82.53
83.60
80.74
81.80
82.32
78.80
78.44
78.55
ASLoss on the last one layer
87.33
89.17
88.46
88.78
85.35
85.00
82,40
83.59
ASLoss on the last two layers
85.96
85.57
86.37
85,95
84.34
81.78
81.71
81.46
ASLoss the last three layers
85.27
85.74
85.74
85.54
86.36
84.32
81.05
82.41
ASLoss on the last four layers
85.62
86.01
84.86
85.39
84.34
81.78
80.20
80.88
and 1.2 distances, ASL Loss aggregates and diverges the representation locations of different classes in deep space. While using cosine similarity, it gathers and diverges the angles of representation vectors. We believe that angular differences are still difficult to distinguish near the origin, com- pared to their positional differences in space. Therefore, ASLoss using LI and L2 distance metries performs slightly better than cosine similarity.
Such experimental results show that ASLoss can improve the performance of various mainstream visual networks, including convolution based networks and transformer-bas based series. And the two best performing models, EFFICIENTNET B3 and Res50, using the ASL.ss of the 1.2 distance metric on the two tasks are selected to conduct the following experiments.
6.2 AGGREGATION SEPARATION LOss ON MULTIPLE SPACES
apply ASLoss to the last layer, the last two layers, the last three layers and the last four layers before the output layer, respectively, and compare their performances for both the acne severity grading task and facial expression recognition task. The schematic is shown in Figure 2 Since the best performing models on acne severity grading and facial expression recognition tasks are EFFICIENT _B3 and Res50 optimized by ASLoss of L2 distance metric, we continue use these two networks as base models. The comparison results are displayed in Table The first row in the table shows the results of the base model without ASLoss. Rows two to five show the experimental results of gradually increasing the number of representation spaces using ASL.oss.
From the experimental results, the performance of the base model can improved by using ASLoss on representations of all four multi-spatial strategies. For the acne severity grading task, the em- ployment of ASLoss on four multi sultti-pace representations improves the accuracy of the base model by about 3%, the precision by about 2%, the recall by more than 4%, and the fl-score by more than 4%. Among them, the model optimized by ASLoss only on the one previous layer of the output layer achieves the best performance. Compared with the base model, the accuracy is increased by nearly 5%, the precision is increased by more than 5%, the recall rate is increased by nearly 8%, and the fl-score is increased by nearly 7%. For the facial expression recognition task, the results similar to the acne severity grading. The employment of ASLoss on four multi-space representa- tions improves the accuracy of the base model by over 3%, the precision by about 3%, the recall by around 3%, and the fl-score by about 4%. Among them, the model optimized by ASLoss only on the one previous layer of the output layer achieves the best performance. Compared with the base model, the accuracy is increased by over 3%, the precision is increased by more than 6%, the recall rate is increased by nearly 4%, and the fl-score is increased by over 5%.
For the performance on the two tasks, increasing the feature space employed by the ASLoss does not gradually improve the model performance. We believe that excellent performance is based on extracting enough representations. Premature aggregation and separation of feature representations may lead to insufficient feature extraction. At the same time, too many layers to perform feature aggregation and separation at the same time may lead to contradictions in gradient update, resulting in inconsistent update directions of network weights. Weight changes in the previous layers will cause the subsequent layers to change. Convergence will become difficult. Therefore, only the aggregation and separation of representations in the one layer before the classification layer achieves the best results.
7

---

Under review as a conference paper at ICLR 2023
Representaaions of Train Data
Representations Trbin bota
Regrosentationn ior Train Data
Represenations of Train Data
Representations of Test Data
Reprosataatios Teut Data
Representations Test
Repr Rupresentations Test Data
Without ASLoss
With ASLoss
Withous
ASLoss
Acne Severity Grading Task
Facial Expression Recognition Task
Figure 5: Visualization of representations. The left and right halves for the acne severity grading (EF- FICIENT B3 with ASLoss L2 distance) and the facial expression recognition (Res50 with ASLoss of L.2 distance). For each task, the left and right two subfigures are the representations extracted without and with ASLoss. The upper and lower subfigures are representations of train data and test data.
6.3 REPRESENTATION VISUALIZATION
We connect the representation extraction layer after the original network output and set the feature extraction layer finally to be two-dimensional. The two-dimensional output representations alized and compared as shown in Figure 5 The left and right halves are for the acne severity grading and the facial expression recognition tasks. The EFFICIENT B3 and Res50 are selected for the two tasks to conduct this experiment due to their excellent performances.
visu-
For the acne severity grading task, the training sample representations extracted by the model with- out ASLoss have no obvious boundary and the distribution is scattered. For the training repre- sentations using ASLoss, the representations of the same category are more concentrated, and the samples of different categories can be clearly separated. For test data, the representation overlap serious when not using ASLoss. After using ASLoss, the overlap of representations in space alleviated. For the facial expression recognition task, three categories concentrated in the coordinate origin and coordinate axes in the training samples without ASLoss, leading to heavy overlap. After using ASLoss, the representations of different classes spread out. For the test data representa- tions, heavy overlap happens without ASLoss. After using ASLoss, the overlap is alleviated. All representations using ASLoss constrained to the ideal range without disorderly dilation.
This result shows that ASLoss can effectively aggregate in-class samples to separate out-of-class samples. For the test samples of the tasks, the effect of widening the distance of different classes is significant. This increases the distinguishability of these samples that are located among different class distributions. So the confusion among different classes can be clarified.
6.4 COMPARISON OF CONFUSION
The confusion matrixes for the two tasks are shown in Figure The EFFICIENT BB and Res50 are also selected as the comparison models for the two tasks respectively. For each task, the left and right two confusion matrixes are from the models without and with ASLoss of L. distance. The upper and lower confusion matrixes are for train and test data.
For both tasks, the use of ASLoss can improve performance of the models on not only the training data but also the test data. On the training data, both base models perform a little confusion on dif- ferent categories. With the help of ASLoss, the confusion on the training dataset almost completely disappeared. On the test data, the predictions of the models without ASLoss perform confusion seriously. After using ASLoss on the models, the performance sufficient data is improved sufficiently, while the performance of the two classes 0 and with more samples decreases slightly. The overall accuracy is significantly improved.
the two classes 2 and with in-
This result demonstrates that the ASLoss effectively clarifies the confusion. Such results can be explained from visualization. The representation distances of the categories are pulled apart. The distribution of the same class is concentrated. So the confusion be clarified.
8

---

Under review as a conference paper at ICLR 2023
Train
Grading
Severity
Train Recognition
Recognition
96%
2%
200%
100%
100%
-08
0.8
2%
0.5
90%
96%
1027
8%
71%
23%
17
0.4
10EN
302%
0.2
0.2
0.2
100%
57%
102%
Phod
Text Acne Severity Graddngg
Severity Grading
Test
Recognition
Test Racial Expression Recognition
91%
7%
89%
12%
08
14%
13%
0.8
0.b
93%
91%
-066
7%
90%
1%
2%
10%
61%
10%
0.4
14%
93%
3%
8%
0.4
0.4
02
0.2
02
96%
32%
71%
69
6%
89%
Reddctte
Without Acne Severity Grading Task
With ASLoss
Without ASt
With ASLoss
Facial Expression Recognition Task
Figure 6: Comparison of confusion matrixes. The left and right halves are for the acne severity grading (EFFICIENT_BS with ASLoss of L2 distance) and the facial expression recognition (Res50 with ASLoss of L2 distance). For each task, the left and right confusion matrixes are from the models without and with ASLoss. The upper and lower confusion matrixes are for train data and test data.
Comparative experimental results with state-of-the-art methods on acne severity grading (left) and facial expression recognition (right), ACC: Accuracy, PRE: Precision, REC: Recall and Fl: Fl-score.
Table 3:
Tasks Methods
Acne Severity Grading PREt
Facial Expression Recognition ACCt
ACC1
REC
Flf
PREt
REC1
FIf
BaseModel Focal Loss
82.19
81.75
79.26
79.25
84.85
83.32
82.62
82.94
BaseModel Weighted CF
83.90
83.87
81.70
82.38
83.84
81.38
82.31
81.69
BaseModel CBLoss
82.88
81.94
79.32
80.46
82.83
80.57
79.77
80.13
BaseModel TDE
85.27
85.32
83.22
83.78
84.85
82.91
81.74
82.15
BaseModel ASLoss(ours)
87.33
89.17
88.46
88.78
85.35
85.00
82.40
83.59
6.5 COMPARISON WITH STATE-OF-THE-ARTS
We compare the proposed ASLoss with some state-of-the-art methods including Focal Loss Lin (020), Weighted Cross Entropy Wang et al. (2017b), CBLoss Cui et al. (2019) and TDE Tang al.
al. 2020 shown in Table
From the experiment results, the proposed ASLoss outperforms other methods on all metrics for the acne severity grading task. For the facial expression recognition, the overall performance of ASLoss outperforms all other methods, although the recall of Focal Loss is slightly higher than the ASLoss. The experimental results show that the performance of ASLoss reaches the state-of-the-arr level.
7 CONCLUSION
In this paper, we propose novel, simple and intuitive ASLoss, as an adjunct for classification loss to clarify the confusion in some common cases. The ASLoss aggregates the representations of the same class samples as near as possible and separates the representations of different classes as far as possible to mine the commonalities of the same class and the gaps among different classes. We use two classification tasks with three simultaneous common confounding characteristics i.e. data insufficiency, class imbalance, and unclear class evidence to demonstrate the performance of ASLoss. We conduct representation visualization, confusion comparison, and detailed comparison experiments. The experimental results show that the model using ASLoss can extract sufficiently clear and distinguishable representations in deep spaces, the confusion among different classes is significantly clarified and the best-performing network reaches the state-of-the-art level.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Antreas Antoniou, Harrison Edwards, and Amos J. Storkey. How to train your maml. In ICLR, 2019.
Philip Bachman, R. Devon Hjelm, and William Buchwalter. Learning representations by maximiz- ing mutual information across views. In Neural Information Processing Systems, 2019.
Mateusz Buda, Atsuto Maki, and Maciej A. Mazurowski. A systematic study of the class imbal- ance problem in convolutional neural networks. Neural networks the official journal of the International Neural Nenwork Sociery, 106:249-2555 2018.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Aréchiga, and Tengyu Ma. Learning imbalanced datasets with abbel-distribution-awaae margin loss. In NeurIPS, 2019.
N. Chawla, K. Bowyer, Lawrence O. Hall, and W. Philip Kegelmeyer. Smote: Synthetic minority over-sampling technique. Artif. Intell. Res., 16:321-333 2002.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. ArXiv, abs/2002.05709, 2020.
Yin Cui, Menglin Jia, Tiung-Yi Lin, Yang Song, and Serge J. Belongie. Class-balanced loss based on effective number of samples. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9260-9269, 2019.
A. Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, M. Dehghami, Matthias Minderer, G. Heigold, S. Gelly, Jakob Uszkoreit, and N. Houlsby. An image is worth 16x16 words: abs/2010. 19929,9,, 2021.
Transformers for image recognition at scale. ArXiv,
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simcse: Simple contrastive learning of sentence embeddings. ArXiv, abs/2104.08821, 2021.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Muhammad Abdullah Jamal, Guo-Jun Qi, and Mubarak Shah. Task agnostic meta-learning for few- shot learning. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11711-11711 2019.
Wenbin Li, Lei Wang. J. Xu, Jing Huo, Yang Gao, and Jicbo Luo. Revisiting local descriptor based image-to-class measure for few-shot learning. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7253-7260, 2019.
Yann Lifchitz, Yannis Avrithis, Sylvaine Picard, and Andrei Bursuc. Dense classification and im- planting for few-shot learning. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9250-9259, 2019.
Tsung-Y Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context, In ECCV, 2014.
Tsung- Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 02:318-327, 2020.
Yi Lin, Yi Guan, Zhaoyang Ma, Haiyan You, Xue Cheng, and Jingchi Jiang. An acne grading framework on face images via skin attention and sfnet. 202 IEEE International Conference Bioinformatics and Biomedicine (BIBM), pp. 2407-2414, 2021.
Yi Lin, Jingchi Jiang, Zhaoyang Ma, Dongxin Chen, Yi Guan, Haiyan You, Xue Cheng. Bingmei Liu, and Gongning Luo. Kieglfn: unified acne grading framework on face images. Computer methods and programs in biomedicine, 221:10691 2022.
10

---

Under review as a conference paper at ICLR 2023
Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.
2017.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. ArXiv, abs/2103. 2021.
Ziwei Liu, Ping Luo. Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X. Yu. Large- scale long-tailed recognition in an open world. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2532-2541, 2019.
Luke Metz, Niru Maheswaranathan, Brian Cheung, and Jascha Narain Sohl-Dickstein. Meta- learning update rules for unsupervised representation learning. In ICLR, 2019.
Aravind Rajeswaran, Chelsea Finn, Sham M. Kakade, and Sergey Levine. Meta-learning with im- plicit gradients. In NeurlPS, 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei- Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115:211-252. 2015.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 815-823, 2015.
K. Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recog- nition. CoRR, 0bss/1409.155 2015.
Flood Sung, Yongxin Yang. Li Zhang, Tao Xiang, Philip H. S. Torr, and Timothy M. Hospedales. Learning to compare: Relation network for few-shot learning. 2018 IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 1199-1208, 2018
Mingxing Tan and Quoe V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. ArXiv, abs/1905.1 1946, 2019.
Kaihua Tang, Jianqiang Huang, and Hanwang Zhang. Long-tailed classification by keeping the good and removing the bad momentum causal effect. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
Antonio Torralba, Rob Fergus, and William T. Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. JEEE Transactions Pattern Analysis and Machine Intelligence, 30: 1958-999977, 2008
Feng Wang, Xiang Xiang, Jian Cheng, and Alan Loddon Yuille. Normface: L2 hypersphere embed- ding for face verification. Proceedings of the 25th ACM international conference on Multimedia, 2017a.
Feng Wang, Jian Cheng, Weiyang Liu, and Haijun Liu. Additive margin softmax for face verifica- tion. IEEE Signal Processing Letters, 25:926 930, 2018a.
H. Wang, Yitong Wang, Zheng Zhou, Xing Ji, Zhifeng Li, Dihong Gong, Jin Zhou, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. 2018 IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 5265-5274, 2018b
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang. Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 548- 558, 2021a.
11

---

Under review as a conference paper at ICLR 2023
Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual pre-training. 2021 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3023-3032, 2021b.
Xinshao Wang, Yang Hua, Elyor Kodirov, Guosheng Hu, Romain Garnier, and Neil Martin Robert- son. Ranked list loss for deep metric learning. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5202-5211, 2019a.
Xun Wang, Xintong Han, Weilin Huang, Dengke Dong, and Matthew Scott. Multi-similarity loss with general pair weighting for deep metric learning. 2019 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 5017-5025, 2019b.
Yiru Wang, Weihao Gan, Wei Wu, and Junjie Yan. Dynamic curriculum learning for imbalanced data classification. 2019 IEEE/CVF International Conference Computer Vision (ICCV), pp- 5016-5025, 2019c.
Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. Learning to model the tail. In NIPS, 2017b.
Xiaoping Wu, Wen Ni, Liang Jie, Yu-Kun Lai, She Cheng, Dongyu, Ming-Ming, and Jufeng Yang- Joint acne image grading and counting via label distribution learning. IEEE International Conference on Computer Vision(ICCV), 2019.
Xinyi Xu, Yanhua Yang. Cheng Deng, and Feng Zheng. Deep asymmetric metrie learning via rich relationship mining. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4071-4080, 2019.
Li Yuan, Yumpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Francis Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. 2021 IEEF/CVF International Conference on Computer Vision (ICCV), pp. 538-547, 2021.
Ye Yuan, Wuyang Chen, Yang Yang, and Zhangyang Wang. defense of the triplet loss again: Learning robust person re-identification with fast approximated triplet loss and label distillation. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1454-1463, 2020.
Chi Zhang, Yujun Cai, Guosheng Lin, and Chunhua Shen. Deepemd: Few-shot image classification with differentiable earth mover's distance and structured classifiers. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12200-1220,, 2020.
Wenzhao Zheng, Zhaodong Chen, Jiwen Lu, and Jie Zhou, Hawaness-aware deep metric learning. 2019 IEEC/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 72-81, 2019.
Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. Bbn: Bilateral-brande network with cumulative learning for long-tailed visual recognition. 2020 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 9716-9725, 2020.
Shengwei Zhou, Caikou Chen, Guojiang Han, and Xielian Hou. Double additive margin softmax loss for face recognition. Applied Sciences, 2019.
12


Reviews:
[{"review_id": "aAGV67xKi7M", "reviewer": "Reviewer_Ziee", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel, simple and intuitive Aggregation Separation Loss (ASLoss), which aggregates the representations of the same class samples as near as possible and separates the representations of different classes as far as possible. The authors conduct extensive experiments on diffirent scenarios i.e. data insufficiency, class imbalance, and unclear class evidence to demonstrate ASLoss, and the results show that representations in deep spaces extracted by ASLoss improve classification performance and reaches the state-of-the-art level.", "summary_of_the_review": "The idea is incremental, where ASLoss combines multiple loss functions (most are known) together to clarify confusion and improve separation over different classes.", "strength_and_weaknesses": "Quality/Clarity: the paper is well written and the techniques presented are easy to follow. Its motivation is to aggregate the representations of the same class samples as near as possible and separate the representations of different classes as far as possible. And the authors design the corresponding Aggregation Separation Loss (ASLoss) to learning these representations. The method is validated our method on two image classification tasks and show promising results.\n\nOriginality/significance: the idea is incremental, which combines multiple loss functions (most are known) together to clarify confusion and improve separation over different classes. Extensive experiments are conducted to show its advantage on confusion-caused common conditions: data insufficiency, class imbalance, and unclear evidence. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XeRWCXBhZt", "reviewer": "Reviewer_7v3V", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new loss function, e.g., Aggregation Separation Loss (ASLoss), to clarify confusion to improve image classification performance. \nSpecifically, the ASLoss aggregates the representations of the same class samples as near as possible and separates the representations of different classes as far\nas possible. Experimental results on two datasets (in the case of data insufficiency, class imbalance, and unclear class evidence) are conducted to illustrate the effectiveness of the proposed method.", "summary_of_the_review": "Overall, I think this paper proposes a new loss function for image classification where the novelty is limited and its effectiveness lacks theoretical support and ablation studies. Thus, the quality is poor and I suggest rejection.\n", "strength_and_weaknesses": "## Strength\n1. Overall, this paper is written rather clearly.\n2. The considered problem is relevant to the community.\n\n\n## Weaknesses\n1. The \"confusion\" issue, as the authors claimed,  lacks a formal definition and seems a little confusing.\n2. This paper lacks theoretical results to support the claims.\n3. For the proposed ASLoss, the novelty is limited since the ideas of aggregating intra-class samples and separating inter-class samples are widely used in image classification. Besides, some parts of the notations are confusing and wrong. Specifically, in Eq.(1), the notation $N_{c_i = c_j}$  is confusing since what $i$ and $j$ come from? The similar issue also occurs for the notation $N_{c_i \\neq c_j}$ in Eq.(2).\n4. For the experiments, the hyper-parameters of the proposed method are selected on the test datasets during training and should be searched by another validation set. Moreover, this paper lacks ablation studies about the different parts of the proposed loss function.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "RctX1CUN0E", "reviewer": "Reviewer_xFz5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a loss function called ASLoss as an adjunct for classification loss to address the confusion issue under data insufficiency and class imbalance cases. The ASLoss can be decomposed into inner aggregation, outer separation and boundary constraint, which contraints the representation to have some invariant characteristics. The effectiveness is demonstrated by experiments.\n", "summary_of_the_review": "The authors ignored the work in metric learning e.g., contrastive learning and triplet loss, and proposed a loss function ASLoss which is very similar to those paper. I found the paper lack of novelty and the experimental results are not strong enough to support that this loss function is critic to this setup.", "strength_and_weaknesses": "Pros:\nThe authors implement the ASLoss with different model architectures and show the effectiveness of the method. \n\nCons:\nThe idea is exactly the same as metric learning loss. The three components, inner aggregation, outer seperation and boundary constraints are widely used in well-known paper like SimCLR and other contrastive learning paper or triplet loss paper. I do not see any references related to metric learning and the experiments in the paper are not strong enough to support that this method is very effective in the imbalanced learning or few-shot learning scenarios. To insert the ASLoss into classification problem is similar to the idea of supervised contrastive learning. I do not see enough novelty in this paper. \n\nSome other baseline models should be considered in imbalanced learning such as LDAM.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "0EZH_Xp-bPn", "reviewer": "Reviewer_nFtj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a new training loss that they claim to be helpful in data insufficient, class imbalance and unclear class evidence scenarios. The core idea is the newly created loss called ASLoss: it tries to maximize the similarities of samples belonging to the same classes; while minimize the similarities of samples across the classes. Despite the simple idea, the authors are able to obtain good results in Acne Severity Grading dataset and CelebA dataset.", "summary_of_the_review": "The main reasons I tend to reject are listed in the previous sections.", "strength_and_weaknesses": "The biggest strength of this paper is clarity, the idea and motivation is clearly stated and verified through experiments.\n\nThe biggest concerns are:\n1. Novelty: the idea seems to be not new. For instance in CosFace(https://arxiv.org/pdf/1801.09414.pdf) the author proposed to maximize the margin of Cosine distance.\n2. Experiments: the authors claim that it will benefit the models in extreme cases such as: data insufficient, class imbalance etc. But they only experimented two (unpopular) datasets: Acne Severity gradient and CelebA. Clearly, more experimental supports are needed. \n3. Unable to calibrate this method with other baselines. Due to unpopular datasets, it is almost impossible to directly compare this method with baselines. Although authors compare quite a few baselines in Table 3, those are not official numbers. For instance in CBLoss(Cui et al.), Long-Tailed CIFAR, iNaturalist, ImageNet are studied. But in this paper the author ignored all those datasets. Because of this, we are not able to make fair comparisons.\n4. (Minor) The formatting of this paper can be improved, many figures/plots are in low res.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject