Label Distribution Learning via Implicit Distribution Representation
Authors: Zhuoran Zheng
Xiuyi Jia
Venue: Submitted to ICLR 2023 (2023)
Keywords: label distribution learning
implicit distribution
Gaussian distribution
self-attention algorithm

Abstract: In contrast to multi-label learning, label distribution learning characterizes the polysemy of examples by a label distribution to represent richer semantics. In the learning process of label distribution, the training data is collected mainly by manual annotation or label enhancement algorithms to generate label distribution. Unfortunately, the complexity of the manual annotation task or the inaccuracy of the label enhancement algorithm leads to noise and uncertainty in the label distribution training set. To alleviate this problem, we introduce the implicit distribution in the label distribution learning framework to characterize the uncertainty of each label value. Specifically, we use deep implicit representation learning to construct a label distribution matrix with Gaussian prior constraints, where each row component corresponds to the distribution estimate of each label value, and this row component is constrained by a prior Gaussian distribution to moderate the noise and uncertainty interference of the label distribution dataset. Finally, each row component of the label distribution matrix is transformed into a standard label distribution form by using the self-attention algorithm. In addition, some approaches with regularization characteristics are conducted in the training phase to improve the performance of the model.

Under review as a conference paper at ICLR 2023
LABEL DISTRIBUTION LEARNING VIA IMPLICIT DIS- TRIBUTION REPRESENTATION
ABSTRACT
In contrast to multi-label learning, label distribution learning characterizes the polysemy of examples by a label distribution to represent richer semantics. In the learning process of label distribution, the training data is collected mainly by manual annotation or label enhancement algorithms generate label distribution. Unfortumately, the complexity of the manual annotation task or the inaccuracy of the label enhancement algorithm leads to noise and uncertainty in the label dis- tribution training set. To alleviate this problem, we introduce the implicit distri- bution in the label distribution learning framework to characterize the uncertainty of each label value. Specifically, we use deep implicit representation learning to construct label distribution matrix with Gaussian prior constraints, where each row component corresponds to the distribution estimate of each label value, and this row component is constrained by prior Gaussian distribution moderate the noise and uncertainty interference of the label distribution dataset. Finally, each row component of the label distribution matrix is transformed into standard label distribution form by using the self-attention algorithm. In addition, some approaches with regularization characteristics are conducted in the training phase to improve the performance of the model.
INTRODUCTION
Label distribution learning (LDL) Geng 2016)) is a novel learning paradigm that characterizes the polysemy of examples. In LDL, the relevance of each label to example is given by exact numerical value between and (also known as description degree), and the description degree of all labels forms distribution to fully characterize the polysemy of an example. Compared with traditional learning paradigms, L.DI is a more generalizable and representational learning paradigm that provides richer semantic information.
LDL has been successful in several application domains (Gao et al. (2018) ; Zhao et al. 202 : Chen al. (2021a): Si al.) (2022)). To obtain the label distribution for learning. there are mainly two ways: one is expert labeling. but labeling is expensive and there is no objective labeling criterion, and the resulting label distribution highly subjective and ambiguous. The other is convert multi-label dataset into label distribution dataset through label enhancement algorithm (Xu jet al. (2019) 2020); Zheng et al. (2021a); Zhao et al. (2022b)). However, label enhancement lacks reliable theory to ensure that the label distribution recovered from logical labels converges to the true label distribution, because logical labels provide a very loose solution space for the label distribution. making the solution less stable and less accurate.
In summary, the label distribution dataset used for training has a high probability of inaccuracy and uncertainty, which significantly limits the performance of LDL algorithms. To characterize and mitigate the uncertainty of the label distribution, we propose a novel LDL method based on the implicit label distribution representation. Our work is inspired by recent work on implicit neural representation in 2D image reconstruction (Sitzmann et al. (2020)). The key idea of implicit neural representation to represent an object as a function that maps sequence of coordinates to the corre- sponding signal, where the function is de- de-parameterized by deep neural network. In this paper, we start with a deep network to extract the latent features of input information. Then, the latent features are looked up against the encoded coordinate matrix generate a label distribution matrix (implicit distribution representation). Finally, the label distribution matrix is computed by self-attention module to yield a standard label distribution. Note that the goal of the proposed implicit distribution representation is to generate a label distribution matrix with Gaussian distribution constraints as a customized representation pattern.

---

Under review as a conference paper at ICLR 2023
Figure 1: Our architecture. This fig- ure shows the architecture of the pro- posed deep implicit function, which consists of two parts. The first part starts with a latent feature predic- tion stream (SNN with an MLP) that learns the input information to predict the feature maps. The second part learns a label distribution matrix to regress label distribution.
m
Inpul Feentures
SNN
Latel Distitanon Self-mmertion Mainis
Coordinates
GCN
To efficiently generate latent features around the coordinate, we design a deep spiking neural net- work (Yamazaki et al.] (2022) with an MLP as an executor to extract latent features in the input information. The architecture of the whole network consists of multiple layers of linear spiking neurons, and the neurons of different layers conduct a shortcut between them. Notably, spiking neural networks have two key properties that are different from the representation of artificial neu- ral networks. First, standard spiking neural network considers the time characteristies T (taking the image as an example, the input tensor X E XRTCCxWW W..,) at a multi-step inference mecha- nism. Here, create several pseudo-feature spaces the native feature space by setting different strategies with data augmentation (Ucar et al. (22I)). These deeuto-features and the native feature are stacked in the time dimension ' to achieve the multi-step inference mechanism. Second, the representation capability of the spiking neural network is underpowered, since the output space is a binarized sequence (f0 ((... 1}). Therefore, we place a standard MLP in the last layer of the spik- ing neural network, which projects the features into the real number space. Our model saves about 30~40% of energy consumption over ANNs with the same network structure on embedded devices such as lynxi HP300, Raspberry Pi, or Apple smartphones (PyTorcc 1.2 support M1 Mac).
The extraction of latent features provides material for the coordinates to generate the label distribu- tion matrix. First, the initialized coordinate matrix (the size is L x 64, where L denotes the number of labels, and 64 denotes features of the nodes) is reconstructed by using a GCN, Note that the fea- tures of the nodes meet the Gaussian distribution since the deep network needs to reconstruct the data from a fixed distribution. Then the coordinate matrix is repeated in N copies, and N denotes the number of samples. Next, the coordinate matrix computes label distribution matrix (the size is N x L x 2L) in the latent feature space by looking up the table]] Each component of the label dis- tribution matrix represents the distribution of each label value, and the components are constrained by priori Gaussian distributions. Finally, the label distribution matrix leverages self-attention mechanism to obtain the corresponding label distribution of the samples.
Yet, deep learning- based approaches are prone to overfitting manually extracted features. To allevi- ate the problem, we propose some regularization approaches to boost the performance of the model, and new dataset based on the image comprehension task is released. Our contribution includes: (i) For LDL, this is a novel method to obtain the label distribution of a sample through the implicit distribution representation. (ii) Spiking neural network with an MLP is developed to save energy consumption of mobile devices, and correlations between labels are deeply mined by a graph convo- lutional network. (iii) To the best of our knowledge, are the first to tackle the tabular LDL issue by using deep learning. Facing the LDL task, some regularization techniques are designed to boost the performance of the model and new LDL dataset is released.
2 BACKGROUND AND MOTIVATION
Starting in 2016, LDL. (Geng (016)) is officially proposed as novel learning paradigm that aims to inscribe the polysemy of a sample through the degree of description. Then, from the viewpoint of
task kinds, LDL is categorized into two domains. 1) Addressing the uncertainty of application tasks
( Gao et al. 2017): Ren & Geng (2017); Gao et al. 2018): Chen et al. (021a): Liu et al. (2021);
Zhao et al. (2021) Li et al. (2022) Si et al. (2022): Cao et al. (2022); Buisson al. (022) 2)
Studying the characteristics of label distributions on customized datasets ( GGegg (2016): Zhao &
Zhoul (2018): |Ren et al. 2019b a): Wang & Geng (2021); Jia et al. (2021a b): Zhao al. 2022a);
Tan et (2022). However, the existing work overlooks the fact that task also requires uncertainty
modeling for the label space. In this paper, we conduct uncertainty modeling on task 2 to boost the learning ability of the LDL algorithm. From a technical viewpoint, our approach has three
ht https://pytorch.orgg/ oriis's e/ nt'ntereed at ste/ 1 transformet www.mertuutrriieeeeeee butorial..hee
2

---

Under review as a conference paper at ICLR 2023
distinct motivations for uncertainty estimation. (i) To avoid the limitations of explicit modeling on the regression space, we introduce deep learning to handle task 2. where uncertainty is enforced on the representation process (label distribution matrix). By modeling the label space, the label distribution matrix provides a smoother estimation space as a feature map in the deep network. (ii) Compared to existing work ( (Ren al. (2019b); Jia al.] (2021a): Qian al. (2022) ) that considers label correlation, we use GCN to deeply mine label correlation. (iii) Besides that, SNN started
:: Kim & Panda (02T); Lian et al. (2022); Yamazaki et al. (2022)), to enhance computational efficiecyy and maintain modeling capabiliiiis, thus introducing handle task 2. Overall, our approach tackles a new problem, while the technical approach is the first of its kind.
to show amazing potential on regression tasks (Ahmadi et al. (202I); Patel et al. (2021);
3 PROPOSED METHOD
As shown in Figure 1 our architecture is two-stage scheme, which is based implicit repre- sentation approach. In the first stage (latent feature extraction). the raw features X' are fed into the encoder for encoding to construct latent feature space Fy E RLxHx In the second stage (label distribution matrix learning), the coordinate tensor (coordinate matrix is reshaped after GCN) C E Lxxxx2 is looked up in the latent feature space Fy to generate a label distribution matrix M E Lx22L and finally a label distribution D obtained by using self-attention algorithm. The dimension 2 of the coordinate tensor comes from a weak assumption that the label relation exists in two-dimensional plane, such as the relative positions of "tree" and "sun" " in an image. Furthermore, some regularization terms are introduced to our model boost its performance.
Latent Feature Extraction. Our model starts with a latent feature extraction task, which is based on spiking neural network. Spiking neural networks have high potential value as 3rd deep model and are efficient and interpretable. In the LDL task, we develop a non-fixed Network Architecture and a Network Implementation due to the diversity of LDL dataset formats.
Network Architecture: Our network consists of 17 layers of function units, which include 8 linear units, 8 nonlinear units, and a transformation layer (including an MI.P, a mean operator, and a reshape operator). Essentially, this is residual network and the transformation layer at the tail of the network. Except for the first layer (the number of neurons is the dimensionality of the input features), each linear unit contains 1024 neurons, followed closely by ReLU non-linearity, and the last layer is a reshape function to generate the feature space Fy. On datasets with a smaller feature space X (e.g., Gene dataset has only 36 features), each linear layer includes 64 neurons. We also attempted to use other activation functions than ReLU, such as PReLU, Swish, and Sigmoid, but without any advantage. Besides, since the spiking network involves time dimension, the time dimension T is squeezed in the output space of the spiking network by using a mean operator. The network's 's tailgate should notice that the feature map is reshaped as F' using a reshape operator and the MLP (the number of neurons is L X W x H.),, where H and W are 32.
Network Implementations: The simplified residual network is implemented as an ANN in two frameworks: PyToorc 1.12 (a standard deep network training library) and SpikingJelly 0.0.13 (a library for deep learning applications that part of the PyTorch ecosystem). The SpikingJelly implementation can be run in spiking or non-spiking modes. To train the spiking network, we use an ANN-SNN conversion training approach. The conversion is simple, we convert the trained model Converter (SpikingJelly) to obtain SNN model. Note
on PyTorch with the help of an that the conversion method defines six modes (max, 99.9 99.9%. 1.0², 2. 1.0),, 1.0 1.0%,5 to obtain SNN with different accuracy, and in this paper, chose 99.9%. The literature (Patel et al. (021 provides a theoretical basis for analyzing the conversion of ANN to SNN.
Learning Label Distribution Matrix. Inspired by confident learning (Northcutt et al. (202 b), we develop a label distribution matrix with Gaussian before estimating the uncertainty of the labels. For the data distribution of each label value, we use a Gaussian distribution delimit the distribution rather than other multi-peaked distribution priors, and numerous kinds of literature have verified that this approach can eliminate the uncertainty (Liu et al. (022)); Zheng et al.) (202Tb); Ghosh et al. (202T) Li et al. (2022). Note that since the label distribution values are in the range of 0 to 1, we also constrain the vector after Gaussian sampling to be in the range of 0 to 1. Furthermore, to capture the global correlation between labels to generate standard label distribution, we employ a self- attention mechanism to model the label distribution matrix. To obtain a label distribution matrix M, need latent feature F, from the SNN and coordinate matrix C. GCN(C,) (GCN denotes the graph convolution network) that passes through the graph convolution network. Specifically, first, initialize a matrix of coordinates C, E R Lxx11 based on the functions (torch.randn) provided by
3

---

Under review as a conference paper at ICLR 2023
PyTorch. L denotes the number of nodes and 64 denotes that assign one feature vector each node, with each feature vector sampled Gaussian distribution. To build a graph structure the data needs information about the edges. where there is edge with direction between the nodes. So far, we built the node with the edge information and have the ability to integrate it into graph to input into the GCN. Our GCN includes four graph convolution layers and four activation layers, where the activation function uses ReLU. These four graph convolutions include 64, 128, 256, and L neurons respectively. There is one key message note, the output matrix C. of the GCN is filled (torch.repeat) with the same number of samples the latent feature space Fy. Then, we use C2 look up the table in F, to obtain a matrix by using flatten operations. The L label distribution values include 1 x 2L vector to form label distribution matrix M. Each vector is constrained by a designated Gaussian distribution M with parameters whose mean is the value of the label distribution and variance of 0.5. Finally, this matrix is squeezed by using self-attention algorithm Vasswani et al. 2017)) to obtain the corresponding label distribution for the samples. For self-attention algorithm,
treat each vector in the label distribution matrix token by using the scaled dot product to obtain a global correlation matrix, and finally an MLP with Softmax to squeeze the matrix for standard label distribution. As shown in Figure 2 extract the heat map calculated by the self-attention algorithm, heat map of the model output before the self-attention algorithm and the heat map of label space Movie dataset. Overall, the label correlations of these heat maps tend to be consistent.
Lf
LOOEE on Movvo
08
08
-04
04
02
82
(a) Raw label space
(b) Self-attention algorithm
(c) w/o Self-attention algorithm
Figure 2: This figure shows the label correlation the Movie dataset. Regularization Techniques. We propose two techniques (Linear Normalization Function, Data Augmentation) to boost the performance of our model.
Linear Normalization Function: Currently, most existing LDL algorithms use Softmax to output vector in the tail of the model.
Softmax(d33 (d"))
where d denotes label value in the vector and K denotes the number of elements. Since the complexity of exponential operations much higher than linear operations, especially in the train- ing phase of the model (e.g. Gamma correction (Ju al. (2019))). Several algorithms (such as ForFormer (Qin et al. (2022)) seek to replace exponential operations. Here, we develop a linear function Lnf replace Softmax to output a label distribution vector. Then predicted label distribu- tion d; d d ...., de can be obtained by:
dh + [Dmin]
Lnf(d
>e (det 1Pminl)
where [Dminl denotes the absolute value of the minimum of the predicted label distribution values. Essentially, our method offsets the values of the label distribution to the positive domain of the x- axis. We evaluate both Softmax and linear normalization function 12 datasets, and our network with the linear normalization function converges 2.1 faster than the model with Softmax.
Data Augmentation: Most LDL datasets are difficult to augment the samples with expert knowl- edge since the input features are extracted manually. To overcome this challenge, we introduce simple and tata-agnostic data augmentation routine, termed mixup (Zhang et al. (018)). In nutshell, mixup constructs virtual training examples:
T Xo; + A)æy, where æ; input vectors
(3)
ÿ = Ad; + (1 - A)d,, where d,, d, are label distribution vectors
4

---

Under review as a conference paper at ICLR 2023
(Iz, 3/5) and (Iz, yj) are two examples drawn at random from our training data, and À [0, 1]. However, a deep network may still be over-fitted during the training phase, which leads to poor generalization. We use the random masking scheme endowed on mixup, formally expressed as:
[0.
I = λr; * mask + (1 - A)z5 * mask, where Ti, 2 are raw input vectors y Lnf(Add + (1- - A)d,), where d,, d, are label distribution vectors
(4)
the mask is a vector represented by O. In this paper, mask contains 80% of the scalar 1 and the rest is O. Moreover, to solidify the definition of the label distribution, we use Lnf to normalize the synthesized label vectors. We also tried using other regularization schemes, such as random cropping, and the results do not improve significantly.
Loss Function. We optimize the weights and biases of the proposed network by minimizing the I,, Kullback Leibler (K-I.) divergence, and regularization of the label distribution matrix the training set,
N Σ d d. ACa d.) + Σ Σ La(M N
(5)
where N the number of training samples, d, is the label distribution vector by our model, and d; is the corresponding ground truth. The weight À of the loss term Lu (K-L) is set to 0.01 in our experiments. In addition. the weight 3 of the loss term (label distribution matrix) set to 0.1 in our experiments. Since it is difficult to capture useful information by encoding small number of tokens, encoding for sequences of long tokens can extract higher-order semantics, such as perceptual loss (Johnson et al. (216))) Therefor, we used the above- mentionet learning strategy with several labels less than 20. On the datasets with the number of labels greater than 20, we used L1, K-L divergence, perceptual loss, and regularization of the label distribution matrix the training set,
the datasets
N d di + Lui da) + A2Gp 3
N
L
C =
La(Mnt,
(6)
1=1
1= 1=1
we use MLPs as the pre-trained model for the perceptual loss function (C,)). The weight Ay. A2. and are set to 0.01, 0.01, 0.08 in our experiments, respectively. These MLPs consist of three linear layers with fixed neurons and three activation layers (ReLU), where the number of neurons is the same as the number of labels, using Kaiming initialization (He et al. (0155).
EXPERIMENTS
Algorithm Configurations. We conduct experiments on 12 datasets, the characteristics of the datasets are reported in Table Except for dataset we-LDL, the configuration of all other datasets is referenced to (Wang & Geng (2021)). This new release dataset (wc-LDL) has 500 watercolor im- ages and corresponding label distribution (12 emotions). We develop the SNN with configurations on different datasets also summarized in Table 5 To evaluate the performance of LDL models, we use the six metrics proposed by (Geng (2016)) including Chebyshev distance J. Clark distance J. Canberra distance J, KL divergence L, Cosine similarity 1, and Intersection similarity 1. LF and DA denote the loss function and data augmentation method, respectively. + represents the indicator's performance favoring low values and 1 represents the indicator's performance favoring high values. Li denotes the number of neurons in the head layer and the rest of the layers of the SNN.
Experimental Setting. We conduct comparative experiments with five LDL algorithms (BFGS- LLD (Geng (2066)) LDL-LRR (Jia et al. 20) LDL-LCLR (Ren et al. (2019)), LDLSF (Ren) et al. (2019a)) and LALOT (Zhao & Zhou (2018)) which also used data augmentation schemes on datasets. Furthermore, to validate the effectiveness of our proposed method, we design a baseline model to be conducted on 12 datasets. For the baseline model (a-LDL), we drop the implicit representation and use only a two-layer network with the label distribution matrix to learn a label distribution. a-LDL without any additional regularization terms and data augmentation mechanisms in the training stage. BFGS-LLD is based on a linear model, the loss function K-L. divergence, and the optimization method is the quasi-Newton approach. LDL-LRR and LDL-LCLR both consider label correlations in the learning process, with the former considering the order relationship of the labels and the latter capturing global relationships between labels. For LDL-LRR, the parameters and / are selected from 1...,,,......---- 12-1) and -0,--3.2,...... respectively. For LDL-L.CLR, the
5

---

Under review as a conference paper at ICLR 2023
parameters A1. >2. A3. X1 and i are set to 0.0001, 0.001, 0.001, 0.001 and 4, respectively. LDLSF leverages abbel-specific features and common features simultaneously, whose parameters }1, 22 and A are selected from 106-6,-6,...-2---1 2,-11, respectively, and p is set to 10- 3 L.ALOT adopts optimal transport distance as the loss function, the trade-off' parameter C' and the regularization coefficient À is set to 200 and 0.2, respectively, Our approach for experimental settings is reported in Table It is worth noting that Early stopping and Greed soup (Wortsman et al. (2022) are also used on all the datasets where the comparison algorithm is executed. Our method is marked as gray
Datasct
Algoridlu
Chebyshev
Clark 1
Canberaa
K-L L
Cosinc
Interscction 1
Ours
0.0779
0.0021
0.3980
0.0051
0.7779 0000300
0,4040 +: 0.0020
0.9883
0.0009
0.0014
a-1.M
0.0835 + 0.0054
0.4667
0.0052
0.8077 + -0.0110
0.4455 + -0.0033
0.978% + 0.0079
0,8705 = -0.079
LDL-LRR
0.1122
0.0030
0,1772
0.0036
0,8802
1: 0.0024
0.5533 1: -0.0049
0.9510
0,0022
0.8555
=0.0017
we-LDL
LDL-LCLR
0.1057
LO569
0.0039
0.0037
0,9668
0.0049
0.8383
0.0018
SP
0,1009-
0.0038
0.4199
0.0044
0.9008
+ 0.0015
0,5199 + 0.0040
0.9777
0.0018
0.8660
0.0022
LALOT
0.0989
0.0019
0.6689
0.0019
0,8089
+ 0.0049
0,4778 +: 0.0018
0,9476
0,0020
0,8700
0.0033
BFGS-LLD
0.1229
0.0039
0.0021
0.7998
0.0020
0.4998 4: 0.0051
0.9704
0.0036
0.0016
Ours
0.0854 + 0.0018
D.4008
0.0018)
0.7955 + 0.0023
0,4100 + 0.0012
0.9799 + 0.0014
0.8809 = 0.0015
a-LDL
0.0899
0.01
0.4189
0.0123
0,8019 + 0.0023
0,4131 + 0.0012
0,9702 0.0014
0,8702
0.0015
LDL-LRR
0,1122 0.0030
0.4772
0.0036
0,8802 0.0024
0.0049
0,9510
0.0047
SJAFHE
LDL-LCLR
LDLSF
0.1123 0.0038
0.1057 + 0.0019
L,069-
0.4597
0.0014
0.0039
0,7890 1: 0.0039
0.9212 + 0.0015
0.5045 1: 0.0037
0.5557 +: 0.0040
0.9668
0,9779
0.0049
0.0018
0.8353 = 0.0018
0,8660
0.0022
LALOT
0,0089
0.6689
0,8099
0.0018
0,9476
0,0053
BFGS-LLD
0.1334 + 0.0139
L66488
0.0023
0.7999 1 0.0022
0,04778
0.0051
0.9711
0.0036
0.8655 = 0.0116
Ours
0 + 0.0020
0.3994
0.0010
0 + 0.0020
0,3650 + 0.0014
0 + 0.0015
0,8900 - 0.0017
LLDL
0,0954
0.0010
0,7774 0.0083
0.0014
0.9788
0,0010
LDL-LRR
0.1109 + 0.0033
0.4477
0.0039
0.666 1: 0.0026
0.5344 1: 0.0028
0.9397
0.0029
0.8592 = 0.0033
SBU
LDL-LCLR
LDLSF
0,1117 0.0048
0.1100 = 0.0025
0,9660
0.4199
0.0044
0.0039
0,9013 1 0.0015
0.7897 : 0.0033
0,5199
0,5101 : 0.0021
1 0.0040
0.9780
0.9677 1 0.0056
0,0029
0.8555
0.8622
0.0032
0.0022
LALOTT
0.0889
0.0019
9.669
0.0019
0.8421 0.0049
0,4766 0.00688
0,9476
0.0020
0,8700
0.0033
BPGS-LLD
0,1119 = 0.0030
1.4657=
0.0022
0.7700 = 0.0025
0.04932
+ 0.0053
0.0036 0.9753
0
0.0019
Ours
0,2998 0.0020
2.3374
0.0018
6,5163 = 0.0018
0,8111 :: 0.0029
0.7890
0.0049
0.5691
0.0010
a-LDL
0.3111 0.0046
2.3881
0.0043
66189 + 0.0066
08111 + 0.0021
0.7516
0.0049
0.5000
0.01 11
LDL-LRR
0,3889 = 0.0111
3.1698
0.0031
6,8777 : 0.0025
0.8999 1 0.0069
0,7014 + 0.0077
0,5441
= 0.0049
Scenc
LDL-LCLR
LDLSF
0.3441
0.3740 0.0066
2.9884
2.4986
0.0066
0.0055
6.6900
66800
0.0055
0.0067
0.8559 1: 0.0039
1: 0.0044
0,7336
0.71 0,0122
0.5660
0.5111
0.0081
0.0041
LALOT
0.3129
+ 0.01552
2.3999
0.0044
6.666
+ 0.0078
0.8226
+ 0.0033
0.7390
0.0100
0.5224
0.0066
BEGS-LLD
0,3598
0.0020
2.4998
0.0033
6,7999
+ 0.0049
0,8400
+ 0.0033
0,7353
0.0064
0,5199
0.0055
Ours
0.0488
2.1029
0.0259
14.0888
0.0551
1 0,0044
0.8395
0.7984
0.0066
a-LDI.
0.0502
0.0032
2.1777
0.0211
14.1221 + 0.0413
0.2443 + 0.0045
0.829%
0.0022
0.7889 + 0.006
LDL-LRR
0.0039 0.0537
2,2887
0.0860
14,3550
0.0144
0,2559
+ 0.0077
0,8288
0,0144
0.7789
0.0040
Gene
LDL-LCLR
LDL.S
0.0511
0
0.0030
2.2201
2.2221 0.0036
0.0444
142101
14.3667
+ 0.0265
0.0510
0.2445 + 0.0077
0,2566 1 0,0047
0.8302
0.8320
0.0010
0.7722
0.7201
0.0026
0,0060
LALOT
0.0505
0,0033
2. 19899
0.0194
14,1855
0,0922
0,2443
+ 0 0.0088
0.8297
0,0060
0,7888
0.0013
BFGS-LLD
0.00%6 0.0578
2.3008
0.0188
14,3559
0556
0,2480
0.0015
0
0.77%
0.0000
Ours
0,1089 = 0.0018
0,5001
0.0064
0.9722 1 0.0010
0.0977 1 0.0008
0,9485 1: 0.0061
0.8602
= 0.0006
LLDL
0.1121 = 0.0023
0,5199=
0.0098
LO549 : 0.0043
0.114] 4 0.0031
0.9477 + 0.0063
0.8566
0.0046
LDL-LRR
0,1135 0.000N
0.5244-
1
0.0061
0,1445 + 0.0049
0
0.0002
0.00077
Movic
LDL-LCLR
LDLSF
0.1155 + 0.0045
0.1177 = 0.0086
0.5345
0,5339
0.0040
0.0062
L1533 : 0.0111
L11122 0.0050
0.1559 1 0.0030
0.1540 1: 0.0041
0,9360 0.0049
0.9445 1 0.0020
0.8222 = 0.0011
0.8551
0.0044
LALOT
0,1221 0.01
D.5440
0.0033
1,1110
0.0040
0.1503 + 0.0008
0,9477
0.0022
0.00012
BFGS-LLD
0.1310 = 0.0032
0.5230
0.0022
1.1170
: 0.0024
0.1595 : 0.0155
0,9400 0.0003
0.8491 = 0.0018
Ours
0.3763 + 0.0022
1,1560
0.0102
2.0889 : 0.0055
0,4880 # 0.0023
0,7998 1: 0.0022
0.6703 = 0.0033
sLDL
0,3810
0.0143
2.1222
+
0.0039
0.0066
LDI-LRRR
0.3993
1,4990
0.0166
2.1884
0.0034
0.5246 + 0.0006
0.2531
0.0023
8.6334
0.0077
M2B
LDL-LCLR
LDLSF
0.4159 0.0055
0.40%0 = 0.0082
1,2444 = 0.0045
1.3105 0.0041
2.2155
2,2000
= 0.0009
0.0076
0.4996 :: 0.0013
0,5002 1 0 0.0006
0,7552 0,0004
0.7760 : 0. 0.0079
0.6234 0.0033
0.6555 = 0.0012
LALOT
0.3881
0.0099
1.4883- 0.0012
2.1257
0.0268
0.4990
+ 0.0008
0,7549
0.0021
8.6620 0.0053
BPGS-LLD
0 + 0.0044
1. 1,3650
0.0002
2.1992
+ 0.0095
0.4995
: 0.0005
0,7699 + 0.0040
0.6532 = 0.0009
Ours
0,3895 0.0021
1,2140 0.0111
2.1995 1 0.0085
0 1 0.0030
0.6990
0,0002
0.6304
0.0001
0,3922
1,2149 0.0133
2.2002-
0.0022
0.499) + 0.0006
0,6800
0.0032
8.6466
0.0009
LDL-LRR
0.4159 = 0.0010
166600
0.0122
2.2006
: 0.0039
0,5388 :: 0.0006
0.6531 + 0.0023
0,5804
= 0.0007
SCUT
LDL-LCUR
0.4240 + 0.0042
0.0055 1. 1,3444
2.2450++
0.00166
0,5131 + 0. 0.0022
0.6261 0.0005
0.5500 = 0.0012
LDLSF
0,4360
L2185=
2.2159 1 0.0076
1 0.0006
0.6261
0.0004
0.0030
LALOT
0.3999 :: 0.0009
1,4933 0.0012
2,2207
: 0.0158
0.4995 1: 0.0002
0.6549
0.0020
0.6411 = 0.0044
BRGS-LLD
O. 0.3992 0.005S
1,5656 + 0.0163
2.2832 + 0.0080
0.4966 + 0.0011
0.66991
0.0040
0.6333
0.0013
Ours
0.1251
2,0980 1 0.0223
1 0.0009
0.9643
0.0015
0.8501
0.0025
zLDDL
0.1272 = 0.0023
1.1990
0.0111
2.1008 + 0.0244
0.1102 +: 0.0022
0.9600
0.0033
0.8499 = 0.0034
LDL-LRR
0.1313 + 0.0031
1,2519
0.0058
2.1992 + D.0095
0.1127 + 0.0977
0,9533
0.0021
0.8412 = 0.0066
1hp5500
LDL-LCLR
0.1277 0.0016
L.1999
2.1194
0.1135 1 0.0006
0.9588
0.0044
0.0014
LDLSF
0.1270 :: 0.0028
1.1909
0.0164
2.1846 : 0.0119
0.1193 4: 0.0041
0.9609
0.0019
0.8450 = 0.0007
LALOT
0.1306 :: 0.0022
L1921
0.0015
2.1111 1 0.0171
0.1120 1: 0.0015
0.9430
1: 0.0019
0.5400
0.0004
BRGS-LLD
0.1 0.0049
1.4655 0.0041
2.1675
0.0024
0.1135 + 0.0055
0.9595 0.0030
0.8419 0.0018
Table 1: The performance of our proposed method with the comparison algorithms on 12 datasets.
Results and Discussion. We conduct 10 times 5-fold cross-validatio on each dataset. The exper- imental results are presented in the form of tst" in Tables | and 7 Overall, our proposed method outperforms other comparison algorithms in all evaluation metrics. Each comparison algo-
6

---

Under review as a conference paper at ICLR 2023
rithm employs some regularization techniques to expand the training sample as well as to prevent overfitting, however, four main factors contribute to the competitive results of our approach. i): Moderate noise, especially the Gene dataset, due to the uncertainty that comes with manual an- notation, our approach has a huge performance gain with the help of implicit distribution representa- tion with Gaussian priors. ii): The ability to capture global features between labels with the help of a self-attention mechanism. Also, a- a-LDL performs sub-optimally probably because the depth of the model is insufficient. iif): The powerful representational capabilities of the deep network, especially on image datasets, give us huge advantage.
AS
Algorithm
Chebyshev
Clark
Canberra
KLLL
Cosite
Intersection
Datuset
(s)
wlo C
0,0888
0.6997
0.0012
0.0021
2,3029
2.1664 0.0177
0,0259
14,0888 0,0551
14,3651
0.0155
0,2338 & 0.0044
0,3488 0.0071
0,8398
0,8200
0.0033
0.074
0.7884
0,7984 0,0066
Gese
Cluss
0.7779 = 0.0221
0.3980 + 0.0011
0.7779 # 0:0000
0,04000 # 0.0021
0,9833
0.0009
018778 #: 0.0014
(be.4)
wio PRI
SNN
0,771
QUW77
0.0012
0.80P)
0,4066
0,4008 1.043
0.0047
0.7581
0.7505
:: 0.0011
+ 0.0014
0,04118
0.04223
0.0016
0,3010
0.9779
0.9801
=0.0012
QUR0KS
0.8664 0.00022
U.3699 0.0012
wesLDL
GNN
0.0804 0.0033
0,4133
0.0011
0.7968 1. 0.0020
0,06991
0,0052
0,9708
0.0036
0,3612 0.0015
Table 2: Ablation study (AS). Effectiveness of the loss functions and the modules on two datasets. Quantitative results demonstrate the effectiveness of each module.
Ablation Study. To demonstrate the effectiveness of the loss function and the module of our model, conduct ablation study involving the following four experiments: (a) w/o perceptual loss function Cp: we remove the loss function on training Gene dataset, shown in Table (b) w/o our proposed regularization techniques (PRT): we remove the linear normalization function and the data augmentation respectively on the training Wc-LDL dataset, shown in Table (c) The effectiveness of SNN: we use standard MLPs to replace SNNs in the same network architecture, shown in Table 2 (d) The effectiveness of GNN: for deep implicit function construction, we use standard MLPs to replace GNNs, as shown in T on 12 data sets with our proposed model, and the results show that using mixup with mask improve overall performance by 6% over using mixup. We conduct 10 times 5-fold cross-validation on the dataset of the ablation experiment.
(e) mixup with mask vs. mixup: we evaluate these two methods
Potential of Model and Network. Our model can be extended to handle classification and semi- supervised tasks. We conduct some experiment reports to show the potential of the public datasets. In addition, there a deep model (DLDL (Gao et al. (2017)) based on a label distribution learning framework training scheme being performed several classification tasks. Since DLDL does not have an adapted network implementation on tabular data, to demonstrate the effectiveness of our proposed method, approach battles with the DLDL algorithm in the age estimation dataset. (1) Facial expression recognition (Zhao et al. 2021)) )): To evaluate the effectiveness of the proposed model, we conduct the experiments on the public in-the-wild facial expression datasets (RAF-DB, CAER-S, AffectNet). For images on all the datasets, the face region is detected and aligned by using Retinaface (Deng et al. (2020)) Then, the image is cropped to a fixed resolution (224 x 224) by bi- linear interpolation. Our approach is pre-trained on the face recognition dataset MS-Celeb-1M 1M, and the 50-layer Residual Network is used as the backbone network. For our network, parameters were optimized using the Adam optimizer with an initial learning rate of 0.01, a mini-batch size of 128, and an epoch is 50. Notably, our network architecture uses a model (both input and output layers changed in the number of neurons and the rest are fixed) executed on the Gene dataset with single RTX GPU, and the loss function uses cross-entropy. We compared it with the current SOTA model (Face2Exp) three popular datasets. The recognition accuracy of Face2Exp in the three data sets (RAF-DB, CAER-S, AffectNet) is 88.54%, 86.16% and 64. 23% respectively. The recognition accuracy of our proposed model in three datasets is 88.52%. 86.36%, and 65.02% respectively. Our algorithm achieves competitive results on most of the face recognition datasets. (2) MedM- NIST Classification Decathlon (Yang et al. (202T)): We evaluate the algorithm's 's performance on the MedMNIST Classification Decathlon benchmark. The area under the ROC curve (AUC) and Accu- racy (ACC) used as the evaluation metrics. Our approach is pre-trained on the Gene dataset, the 18-layer Residual Network is adopted as the backbone network. Our model is trained for 100 epochs, using a cross-entropy loss and an Adam optimizer with a batch size of 128 and an initial learning rate of 1 x 10 The overall performance of the methods is reported in Table 3 Our model is pre- trained during the training phase and thus achieves competitive performance on most of the datasets. Literature (Yang al. (202 1)) includes the bibliography of the full comparison method. (3) CIFAR- 10, SVHN, CIFAR- 100 (Hu et al. (202T)): Since our approach involves graph structure, it battles with other SOTAs the semi-supervised task. We use ResNet-28-2 as our backbone and Adam with
7

---

Under review as a conference paper at ICLR 2023
Methods
PatMMNIS
CherMNIST ACC
DemaMNIST
OCTMNIST
PneumonisMNIST
AUC
AUIC
ACC
ResNet-18 18 (28)
0,972
0.844
0.706
0.947
0.899
0.721
0.951
0.75K
0.957
0.843
ResNee-18 (224)
0:978
0.948
0.896
0.960
0.752
ResNet 50 (28)
0.979
0.864
0.692
0.947
0.886
0.710
0.939
0.745
0.949
0.857
ResNet- 50 (224)
0.978
0.848
0.706
0.947
0.895
0.719
0.951
0.750
0.965
0.896
skleun
0.500
0.642
0.906
0.883
0.595
0.547
AutoKeras
0.979
0.921
0.736
Google AutoMI. Vision
0.962
0.811
0.718
0.947
0.925
0.766
0.965
0.732
0.993
0.941
Ours
0,985
0.864
0.722
0.948
0.929
0.765
0.969
0.756
0.992
0,94
Methods
AUC
ReinsMNIST
ACC
AUC
BreseMNIST
ACCC
AUC
OrgarMNIST.A
ACC
OrgwMNiSTT.C
AUC
ACC
ALC
ACC
ResNee- 18 (28)
0.727
0.515
0.597
0.859
0.995
0.921
0.990
0.389
0.967
0,762
RasNet 18 (22.)
0.721
0.543
0.915
0.876
0.997
0.931
0.991
0.907
0.974
0.777
ResNet-50 (28)
0.719
0.853
0.995
0.893
ResNet 50 (224)
0.217
0.555
0.563
0.833
0.997
0.931
0.992
0.898
0.97
0.770
auto skleam
0.694
0.525
0.348
0.508
0.797
0.563
0.896
0.676
0.855
0.601
AutoNerzs
0,655
0.301
0.996
0.992
Google AutoML Vision
0,762
0.988
0.986
0.361
0.706
Ours
0,760
0.570
0.931
0.890
0.996
0.933
0.990
0.917
0.971
0.810
Table 3: Overall performance of MedMNIST in metrics of AUC and ACC, using R8sNet-18 /
ResNet-50 with resolution 28 and 224, auto-skleam, AutoKeras and Google AutoML Vision.
Matheds
1000 labels
CIFAR-I0
4000 lobels
1000 Inbels
SVHN
4000 lahols
10000 labels
CIFAR-100
55000 tahels
Backhone
VAT
81.36
$8.95
94.02
95.80
77.54
81.55
WRN-28-8
MeanTeachhet
K2.68
89.00 64
96.25
96.61
72.68
79.62
WKN-28-8 28-8
MicMitch
92:25
76
79.13
WRN-28-8
ReMacManch
94.27
97.17
97.58
76.97
WRN-28-8
FixMarch
95.09
97.64
77.40
64.25
WKN-28-8
SimPLF
94.84
94.95
97.54
97.31
78.11
H2.40
WRN-28-8
Ours
95.99
97.62
78.64
81,002
WRN-28-8
Table 4: CIFAR-10, CIFAR-100, and SVHN Top-1 test accuracy.
weight decay for optimization in all experiments. Our model is trained for 100 epochs, using a cross- entropy loss with a batch size of 128 and initial learning rate of x 10-2 Early-stopping and data augmentation (Mixup) are also adopted in the training phase. Two datasets (CIFAR-10 10, and SVHN) are used for us to evaluate the performance of the algorithm. The overall performance of the methods is reported in Table Our model still selects MIPs pre-trained on the Gene dataset and performs competitively on the semi-supervised task. Literature (Hu et al. (202T)) includes the bibliography of the full comparison method. (4) Semi-supervised label distribution learning (Jia et al. (2021b)): To evaluate the capability of our model, our proposed algorithm is compared to the SOTA model (PGE- SLDL (Jia et al. (2021b)) in the Gene with 50% missing rate. We conduct it 10 times on Gene and average the 10 results as the final result. PGE-SLDL yield 0.0059 00000,, 0,4298+000006, 0.8911=0.000 and 0.8789+00.007 respectively on the data set with 6 evaluation metrics (Chebyshev Clark4, Canberrat, K-L divergence., Intersection"" Cosinet). Our method yield 0.0053 0.4112++ 4.2990 0.09661 0.00055 0.0001 and 0.8797 0000 respectively on the data set with corresponding 6 evaluation metrics. Our method still shows competitiveness, where the model remains fixed, the learning rate and the number of iterations are not changed, and the X of Equation 6 is adjusted to 0.25, (5) Our approach vs. DLDL (Age estimation (Gao al. (2017)): Two age estimation datasets are used in our experiments. The first is Morph (Ricanek & Tesafaye (2006), which is one of the largest publicly available age dltasetss. The second dataset is from the apparent age estimation competi- tion, the first competition track of the ICCV Chalearn 2015 workshop (Escalera et al. )2015). We employ the DPM model to detect the main facial region. Then, the detected face is fed into a cascaded convolution network to get the five facial key points, including the left/rght eye centers, nose tip, and left/right mouth corners. Finally, based on these facial points, we align the face to the upright pose. The implementation details of our method with the preprocessing unit are referenced in DLDL. We used MAE and c-error to evaluate the performance of our model and the comparison method, respectively, DLDL algorithm on Morph and ChaLearn with metrics MAE J and e-error yields performance that is {2.42, 0.23; 3.51, 0.31J. Our algorithm achieves competitive results (f2.40, 0.20; 3.23, 0.273).
0.0890=000005, 0.09788=0.0005,
Visualize of Label Distribution Matrix. We conduct an experiment to verify the validity of our algorithm. We extract one sample (including features and labeled distributions) on Movie, and then
8

---

Under review as a conference paper at ICLR 2023
visualize the matrix of label distribution (5 10) learned by the network and the corresponding label distribution. As shown in the Figure 3 we notice that the data distribution (means and variances) of the label distribution vector (each row of the label distribution matrix) is consistent with the corresponding label distribution.
200
26
1.19
1.9
10
6/6
60
600
(a) L1 of the matrix
(b) L2 of the matrix
(c) L3 of the matrix
Diebs dediuten
08
31 02
-22
80
92
09 18 -05
65
13
L1 13
(d) L4 matrix
(6) L5 matrix
(f) Label distribation
Figure 3: This figure shows the distribution of data for each vector of the label distribution matrix and the corresponding label for this label distribution matrix.
APPLICATION AND ENERGY CONSUMPTION
We select the algorithm executed Gene the evaluation model. The evaluation model is con- ducted on three platforms (lynxi HP300, Raspberry Pi, and Apple smartphones) to check the energy consumption with the same number of iterations (the power is evaluated thanks to the adb script [Dzhagaryan et al. (2)16))). Specifically, our model is trained on Gene with the accuracy of float 16. Then, the baseline model (MLP replaces SNN and GNN) and our model run inference on a mobile platform, each model executing 500 epochs with 32 batch sizes in the iteration. Experimental results show that our algorithm saves 34.6%, 41. 1.2%, and 40.8% energy compared to the baseline algorithm on the three platforms, respectively.
6 RELATED WORKS
Label distribution Learning. Label distribution learning has attracted several attention new learning paradigm. Label distribution learning comes from the scheme proposed by (Geng (2016) to address the age estimation task. Since then large number of approaches have been proposed, such as low-rank hypothesis-based (Jia et al. (2019); Ren et al. (2019b)), metric-based (Gao et al. (2018). manifold-based (Wang & Geng (202T)) h. and label correlation-based TTegg & Jia (2021) Qian et al. (2022)). Among them, some approaches are executed computer vision (Chen et al.
(2021a)), and speech recognition (Si et al. 2022)) ) tasks improve the performance of classifiers. In this paper, we try to build a distribution of label distributions to moderate noise and uncertainty.
Implicit Neural Representations. In implicit neural representation, an object is usually represented as a multi-layer perception (MLP) that maps coordinates to a signal. This idea has been widely applied in modeling 3D object shapes (Lin al. (2020); Kohli et al. (2020)). 3D surfaces of the scene (Sitzmann et al. (2019); Yariy et al. (2020;; Niemeyer et al. (2020); Jiang et al.) (2020)), the appearance of the structure as well as 2D image enhancement (Skorokhodov al. (202T); Chen et al.) (2021b); Anokhin et al. (2021): Karras et al. (202T)). In this paper, we seek to explore this technique to address the label distribution learning issue.
7 CONCLUSION
In this paper, we design an implicit distribution representation algorithm to moderate the uncertainty of the label values, where the implicit function can be a good estimate of the continuous distribu- tion space. Furthermore, Gaussian prior methods and self-attention mechanisms help the model leam both local signals and global information of the label distribution matrix. Numerous experi- ments have verified the effectiveness of our approach as well as the suitability the regularization technique. The application session demonstrates the high efficiency of our proposed model.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Mohsen Ahmadi, Abhas Sharifi, Shayan Hassantabar, and Saman Enayati. Qais-dsnn: tumor area segmentation of mri image with optimized quantum matched-filter technique and deep spiking neural network. BioMed Research International, 2021.
Ivan Anokhin, Kirill Demochkin, Taras Khakhulin, Gleb Sterkin, Victor Lempitsky, and Denis Ko- rzhenkov. Image generators with ceeiitionalll-indeeeeeeee pixel synthesis. In CVPR. 2021.
Morgan Buisson, Pablo Alonso-Jiménez, and Dmitry Bogdanov. Ambiguity modeling with label distribution learning for music classification. In ICASSP, 2022.
Zhiwen Cao, Dongfang Liu, Qifan Wang, and Yingjie Chen. Towards unbiased label distribution learning for facial pose estimation using anisotropic spherical gaussian. In ECCV, 2022.
Jingying Chen, Chen Guo, Ruyi Xu, Kun Zhang, Zongkai Yang, and Honghai Liu. Toward children's empathy ability analysis: Joint facial expression recognition and intensity estimation using label distribution learning. IEEE Transactions on Industrial Informatics, 2021a.
Yinbo Chen, Sifei Liu, and Xiaolong Wang. Learning continuous image representation with local implicit image function. In CVPR, 2021b.
Jiankang Deng, Jia Guo, Evangelos Ververas, Irene Kotsia, and Stefanos Zafeiriou. Retinaface: Single-shot multi-level face localisation in the wild. In CVPR, 2020.
Armen Dzhagaryan, Aleksandar Milenkovic, Mladen Milosevic, and Emil Jovanov. An environment for automated measuring of energy consumed by android mobile devices. In PECCS, 2016.
Sergio Escalera, Junior Fabian, Pablo Pardo, Xavier Baró, Jordi Gonzalez, Hugo J Escalante, Dusan Misevic, Ulrich Steiner, and Isabelle Guyon. Chalearn looking at people 2015: Apparent age and cultural event recognition datasets and results. In ICCVW, 2015.
Bin-Bin Gao, Chao Xing, Chen-Wei Xie, Jianxin Wu, and Xin Geng. Deep label distribution learn- ing with label ambiguity. IEEE TIP, 2017.
Bin-Bin Gao, Hong-Y: Yu Zhou, Jianxin Wu, and Xin Geng. Age estimation using expectation of label distribution learning. In IJCAI, 2018.
Xin Geng. Label distribution learning. IEEE TKDE, 2016.
Aishik Ghosh, Benjamin Nachman, and Daniel Whiteson. Uncertainty- aware machine learning for high energy physics. Physical Review D, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In ICCV, 2015.
Zijian Hu, Zhengyu Yang, Xuefeng Hu, and Ram Nevatia. Simple: similar pseudo label exploitation for semi-supervised classification. In CVPR, 2021.
Xiuyi Jia, Xiang Zheng, Weiwei Li, Changqing Zhang, and Zechao Li. Facial emotion distribution learning by exploiting low-rank label correlations locally. In CVPR, 2019.
Xiuyi Jia, Xiaoxia Shen, Weiwei Li, Yunan Lu, and Jihua Zhu. Label distribution learning by maintaining label ranking relation. IEEE TKDE, 2021a.
Xiuyi Jia, Tao Wen, Weiping Ding, Huaxiong Li, and Weiwei Li. Semi-supervised label distribution learning via projection graph embedding. Information Sciences, 2021b.
Chiyu Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang. Matthias Nießner, Thomas Funkhouser, al. Local implicit grid representations for 3D scenes. In CVPR, 2020.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, 2016.
Mingye Ju, Can Ding, Y Jay Guo, and Dengyin Zhang. Idgep: Image dehazing based on gamma correction prior. IEFE TIP, 2019.
10

---

Under review as a conference paper at ICLR 2023
Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. NewrIPS, 2021.
Youngeun Kim and Priyadarshini Panda. Optimizing deeper spiking neural networks for dynamic vision sensing. Neural Networks, 2021.
Amit Kohli, Vincent Sitzmann, and Gordon Wetzstein. Inferring semantic information with 3D neural scene representations. arXiv preprint arXiv:2003 2020.
Qiang Li, Jingjing Wang, Zhaoliang Yao, Yachun Li, Pengju Yang, Jingwei Yan, Chunmao Wang, and Shiliang Pu. Unimodal-concentrated loss: Fully adaptive label distribution learning for ordi- nal regression. In CVPR, 2022.
Shuang Lian, Qianhui Liu, Rui Yan, Gang Pan, and Huajin Tang. Training deep convolutional spiking neural networks with spike probabilistic global pooling. Neural Computation, 2022.
Chen-Hsuan Lin, Chaoyang Wang, and Simon Lucey. SDF-SRN: Learning signed distance 3D
object reconstruction from static images. In NeurIPS, 2020.
Tingting Liu, Jixin Wang, Bing Yang, and Xuan Wang. Ngdnet: Nonuniform gaussian-label dis- tribution learning for infrared head pose estimation and on-task behavior understanding in the classroom. Neurocomputing, 2021.
Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumet- ric rendering: Learning implicit 3D representations without 3D supervision. In CVPR, 2020.
Curtis Northcutt, Lu Jiang, and Isaac Chuang. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research, 2021.
Kinjal Patel, Eric Hunsberger, Sean Batir, and Chris Eliasmith. A spiking neural network for image segmentation. arXiv preprint arXiv:2106.000 089211 2021.
Wenbin Qian, Yinsong Xiong, Jun Yang, and Wenhao Shu. Feature selection for label distribution learning via feature similarity and label correlation. Information Sciences, 2022.
Zhen Qin, Weixuan Sun, Hui Deng, Dongxu Li, Yunshen Wei, Baohong Lv, Junjie Yan, Lingpeng Kong, and Yiran Zhong. cosformer: Rethinking softmax in attention. In ICLR, 2022.
Tingting Ren, Xiuyi Jia, Weiwei Li, Lei Chen, and Zechao Li. Label distribution learning with label-specific features. In IJCAI, 2019a.
Tingting Ren, Xiuyi Jia, Weiwei Li, and Shu Zhao. Label distribution learning with label correlations via low-rank approximation. In IJCAI, 2019b.
Yi Ren and Xin Geng. Sense beauty by label distribution learning. IJCAI, 2017.
Karl Ricanek and Tamirat Tesafaye. Morph: A longitudinal image database of normal adult age- progression, In FGR06, 2006.
Shijing Si, Jianzong Wang. Junqing Peng, and Jing Xiao. Towards speaker age estimation with label distribution learning. In ICASSP, 2022.
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Con- tinuous 3D-structure-aware neural scene representations. In NeurIPS. 2019.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. In NeurIPS, 2020.
Ivan Skorokhodov, Savva Ignatyev, and Mohamed Elhoseiny. Adversarial generation of continuous images. In CVPR, 2021.
Chao Tan, Sheng Chen, Xin Geng, and Genlin label distribution manifold learning algorithm. PR, 2022.
11

---

Under review as a conference paper at ICLR 2023
Qifa Teng and Xiuyi Jia. Incomplete label distribution learning by exploiting global sample correla- tion. In Multimedia Understanding with Less Labeling on Multimedia Understanding with Less Labeling. 2021.
Talip Ucar, Ehsan Hajiramezanali, and Lindsay Edwards. Subtab: Subsetting features of tabular data for self-supervised representation learning. In NeurIPS, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurlPS, 2017.
Jing Wang and Xin Geng. Label distribution learning by exploiting label distribution manifold. IEEE 7NNLS, 2021.
Mitchell
Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farbadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In ICML, 2022.
Ning Xu, Yun-Peng Liu, and Xin Geng. Label enhancement for label distribution learning. IEEE TKDE, 2019.
Ning Xu, Jun Shu, Yun-Peng Liu, and Xin Geng. Variational label enhancement. In ICML, 2020.
Kashu Yamazaki, Viet-Khoa Vo-Ho, Darshan Bulsara, and Ngan Le. Spiking neural networks and their applications: A review. Brain Sciences, 2022.
Jiancheng Yang, Rui Shi, and Bingbing Ni. Medmnist classification decathlon: lightweight automl benchmark for medical image analysis. In ISBI, 2021.
Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lip- man. Multiview neural surface reconstruction by disentangling geometry and appearance. 2020.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. Mixup: Beyond empiri- cal risk minimization. In /CLR, 2018
Peng Zhao and Zhi-Hua Zhou. Label distribution learning by optimal transport. AAAI, 2018.
Xingyu Zhao, Yuexuan An, Ning Xu, and Xin Geng. Continuous label distribution learning. Pattern Recognition, 2022a.
Xingyu Zhao, Yuexuan An, Ning Xu, and Xin Geng. Fusion label enhancement for multi-label learning. In IJCAI, 2022b.
Zengqun Zhao, Qingshan Liu, and Feng Zhou Robust lightweight facial expression recognition network with label distribution training. AAAI, 2021.
Qinghai Zheng, Jihua Zhu, Haoyu Tang, Xinyuan Liu, Zhongyu Li, and Huimin Lu. Generalized label enhancement with sample correlations. IEEE TKDE, 2021a.
Rui Zheng, Shulin Zhang, Lei Liu, Yuhao Luo, and Mingzhai Sun. Uncertainty in bayesian deep label distribution learning. Applied Soft Computing, 2021b.
12

---

Under review as a conference paper at ICLR 2023
8 APPENDIX
The appendix is supplemented with the parameter settings of the network and the performance of all comparison algorithms on multiple data sets. Furthermore, supplement the t-test in 12 data sets to validate the stability of our method. Since there are many hyperparameters, we conduct a parametric analysis a Gene dataset.
Examples
Festures
Labels
Number
SNN
I.F
DA
we-LDL
500
243
12
2 (2tendd + [243, 1024]
Eq.5
2
SIAFFE
213
243
[2rend [243, 1024]
5q.5
3
SHU-DDFE
2500
243
2 1.2iend 1243, 10241
Eq.5
4
Scene
2000
294
21 112ie + [294, 1024]
Ey.5
Gene
17892
L [2rend
64]
Fq.5
x
6
Mioiie
7755
1869
28' (2:end [1869, 1024]
E4.5
X
7
M2B
1240
250
S (2tendd - [256, 1024]
Fq.5
SCUT
1500
300
21 .2eedd [300, 10241
lq.55
Rbp5500
5500
512
2 .2iend [512, 1024]
Ey.5
X
10
RAF-M.
4008
200
5 22annd [200, 1034]
F4.5
Twiter
10040
200
21 .2endd [200, 1024]
Eq.5
x
Flickr
11150
200
L 22and [2000 1024]
Eq.5
X
Table 5: Statistics of the experimental datasets with models.
10
Daraset
Botch size
Epoch
Learning rate
Weight decay
Early stopping
Groed
wo.EDI.
500
10-3
10-4
2
STAFFE
213
200
2 10-2
1 X 10
3
SRU-ADFE
1000
1 10-3
10-4
4
Some
1000
120
1 10-3
1 10-1
5
Gene
5000
150
2 10-3
1 10-4
Movie
2000
X 10
10
7
MZB
500
150
2 10
1 10-4
SCUT
500
1 10-2
10
Rtp5800
1500
300
2 10
10-1
10
RAF ML
2000
100
1 10-3
1 10-40-4
Twister
5000
10
1x
X
12
Flickr
5000
200
1 10
x 10
Table 6: Training configuration of our model.
Dataset
Algorishm
Chebyshev
Clark
Cunherna
K-L.
Cosine
Intersection 1
Ours
0.1456 1: 0.0021
1.3651 1: 0.0441
2.6888 1: 0.0023
0,2017 1 0.0012
0.9394
0.0026
0.8247
0.0077
a-LDL
0.0032
0.0441
1
0.2112
0.3300
0.0031
0.0075
LDL-IRR
0.1526
0.0033
1,5651
1: 0.0111
2.7594
1 0.0422
0.2449
0.0007
0.9251
0.0003
0.8141
0.0044
RAF-ML
LDL-ECUR
0,1515
0.0022
1,592 + 0.0117
2.7779
+ 0.0239
0.2244 + 0.0030
0.9262
0.0062
0.8189
0.0098
LDLSF
0,1488
0.0024
1.3889-: 0.0086
2.7672
0.2302
0.0044
019111
0.0051
0.0622
LALOT
0,1479 :: 0.0010
1,3659
+ 0.0099
2.6956 + 0.0144
0.2221
+ 0.0064
0.9311
0.0021
0.8107
0.0008
BFGS-LLD
0,1499
0.0009
1.6656 0.0066
2,7101 + 0.0211
0.2541
0.0055
0.9204
0.0023
0.8157
0050
Ours
0,2777 0.0021
2.2374
0.0110
5,1163 : 0.0018
0.5111
0.0029
0.8807
0.0049
0.7891
0.0014
n-L.D.
0,2985 0.0011
2,3002
+ 0.0112
5,4414 + 0.0065
0,5242 + 0.0033
0.8554
0.0046
0.7709
0.0084
LDL-LRR
0.0021
0.6616 0.0035
0.8002
0.0042
0.7411
0.0014
Twitter
LDL-LCLR
0,2994 0.0045
2.4900
1 0.0012
6.9609 1 0.0041
0.6056
0.0031
0.71
0.0021
0.7110
0.0088
LDLSF
0.3007
0.0002
2.7887
: 0.0057
5.6101
: 0.0118
0.6396
0.0022
0.7939
0.0098
0.7660
0.0007
LALOT
0,3133
0.0021
2.3141
0.0016
5.5336
0.5233
0.055
0.0049
BFGS-LLD
0,3114 = 0.0014
2.5511
: 0.0028
5.7145
+ 0.0041
0,5461 = 0.0153
0.8335
0.0055
0.7744
0.0020
Ours
0,2816
0.0031
2.3356
0.0097
5,2222 1
0,5314
0.8406
0.0043
0.0025
a-LDL.
0,2998 0.0088
2,4388
1 0.0089
5.3111 1 0.0119
0.5729
0.0067
0.8331
0.0009
0.7500 - 0.0633
LDI
0.3329
0.0012
3,4000
+ 0.0174
6.3459 + 0.0229
0.6516
0.0031
0.8450
0.0040
0.7399
0.0037
Flickr
LDL-LCLR
LDL.F
0.2970
0.3301 + 0.0009
0.0009
2,8888
+ 0.0459
0.0063
5.9152 + 0.0121
0.0041
0.6100
0.6222
+ 0.0021
0.7919
0.8139
0.0029
0.0098
0.7360
0.0670
0.0637
LALOT
0,3411
0.0026
2.9140
0.0019
5,3333
0.0243
0,5737
0.0012
0.8225
0.020
0.7144
0.0004
BEGS-LLD
0,3200 0.0041
2.7517
+ 0.0060
5.8149
+ 0. 0.0048
0.5961 0.0099
0.8131
0.0011
0.7407 0.0077
Table 7: The performance of our proposed method with the comparison algorithms on 12 datasets.
1-test on 12 datasets. We evaluate the range of p-values for the six metrics on 12 data sets.
Cheby.[1.65e 105, 1.00e + 00). Clark|6.87e: 97, 1.99e 02). Canbe. Canbe.19.754 99.75e 99, 1.23e 01], KL91.99e - 101, 1.23e 01], Cosine[1.02e - 98, 2.99c - 01]. and Inter. (5.0ce - 112,3.67 - 01 According to the test results, the LDL methods have significantly different performance in terms of each metric on all datasets except Gene (at a 0.05 significance level). The label distribution of Gene
tends to be uniformly distributed, which may result in the equal performance of the LDL. approaches.
13

---

Under review as a conference paper at ICLR 2023
Parameter Sensitivity Analysis. Our method has three parameters, including the regularization parameter A1, Ag. and 3. To analyze the sensitivity of λ1, A2, and B, we run our method with three sets (f0.001, 0.005, 0.01, 0.05, 0.1), {0.001, 0.005, 0.01, 0.05, 0.1), and {0.001, 0.005, 0.08, 0.02, 0.08J) on the Gene dataset (see Figure
$000
6.8
8.50
600
8309
Facenter
(0) A
(b) Xa
(c) B
Figure 4: This figure shows the sensitivity of parameters the Gene dataset.
Formula Supplement.
(1) Scaled Dot-Product attention:
Attention K, V) = softmax( QKT vdx
where dd is the dimension of the key vector kt and query vector q
(2) ReLU:
ReLU(()) max((0 2).
(3) L1 loss:
D
|rg Vel.
(4) KL divergence:
M
Kl.(ill)) Σ
De log De yc
(5) Perceptual loss:
L1 (MLL(()).
14


Reviews:
[{"review_id": "fYMiLe0xHc", "reviewer": "Reviewer_7uiR", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper describes a method for label distribution learning (LDL). By my understanding, the contributions of this work fall into two categories: (i) network architecture and (ii) optimization objective.\nFor (i), the authors use an SNN (Spiking neural network) and a Graph Convolutional Neural Network (GCN) to generate a label distribution matrix. The label distribution matrix is the \"implicit distribution representation\" referred to in the title and abstract. Intuitively, this matrix is a distribution over label distributions, which are regularized towards Gaussians.\nFor (ii), the authors use L1 , KL, and perceptual losses, along with mix-up, random masking and linear normalization to yield their final framework.", "summary_of_the_review": "This paper appears marginal. At first glance, there is some novelty with regards to the model architecture (specifically, all the stuff between input features and output label distribution), and there are some convincing empirical results. \n\nHowever, most empirical results presented are based on a combination of the novel architecture changes and several standard regularization tricks. Consequently, most comparisons are in my opinion unfair, and the empirical significance of the architecture contributions is unclear.", "strength_and_weaknesses": "Strengths:\n(1) The empirical results appear strong and definitely exceed state-of-the-art.\n\nWeaknesses:\n(1) The only major argument I see against acceptance would be lack of novelty. I will split comments into 2 parts: (i) innovations in neural architecture (model/backbone design) and (ii) optimization objective (loss function). \n\n(i) The architecture takes as input data features and outputs a discrete distribution over possible labels. The authors make two innovations: (a) using SNN and (b) training a label distribution matrix, which is an estimate of the distribution over label distributions. (b) is their main contribution, since it is highlighted in the title of the work. (a) is a minor contribution, since it is an off-the-shelf component used in a new setting.\n\nRegarding the label distribution matrix:\n\nQ1: From the paper, it is not clear *how novel* the label distribution matrix (implicit distribution representation) part of the work is? Specifically, looking at the paragraph with the heading ``Label Distribution Matrix Learning'', I see numerous citations regarding confidence and uncertainty learning. How is this work different?\n\nQ2: From my reading of the paper, I do not see a robust justification for the addition of this label distribution matrix. Why do we expect this addition to the model architecture to improve results?\n\nQ3: Why is the shape of the label distribution matrix L x L x 2 and the distribution values 1 x 2L? Specifically, where does the 2 come from?\n\n(ii) I find the optimization objective to be a collection of standard approaches and therefore not novel (All the content after \"Regularization Techniques\" in Section 2). Specifically:\n\n(a) The linear normalization function is just the standard way of normalizing a vector of values with a constant offset.\n\n(b) Data augmentation: Mix-up is one of the most ubiquitous tricks in computer vision. Random masking is similarly ubiquitous (Drop-out, for example).\n\n(c) The loss function consists of L1 and KL distances, regularizing a distribution towards a Gaussian, perceptual loss ... these are all standard.\n\nExperimental Results:\n\nTable 3 shows that the entire method as a package achieves state-of-the-art on many benchmarks. However, given that the novel part of the method is the model *architecture*, I would expect some architecture-to-architecture comparisons. For example, \"Ours\" in the Table includes mix-up, random masking, and some regularization; do the baselines use these techniques? If not, I would consider these comparisons unfair, since these techniques are straightforward to apply. \n\nThe first two rows and Table 4 show that on the Gene dataset, removing the perceptual loss from the proposed framework lowers the performance to be about the same as the state-of-the-art. This is concerning, since this suggests that all of the gains are coming from the perceptual loss. \n\nOther comments:\n\nQ4: Why does the Gaussian regularization term make sense? (referring to the last term in equation (6) where you regularize the matrix $\\mathcal{M}$ towards the constant $\\overline{\\mathcal{M}}$) If I understand correctly, each row of matrix $\\mathcal{M}_i$ is a distribution over probability of the true label being label $i$. This is a number between 0 and 1. (see Figure 2). A distribution over the range [0,1] cannot be a Gaussian. For example, Figure 2(e) shows that the model gave some weight to the probability of true label being L5 to be over 1.0 or under 0.0. The true label cannot be L5 with 150% probability. Furthermore, the label distribution has to add up to one -- that is, every time you sample a column vector using the rows of $\\mathcal{M}$ as the probability distribution, you should get $L$ non-negative values that sum up to 1. So it doesn't make sense to me why you would regularize each row of $\\mathcal{M}$ to be an independent Gaussian. Perhaps I misunderstood somethings -- please clarify.\n\nMinor comments:\n(1) Notation in the method section is ambiguous and hard to follow:\n\n(a) C is used in Equation (2); it has already been used to denote the coordinate matrix, perhaps use $K$?\n\n(b) Before Equation (2), you use bold-case $\\mathbf{d}_i$ to denote a vector and $d^y_i$ to denote scalar entries into this vector. Later on in equation (5) and (6), you do not bold the vectors $\\hat{d}_i$ and $d_i$. Adding to the confusion, you use both superscripts and subscripts to index vectors and matrices, so the math is hard to follow.\n\n(c) Eq. 5: Please be specific as to which norm is used in $\\|\\| \\cdot \\|\\|$. Please specify which distributions are being compared by the KL loss term $\\mathcal{L}_{kl}$\n\n(d) Eq. 5: in the last term index $i$ is abused. The way you wrote it is looks as though the loss only operates on the diagonal of matrix $\\mathcal{M}$.\n\n(e) In the sentence after Eq. (5), `` $d_i$ is the label distribution ... $\\hat{d}$ is the ground truth''. Is this a typo? Please do not use hats to indicate ground truth.\n\n(f) Eq. (4) Please do not use $\\times$ for element wise vector multiplication, it is confusing.\n\n(g) In Eq. 4 the $y_i$s are clearly vectors. In equation (2), you use the exact same symbol as scalar indices into the vector $\\mathbf{d}_i$ as $d_i^{y_i}$.\n\n(h) Eq. (1) Please either use the operator $\\exp d$ or write $e^d$.\n\n(i) Tables: Bolding the standard deviation does not make sense. I recommend bolding the best number instead. \n\n(j) There is no Appendix to this paper. While that by itself is not an issue, there are small experimental details throughout the paper that would logically belong in an Appendix. For example, in the Method section, the authors spend many sentences describing details such as \" each linear unit contains 1024 neurons ... contains 64 neurons ... we attempted other activation functions ReLU, Swish, Sigmoid ...''. There are details regarding the specific library and version used to implement the method -- all before the actual method is presented.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Zhy4nzKqx1y", "reviewer": "Reviewer_JLrG", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces the implicit distribution in the label distribution learning framework to handle the uncertainty of each label value in the label distribution training set. It uses deep implicit representation learning to construct a label distribution matrix with Gaussian prior constraints to moderate the noise and uncertainty interference of the label distribution dataset, and label distributions are transformed by using the self-attention algorithm through each row component of the label distribution matrix.\n\nThe main contributions of this paper are:\n\n- It points out that the label distribution dataset used for training has a high probability of inaccuracy and uncertainty, which significantly limits the performance of LDL algorithms.\n- This paper introduces the implicit distribution in the label distribution learning framework to characterize the uncertainty of each label value.\n- This paper adopts Spiking neural network with an MLP to save energy consumption of mobile devices, and correlations between labels are deeply mined by a graph convolutional network.\n- This paper designs some regularization techniques to boost the performance of the model and a new LDL dataset is released.", "summary_of_the_review": "If the authors can address my concerns well, I would tend to accept this paper.\n\n", "strength_and_weaknesses": "### Strength\n\n- In my opinion, the motivation for this paper is interesting. In detail, both noise and uncertainty in label distribution learning are important and urgent problems to be solved. In addition, the introduction of deep implicit representation learning is also a very valuable attempt.\n- This paper provides some novel perspectives. First, the introduction of implicit label distribution representation brings an effective way to characterize and mitigate the uncertainty of the label distribution. Second,  the proposed Label Distribution Matrix Learning also presents a novel view for estimating the uncertainty of the labels. Besides, the regularization techniques designed in this paper can give some inspiration to researchers in the field of label distribution learning.\n- For the empirical studies, this paper conducts extensive experiments and demonstrates the advantages of the proposed method over the state-of-the-art LDL methods, which can effectively reflect the performance of the algorithm.\n- The paper is well-organized and clearly written, which is easy to follow.\n\n### Weaknesses\n\n- I am concerned about the self-attention mechanism in the method. Concretely, the authors mention that, \"to capture the global correlation between labels to generate a standard label distribution, we employ a self-attention mechanism to model the label distribution matrix\". However, in my opinion, this statement is not strongly supported. I suggest the authors further analyze how the self-attention mechanism captures the global correlation between labels.\n- Although the proposed method seems to be very effective compared with existing methods, some statistical methods should be used to illustrate the experimental results, such as t-test and Bonferroni-Dunn test, to confirm this observation.\n- I find there are many hyper-parameters that should be tuned in the experiments, such as batch size, the number of epochs, learning rate, the weights $\\lambda_1$, $\\lambda_2$ and $\\beta$. The authors should analyze the influence of these parameters on the final performances.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "UH26D6PhoIS", "reviewer": "Reviewer_vTyL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Since the complexity of the manual annotation task or the inaccuracy of the label enhancement algorithm leads to noise and uncertainty in the label distribution training set, it is hard to generate accurate label distribution. This paper proposes a novel framework for label distribution learning by introducing implicit neural representation. It is achieved by generating a coordinate matrix, and then using this matrix to sample latent features, which is extracted by a deep spiking neural network. In addition, a series of regularization techniques are used to mitigate the overfitting problem. The experiments have proven the effectiveness of the proposed method.", "summary_of_the_review": "This paper proposes a novel framework for label distribution learning by introducing implicit neural representation. It is achieved by generating a coordinate matrix, and then using this matrix to sample latent features, which is extracted by a deep spiking neural network. However, some details of the proposed strategy are missing, and the written of the paper should be further improved.", "strength_and_weaknesses": "Strength:\n1.\tThe idea of considering the label distribution as an implicit neural representation is interesting.\n2.\tThe experimental results show that the proposed method is promising.\n3.\tIn the proposed method, a novel augmentation strategy is proposed to improve the performance of LDL.\n\nWeakness:\n1.\tSome details of the proposed method are missing, such as the definition of \\mathcal{L}{{kl} and the representation of the augmentation samples in the function.\n2.\tIn the proposed method, a novel augmentation strategy is proposed with mask. In the ablation study, the effectiveness of the proposed strategy has not been verified by comparing with mixup and so on. The main contributions of the proposed method should be further verified in the experiments.\n3.\tMeanwhile, more details about the proposed method should be presented, such as how the implicit distribution characterize the uncertainty of each label value and how the model mitigrate the uncertainty of the label distribution.  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "G0DstTkL-p", "reviewer": "Reviewer_eABN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an implementation of the spike-based learning method for the label distribution learning (LDL) problem. Main idea is to use a spiking neural network (SNN) to construct a latent feature space in which the coordinate matrix learned from a graph convolution network look up the table. The experimental results showed that the proposed method can achieve high accuracy in several tasks compared to existing methods.", "summary_of_the_review": "The paper presents a nice step in an interesting direction, but it does not clarify what exactly its innovations are.", "strength_and_weaknesses": "Strengths:\n1. The paper is to the best of my knowledge the first to try out the improvement of SNN on LDL.\n2.  This paper conducts sufficient experiments, and the results look superior.\n\nWeaknesses:\n1. The proposed method is not well-motivated. This paper provides a dense description of the algorithm, but it is difficult for the reader to understand the ideas behind the method. I think it is interesting idea to use SNNs to solve LDL, but it requires more reasoning about why it has the potential to go beyond existing methods and what is its unique methodological contribution given quite a number of existing SNN works.\n2. The paper lacks formal introductions of the problem studied (LDL) and several techniques used (SNN, GCN, et al.). It is hard to understand basic concepts without reading the reference, so it would be better to explain more details about the background. \n3. The baselines use different model and some of them just use a linear model. So I am concerned whether the superiority of the proposed algorithm comes from the complex network structure. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "D9Mr1UynHw1", "reviewer": "Reviewer_Paus", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The polysemy of data introduces a big challenge for the labeling process. This paper proposes an implicit distribution representation method based on label distribution learning, which also considers the uncertainty of label values, in which the Gaussian prior and self-attention-based methods are also adopted for learning the local and global label distribution matrix. A relatively large-scale evaluation on 12 datasets is performed, confirming the potential of the proposed methods. ", "summary_of_the_review": "Overall, this paper proposes a sound and feasible solution for label distribution learning. The evaluation results also show to be promising. However, it still posts a few concerns, regarding technical novelty, contribution, and application to a more wide range of applications. This paper works on an important problem, and these concerns should be carefully discussed and justified for further enhancement.", "strength_and_weaknesses": "Strength:\n\n- Important and challenging problem\n- Sound and feasible solution\n- Extensive evaluation to demonstrate the potential usefulness\n- Promising results\n\nWeakness:\n\n- The presentation needs to be improved, lacking motivation examples, discussion on intuition and insights.\n- Unclear about the technical novelty, which is more an assemble of existing techniques.\n- Unclear why energy consumption could be a concern of this paper, since there are quite a big bunch of community that works on model compression and optimization for diverse edge devices\n- Unclear how to extend to streaming data cases.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject