Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The polysemy of data introduces a big challenge for the labeling process. This paper proposes an implicit distribution representation method based on label distribution learning, which also considers the uncertainty of label values, in which the Gaussian prior and self-attention-based methods are also adopted for learning the local and global label distribution matrix. A relatively large-scale evaluation on 12 datasets is performed, confirming the potential of the proposed methods. 
strength_and_weaknesses: Strength:

- Important and challenging problem
- Sound and feasible solution
- Extensive evaluation to demonstrate the potential usefulness
- Promising results

Weakness:

- The presentation needs to be improved, lacking motivation examples, discussion on intuition and insights.
- Unclear about the technical novelty, which is more an assemble of existing techniques.
- Unclear why energy consumption could be a concern of this paper, since there are quite a big bunch of community that works on model compression and optimization for diverse edge devices
- Unclear how to extend to streaming data cases.
clarity,_quality,_novelty_and_reproducibility: The overall presentation of this paper is mostly ok, but still hard to follow in a few places. It lacks a concrete example to motivate the paper, and gives a very intuitive introduction to the considered contexts. Although the paper works on a relatively new and important problem, it is not clear what is the technical novelty of this paper, since each part of the proposed workflow has been extensively studied before. In addition, one concern is that the paper mostly discusses what the proposed methods are, but lacks a discussion on intuition and insights, e.g., why each component is designed in the proposed way, and why not other alternatives. These could be clearly justified.
summary_of_the_review: Overall, this paper proposes a sound and feasible solution for label distribution learning. The evaluation results also show to be promising. However, it still posts a few concerns, regarding technical novelty, contribution, and application to a more wide range of applications. This paper works on an important problem, and these concerns should be carefully discussed and justified for further enhancement.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper describes a method for label distribution learning (LDL). By my understanding, the contributions of this work fall into two categories: (i) network architecture and (ii) optimization objective.
For (i), the authors use an SNN (Spiking neural network) and a Graph Convolutional Neural Network (GCN) to generate a label distribution matrix. The label distribution matrix is the "implicit distribution representation" referred to in the title and abstract. Intuitively, this matrix is a distribution over label distributions, which are regularized towards Gaussians.
For (ii), the authors use L1 , KL, and perceptual losses, along with mix-up, random masking and linear normalization to yield their final framework.
strength_and_weaknesses: Strengths:
(1) The empirical results appear strong and definitely exceed state-of-the-art.

Weaknesses:
(1) The only major argument I see against acceptance would be lack of novelty. I will split comments into 2 parts: (i) innovations in neural architecture (model/backbone design) and (ii) optimization objective (loss function). 

(i) The architecture takes as input data features and outputs a discrete distribution over possible labels. The authors make two innovations: (a) using SNN and (b) training a label distribution matrix, which is an estimate of the distribution over label distributions. (b) is their main contribution, since it is highlighted in the title of the work. (a) is a minor contribution, since it is an off-the-shelf component used in a new setting.

Regarding the label distribution matrix:

Q1: From the paper, it is not clear *how novel* the label distribution matrix (implicit distribution representation) part of the work is? Specifically, looking at the paragraph with the heading ``Label Distribution Matrix Learning'', I see numerous citations regarding confidence and uncertainty learning. How is this work different?

Q2: From my reading of the paper, I do not see a robust justification for the addition of this label distribution matrix. Why do we expect this addition to the model architecture to improve results?

Q3: Why is the shape of the label distribution matrix L x L x 2 and the distribution values 1 x 2L? Specifically, where does the 2 come from?

(ii) I find the optimization objective to be a collection of standard approaches and therefore not novel (All the content after "Regularization Techniques" in Section 2). Specifically:

(a) The linear normalization function is just the standard way of normalizing a vector of values with a constant offset.

(b) Data augmentation: Mix-up is one of the most ubiquitous tricks in computer vision. Random masking is similarly ubiquitous (Drop-out, for example).

(c) The loss function consists of L1 and KL distances, regularizing a distribution towards a Gaussian, perceptual loss ... these are all standard.

Experimental Results:

Table 3 shows that the entire method as a package achieves state-of-the-art on many benchmarks. However, given that the novel part of the method is the model *architecture*, I would expect some architecture-to-architecture comparisons. For example, "Ours" in the Table includes mix-up, random masking, and some regularization; do the baselines use these techniques? If not, I would consider these comparisons unfair, since these techniques are straightforward to apply. 

The first two rows and Table 4 show that on the Gene dataset, removing the perceptual loss from the proposed framework lowers the performance to be about the same as the state-of-the-art. This is concerning, since this suggests that all of the gains are coming from the perceptual loss. 

Other comments:

Q4: Why does the Gaussian regularization term make sense? (referring to the last term in equation (6) where you regularize the matrix $\mathcal{M}$ towards the constant $\overline{\mathcal{M}}$) If I understand correctly, each row of matrix $\mathcal{M}_i$ is a distribution over probability of the true label being label $i$. This is a number between 0 and 1. (see Figure 2). A distribution over the range [0,1] cannot be a Gaussian. For example, Figure 2(e) shows that the model gave some weight to the probability of true label being L5 to be over 1.0 or under 0.0. The true label cannot be L5 with 150% probability. Furthermore, the label distribution has to add up to one -- that is, every time you sample a column vector using the rows of $\mathcal{M}$ as the probability distribution, you should get $L$ non-negative values that sum up to 1. So it doesn't make sense to me why you would regularize each row of $\mathcal{M}$ to be an independent Gaussian. Perhaps I misunderstood somethings -- please clarify.

Minor comments:
(1) Notation in the method section is ambiguous and hard to follow:

(a) C is used in Equation (2); it has already been used to denote the coordinate matrix, perhaps use $K$?

(b) Before Equation (2), you use bold-case $\mathbf{d}_i$ to denote a vector and $d^y_i$ to denote scalar entries into this vector. Later on in equation (5) and (6), you do not bold the vectors $\hat{d}_i$ and $d_i$. Adding to the confusion, you use both superscripts and subscripts to index vectors and matrices, so the math is hard to follow.

(c) Eq. 5: Please be specific as to which norm is used in $\|\| \cdot \|\|$. Please specify which distributions are being compared by the KL loss term $\mathcal{L}_{kl}$

(d) Eq. 5: in the last term index $i$ is abused. The way you wrote it is looks as though the loss only operates on the diagonal of matrix $\mathcal{M}$.

(e) In the sentence after Eq. (5), `` $d_i$ is the label distribution ... $\hat{d}$ is the ground truth''. Is this a typo? Please do not use hats to indicate ground truth.

(f) Eq. (4) Please do not use $\times$ for element wise vector multiplication, it is confusing.

(g) In Eq. 4 the $y_i$s are clearly vectors. In equation (2), you use the exact same symbol as scalar indices into the vector $\mathbf{d}_i$ as $d_i^{y_i}$.

(h) Eq. (1) Please either use the operator $\exp d$ or write $e^d$.

(i) Tables: Bolding the standard deviation does not make sense. I recommend bolding the best number instead. 

(j) There is no Appendix to this paper. While that by itself is not an issue, there are small experimental details throughout the paper that would logically belong in an Appendix. For example, in the Method section, the authors spend many sentences describing details such as " each linear unit contains 1024 neurons ... contains 64 neurons ... we attempted other activation functions ReLU, Swish, Sigmoid ...''. There are details regarding the specific library and version used to implement the method -- all before the actual method is presented.
clarity,_quality,_novelty_and_reproducibility: Quality  and clarity of writing is below average. In particular, the notation used in the Method section is confusing and ambiguous. I encourage the authors to proofread the Methods section. 

The model architecture is novel. However, the authors do not evaluate the architecture by itself, so it is hard to determine whether the performance gains seen in Table 3 can be attributed to the novel architecture or the additional regularization tricks being used.
summary_of_the_review: This paper appears marginal. At first glance, there is some novelty with regards to the model architecture (specifically, all the stuff between input features and output label distribution), and there are some convincing empirical results. 

However, most empirical results presented are based on a combination of the novel architecture changes and several standard regularization tricks. Consequently, most comparisons are in my opinion unfair, and the empirical significance of the architecture contributions is unclear.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Since the complexity of the manual annotation task or the inaccuracy of the label enhancement algorithm leads to noise and uncertainty in the label distribution training set, it is hard to generate accurate label distribution. This paper proposes a novel framework for label distribution learning by introducing implicit neural representation. It is achieved by generating a coordinate matrix, and then using this matrix to sample latent features, which is extracted by a deep spiking neural network. In addition, a series of regularization techniques are used to mitigate the overfitting problem. The experiments have proven the effectiveness of the proposed method.
strength_and_weaknesses: Strength:
1.	The idea of considering the label distribution as an implicit neural representation is interesting.
2.	The experimental results show that the proposed method is promising.
3.	In the proposed method, a novel augmentation strategy is proposed to improve the performance of LDL.

Weakness:
1.	Some details of the proposed method are missing, such as the definition of \mathcal{L}{{kl} and the representation of the augmentation samples in the function.
2.	In the proposed method, a novel augmentation strategy is proposed with mask. In the ablation study, the effectiveness of the proposed strategy has not been verified by comparing with mixup and so on. The main contributions of the proposed method should be further verified in the experiments.
3.	Meanwhile, more details about the proposed method should be presented, such as how the implicit distribution characterize the uncertainty of each label value and how the model mitigrate the uncertainty of the label distribution.  

clarity,_quality,_novelty_and_reproducibility: Some details are missing, such as the definition of \mathcal{L}_{kl}. To make the paper easy to follow, it is better for the paper to be reorganized, such as the application of the augmentations in the loss function. There exist some novelties in the proposed method, such as the regularization technique.

summary_of_the_review: This paper proposes a novel framework for label distribution learning by introducing implicit neural representation. It is achieved by generating a coordinate matrix, and then using this matrix to sample latent features, which is extracted by a deep spiking neural network. However, some details of the proposed strategy are missing, and the written of the paper should be further improved.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an implementation of the spike-based learning method for the label distribution learning (LDL) problem. Main idea is to use a spiking neural network (SNN) to construct a latent feature space in which the coordinate matrix learned from a graph convolution network look up the table. The experimental results showed that the proposed method can achieve high accuracy in several tasks compared to existing methods.
strength_and_weaknesses: Strengths:
1. The paper is to the best of my knowledge the first to try out the improvement of SNN on LDL.
2.  This paper conducts sufficient experiments, and the results look superior.

Weaknesses:
1. The proposed method is not well-motivated. This paper provides a dense description of the algorithm, but it is difficult for the reader to understand the ideas behind the method. I think it is interesting idea to use SNNs to solve LDL, but it requires more reasoning about why it has the potential to go beyond existing methods and what is its unique methodological contribution given quite a number of existing SNN works.
2. The paper lacks formal introductions of the problem studied (LDL) and several techniques used (SNN, GCN, et al.). It is hard to understand basic concepts without reading the reference, so it would be better to explain more details about the background. 
3. The baselines use different model and some of them just use a linear model. So I am concerned whether the superiority of the proposed algorithm comes from the complex network structure. 
clarity,_quality,_novelty_and_reproducibility: The writing can be improved for easier understanding. 
summary_of_the_review: The paper presents a nice step in an interesting direction, but it does not clarify what exactly its innovations are.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces the implicit distribution in the label distribution learning framework to handle the uncertainty of each label value in the label distribution training set. It uses deep implicit representation learning to construct a label distribution matrix with Gaussian prior constraints to moderate the noise and uncertainty interference of the label distribution dataset, and label distributions are transformed by using the self-attention algorithm through each row component of the label distribution matrix.

The main contributions of this paper are:

- It points out that the label distribution dataset used for training has a high probability of inaccuracy and uncertainty, which significantly limits the performance of LDL algorithms.
- This paper introduces the implicit distribution in the label distribution learning framework to characterize the uncertainty of each label value.
- This paper adopts Spiking neural network with an MLP to save energy consumption of mobile devices, and correlations between labels are deeply mined by a graph convolutional network.
- This paper designs some regularization techniques to boost the performance of the model and a new LDL dataset is released.
strength_and_weaknesses: ### Strength

- In my opinion, the motivation for this paper is interesting. In detail, both noise and uncertainty in label distribution learning are important and urgent problems to be solved. In addition, the introduction of deep implicit representation learning is also a very valuable attempt.
- This paper provides some novel perspectives. First, the introduction of implicit label distribution representation brings an effective way to characterize and mitigate the uncertainty of the label distribution. Second,  the proposed Label Distribution Matrix Learning also presents a novel view for estimating the uncertainty of the labels. Besides, the regularization techniques designed in this paper can give some inspiration to researchers in the field of label distribution learning.
- For the empirical studies, this paper conducts extensive experiments and demonstrates the advantages of the proposed method over the state-of-the-art LDL methods, which can effectively reflect the performance of the algorithm.
- The paper is well-organized and clearly written, which is easy to follow.

### Weaknesses

- I am concerned about the self-attention mechanism in the method. Concretely, the authors mention that, "to capture the global correlation between labels to generate a standard label distribution, we employ a self-attention mechanism to model the label distribution matrix". However, in my opinion, this statement is not strongly supported. I suggest the authors further analyze how the self-attention mechanism captures the global correlation between labels.
- Although the proposed method seems to be very effective compared with existing methods, some statistical methods should be used to illustrate the experimental results, such as t-test and Bonferroni-Dunn test, to confirm this observation.
- I find there are many hyper-parameters that should be tuned in the experiments, such as batch size, the number of epochs, learning rate, the weights $\lambda_1$, $\lambda_2$ and $\beta$. The authors should analyze the influence of these parameters on the final performances.


clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is well-organized and clearly written.

Quality: Technically solid paper, with a high impact on the research field of label distribution learning.

Novelty: The paper makes non-trivial advances over the current state-of-the-art.

Reproducibility:  Key details are sufficiently well-described for competent researchers to confidently reproduce the main results.
summary_of_the_review: If the authors can address my concerns well, I would tend to accept this paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Thanks for your submission to ICLR.

This paper ended up after discussion with three somewhat negative reviews and two somewhat to reasonably positive reviews.  In terms of strengths, the reviewers noted that it was an interesting problem tackled and there are some promising results.  On the negative side, multiple reviewers noted issues in presentation / details in the paper, multiple reviewers took issue with the novelty, and multiple reviewers noted some issues with the experimental results.

After discussion, some of these issues seemed to be resolved but others were still unresolved.  After discussion, three of the reviewers still remained on the negative side, not advocating for accepting the paper.  Unfortunately, it seems that there's not enough here to warrant accepting the paper for ICLR.  It seems with some additional work this paper could be quite strong.  Please do keep in mind the suggestions of the reviewers when preparing a future version of the manuscript.
justification_for_why_not_higher_score: With three of the five reviewers not willing to advocate for accepting the paper, it does not appear that the paper is ready for publication.  The second reviewer noted several issues with the paper in particular.
Decision: Reject