DNA: Denoised Neighborhood Aggregation for Fine-grained Category Discovery
Authors: Wenbin An
Feng Tian
Wenkai Shi
Yan Chen
Qinghua Zheng
QianYing Wang
Ping Chen
Venue: EMNLP 2023 Main (2023)
Keywords: Fine-grained Category Discovery
Denoised Neighborhood Contrastive Learning
TL;DR: We propose a Denoised Neighborhood Aggregation framework to discover fine-grained categories from coarsely labeled data.

Abstract: Discovering fine-grained categories from coarsely labeled data is a practical and challenging task, which can bridge the gap between the demand for fine-grained analysis and the high annotation cost. Previous works mainly focus on instance-level discrimination to learn low-level features, but ignore semantic similarities between data, which may prevent these models learning compact cluster representations. In this paper, we propose $\textit{Denoised Neighborhood Aggregation}$ (DNA), a self-supervised framework that encodes semantic structures of data into the embedding space. Specifically, we retrieve $\textit{k}$-nearest neighbors of a query as its positive keys to capture semantic similarities between data and then aggregate information from the neighbors to learn compact cluster representations, which can make fine-grained categories more separatable. However, the retrieved neighbors can be noisy and contain many false-positive keys, which can degrade the quality of learned embeddings. To cope with this challenge, we propose three principles to filter out these false neighbors for better representation learning. Furthermore, we theoretically justify that the learning objective of our framework is equivalent to a clustering loss, which can capture semantic similarities between data to form compact fine-grained clusters. Extensive experiments on three benchmark datasets show that our method can retrieve more accurate neighbors (21.31% accuracy improvement) and outperform state-of-the-art models by a large margin (average 9.96% improvement on three metrics).  Our code and data are available at https://github.com/Lackel/DNA.

DNA: Denoised Neighborhood A ggregation for Fine-grained Category Discovery
Wenbin An', Feng Tian"*, "ean"" Shi', Yan Chen', Qinghua Zheng
QianYing Wang', Ping Chen
School of Automation Science and Engineering, Xi'an Jiaotong University
2 School of Computer Science and Technology, MOEKLNNS Lab, Xi' an Jiaotong University
3 Lenovo Research Department of Engineering, University of Massachusetts Boston wenbinan@stu xjtu edu. en, { (fengtian, chenyan @mail 1. xjtu. cn shiyibai7788 gmaallmaalll. com, wangqyal lenovo. com, ping. chen@ edu
Abstract
want buy computer for business trips.
Discovering fine-grained categories from coarsely labeled data is practical and chal- lenging task, which can bridge the gap be- tween the demand for fine- mne-grained analysis and the high annotation cost. Previous works mainly focus instance-level discrimination to learn low-level features, but ignore seman- tic similarities between data, which may pre- vent these models learning compact cluster representations. In this paper, we propose Denoised Neighborhood Aggregation (DNA), a self-supervised framework that encodes se- mantic structures of data into the embedding space. Specifically, we retrieve k-nearest neigh- bors of query as its positive keys to capture semantic similarities between data and then aggregate information from the neighbors to learn compact cluster representations, which make fine-grained categories more sepa- ratable. However, the retrieved neighbors can be noisy and contain many calse-positive keys, which can degrade the quality of learned em- beddings. To cope with this challenge, we pro- pose three principles filter out these false neighbors for better representation learning. Furthermore, we theoretically justify that the learning objective of our framework is equiv- alent to clustering loss, which can capture semantic similarities between data form compact fine- mne-grained clusters. Extensive periments on three benchmark datasets show that our method can retrieve more accurate neighbors (21.31% % accuracy improvement) and outperform state-of-the-aa models by large margin (average 9.96% improvement on three metrics). Our code and data are available at httpa: / thub. com/ com/Lacke Lacke) /DNA.
Intent: Buy Computer
recommeed you
Campater
Speets
desitop.
Intent Buy Laptop
recommend you laptop.
Destup Lagtog Tomaks Getr
Figure 1: Left: An example of coarse- and fine-grained intent detection for recommendation. Right: Label hierarchy with coarse- and fine-grained categories.
Nauta al., 2021) and Natural Language Process- ing (Suresh and Ong, 2021; Vaid al., 2022; Mu- nikar et al., 2019; Almeida et al., 2021), since it can provide much more information than coarse- grained analysis. For example, detecting more fine- -gained user intents can help to provide more accurate recommendation and better services for customers (Figure 1 Left). However, labelling fine-grained categories can be time-consuming and labour-intensive since requires more expert knowledge. To get out of this dilemma, a novel task called Fine-grained Category Discovery under Coarse grrsse-graanddd supervision (FCDC) was recently proposed by An et al. (2022a). Taking Figure I Right as an example, FCDC aims at discover- ing fine-graines categories (e.g., Desktop and Ten- nis) using only coarse- -raiined (e.g., Computer and Sports) labeled data which are easier and cheaper to annotate.
To solve the FCDC task, previous methods mainly focus on instance-level discrimination to learn low-level features through contrastive learn- ing (An et al., 2022a; Bukchin et al., 2021). De- spite the improved performance, these instance- based methods fail to encode cluster-level semantic structures of data. This is because these methods simply treat each instance as a single class and push away other instances, regardless of their se-
1 Introduction
Many Al fields have progressed into fine-grained analysis, e.g., Computer Vision (Wei et al., 2021;
"Carresponding Author.

---

mantic similarities (Li al., 2020), which can hin- der the formation of compact fine-grained clusters. Here we define 'compact' as samples with the same fine-grained categories are compactly clustered into the center of category and away from other sam- plcs from different fine-grained categories, which means smaller intra-class distance and larger inter- class distance. Since samples located around deci- sion boundaries easily misclassified into wrong categories, distributing samples near the category center compactly can avoid overlapping decision boundaries and make these categories more distin- guishable. So learning compact cluster representa- tions is important for the FCDC task to learn more separable fine-grained categories.
ter representations, which essential for the FCDC task.
we propose Denoised Neighbor- hood Aggregation, self-supervised frame- work that captures semantic similarities be- tween data and aggregates information from neighbors. We further propose three princi- ples to filter out false neighbors for better rep- resentation learning.
Framework:
Theory: we interpret our framework from a generalized EM perspective and theoretically prove that the learning objective of our frame- work is equivalent to clustering loss. So our model can alternately retrieve more accu- rate neighbors and learn more compact cluster representations.
To encode semantic structures of data learn more compact cluster representations, we propose a novel model named Denoised Neighborhood Ag- gregation (DNA). DNA can capture semantic simi- larities between data by retrieving k-nearest neigh- bors of a query and aggregating information from them. However, the retrieved neighbors can be noisy and contain many false-positive keys (i.e.. keys with different fine- fine-grained categories from the query), which can reduce the quality of repre- sentation learning. This situation is more severe in the FCDC setting since pretraining on coarse- grained labels can easily include wrong neighbors for those samples with the same boarse-grained la- bels but different fine-grained ones. To solve this problem, we propose three principles (named La- bel Constraint, Reciprocal Constraint, and Rank Statistic Constraint) to filter out these false neigh- bors. These constraints consider bidirectional se- mantic structures and statistical features of data to help to retrieve more accurate neighbors. Further- more, we interpret our framework from a general- ized Expeetation-MMaximzznn
Experiments: Extensive experiments on three benchmark datasets show that our model es- tablishes state-of-the-art performance on the FCDC task (average 9.96% improvement) and retrieves more accurate neighbors (21.31% % ac- curacy improvement), which validates the- oretical analysis.
Related Work
2.1 Novel Category Discovery
Novel Category Discovery aims discovering novel categories from unlabeled data to expand ex- isting class taxonomy (Scheirer et al., 2014; Zhang et al., 2021a; Vaze et al., 2022; Yu et al., 2022; Badirli et al., 2023; An et al., 2023). To discover novel categories without any annotation, previous models usually adopted self-supervised methods. For example, Han et al. (2020) utilized ranking statistics as pseudo-labels train their model with binary cross-entropy loss. An al. (2022b) pro- posed to decouple known and novel categories from unlabeled data and performed representation learn- ing with prototypical network. However, these methods only focus on the scenario where known and novel categories are of the same granularity. To discover fine-grained categories, a novel task called Fine- sine-grrined Category Discovery under Coarse- grained supervision (FCDC) was proposed by An et al. (2022a). They also proposed a weighted self- contrastive strategy acquire fine-grained knowl- edge. And Mekala et al. (2021) proposed to per- form fine- -nai-gea text classification with the help
(EM) perspective. At the E-step, we retrieve reliable neighbors from a dynamic queue under the proposed constraints, then at the M-step, perform neighborhood ag- gregation to encode semantic structures of data to learn more compact representations. Last but not least, we theoretically prove that the learning objec- tive of our model is equivalent to a clustering loss, which can help to learn compact cluster representa- tions to facilitate fine-grained category discovery. Our main contributions can be summarized as follows:
Perspective: we propose to model semantic structures of data to learn more compact clus-

---

of fine-grained label names and coarse-grained la- beled data. In Computer Vision, Bukchin et al. (2021) proposed angular contrastive learning to perform few-shot fine-grained image classification with only coarse-grained supervision. However, these methods only focus on instance-level discrim- ination, which may prevent them from learning compact cluster representations for fine-grained category discovery.
Model performance will be measured on another testing set Diest { yr) 3: (ine ) through clustering (e.g., K-Means) based on the embeddings extracted by Fg- It should be noted that we only use the number of fine-grained cate- gories K when testing so that we can make fair comparison with different methods, following the settings in previous work (An et al., 2022a).
3.2 Proposed Approach
2.2 Contrastive Learning
To achieve the learning objective of FCDC, pro- pose Denoised Neighborhood Aggregation (DNA), an iterative framework to bootstrap model perfor- mance on retrieving reliable neighbors and learn- ing compact embeddings. As shown in Fig. 2, our model mainly contains three steps. Firstly, we maintain dynamic queue to retrieve neighbors for queries based on their semantic similarities (Sec. 3.2.1). Secondly, we propose three principles to filter out false-positive neighbors for better repre- sentation learning (Sec. 3.2.2). Thirdly, we per- form neighborhood aggregation to learn compact embeddings for fine-grained clusters (Sec. 3.2.3). Last but not least, we interpret our framework from the generalized EM algorithm perspective and theoretically prove that our learning objective is equivalent to a clustering loss, which can help to learn more compact fine-grained cluster embed- dings (Sec. 3.2.4).
Contrastive Learning (CL) performs representation learning by pulling similar samples closer and push- ing dissimilar samples far away (Chen et al., 2020). And how to build high-quality positive keys for the given queries is challenging task for CL. Most previous methods took two different transforma- tions of the same input as query and positive key, respectively (Dosovitskiy et al., 2014; Chen et al., 2020; He et al., 2020). Li et al. (2020) proposed to utilize prototypes learned by clustering as their pos- itive keys. Furthermore, An et al. (2022a) proposed use shallow features extracted by BERT as pos- itive keys. Recently, Neighbourhood Contrastive Learning (NCL) was proposed by treating the near- est neighbors of queries as positive keys (Dwibedi et al., 2021a), which can avoid complex data aug- mentations. Zhong et al. (2021) further utilized k-nearest neighbors to mine hard negative keys for CL. And Zhang et al. (2022a) randomly selected positive key from k-nearest neighbors for repre- sentation learning. Even though NCL has achieved better results on many tasks, previous methods ig- nored the fact that the retrieved neighbors can be noisy (i.e., neighbors and the query come from different categories) due to lack of supervision, and these false-positive keys can be harmful for representation learning since they provide wrong supervision signals.
3.2.1 Neighborhood Retrieval
We maintain a dynamic queue M to store sample features for subsequent training. The features in the queue are extracted by a momentum encoder Fgm- and are progressively updated at each iteration. To keep consistency of features used for neighborhood retrieval and representation learning, we update Fgm in moving-average manner (He et al., 2020):
0 a0.0 + a))0+
(1)
3 Method
where a E (0,1) is momentum coefficient, 0m and f are parameters of F&m and Fg, respectively. Fe is a query encoder for representation learning and is updated by back-propagation.
3.1 Problem Formulation
Given set of coarse-grained categories Decarse = C2., Car) and coarsely labeled training set Derain = [ G.) G E Veoarse) = the FCDC task aims at learning feature encoder Fg that maps samples into compact D-dimension embed- ding space to further separate them into different fine-grained categories fine = [2,, ..., F } even though without any prior finc- grainer knowl- edge, where Dime are sub-classes of Vevarse.
In order to learn compact representations, we retrieve neighbors of each query from the queue M. Specifically, we first pretrain Fg and Føm with cross-entropy loss on coarse-graine grained labels ini- tialize models. Then given a query embedding qi = ==F (2;), we search its k-nearest neighbors A from the queue M by measuring their semantic

---

Neighborbood Piltering
1. Label Constraint
Query A A
① Neighborheod
Agreetat ggregation
Encodor
2. Reciprocal Constraint
A djjacency matrix
Refined djacency
Momentum Update
Neighborhood Retrteval
matnx
0
1
DNA
Refine
0
Data
3. Rank Statistic Constraint
0.8 0.2 L.2
Top Ranks: (0,
Stop Gradient
L. 0.3 0.9
Momentum Encoder
Queue
Tup 2 Ranks:
0033 1.5 0.8
Racks:
Figure The overall architecture of our DNA framework.
similarities:
et al., 2011) 1) (i.e., the query should also be its neigh- bors" neighbor). This constraint is intuitive since samples in compact cluster should be neighbors to each other. Traditional k-NN simply retrieved neighbors for each sample, which can introduce many false neighbors, especially for data with long- tailed distribution. As the example of reciprocal constraint in Fig. 2, the blue sample should only have two neighbors, but 3-NN introduces a false neighbor (the green sample) for it. The reciprocal constraint can filter out this false neighbor since the blue sample is a neighbor of the green one. So the reciprocal constraint can capture bidirectional semantic structures between samples and provide different number of neighbors for different samples. And the refined neighbor set for query q is:
N₁ - {hy] hy E (sit, h())
(2)
hcM
where sim() is a similarity function and here we use cosine similarity sim((g; h2) -
of
3.2.2 Neighborhood Refining
After neighborhood retrieval, previous methods (Dwibedi et al., 2021a; Zhang et al., 2022a) sim- ply used these neighbors positive keys for con- trastive learning. However, they ignored the fact that the retrieved neighbors contain many false- positive keys (i.e., keys with different fine-grained categories from the query), which can significantly degrade their model performance. This is because we lack of fine-grained supervision and samples with different mne-grained categories can be clus- tered together after pretraining. To mitigate this problem, we propose three principles to filter out these false-positive neighbors, which consider bidi- rectional semantic structures and statistical features of samples (illustrated in Fig. 2).
R; (hy (hg A (a: E A,))
(4)
Rank Statistic Constraint requires the rank statis- tics between neighbors to be the same. Specifically, we rank the values of feature embeddings by magni- tude and extract the index of top-m values form rank set. Then we filter out neighbors who have different rank set from the query. The rank statis- tic constraint is effective for two reasons. Firstly, rank statistic more robust than cosine similarity, especially for high-dimensional data (Friedman, 1994; Han et al., 2020). Secondly, samples with the same coarse -raiined labels but different fine- grained ones can have high cosine similarities after pretraining, which can lead false-positive neigh- bors. However, we think the pretrained embeddings also contain information about fine- fine-grained cate- gories which are interrupted by other noisy informa- tion. So if we only focus on the main components
Label Constraint aims filtering neighbors with different coarse-en coarse-grained labels from the query, since samples with the same fine-grat -ne-gga labels must also have the same coarse-grained ones. The refined neighbor set for query is:
A = [hy (hg (N) A (;)}
(3)
where &; and ej are coarse-grained labels for the query qi and its neighbor hy. Reciprocal Constraint requires that the neighbor- hood relationships should be bidirectional (Qin

---

of these embeddings, we can filter out the noisy information and discover the hidden fine-grained information. The refined neighbor set for query qi is:
gory discovery.
(Align.)
Σ hJ)
7|D]
55
MED
Nye5:
T
S [hg 15.1(0 (hg E Ri)) A (topm(hj) = topm (4:))
7 [D]
qi 14i
IED
(5)
1
Σ RED
{(0 m1) - -1aa112 - 1lass 1?}
where lopm(4: (4:) maps the d-dimensional embed- ding 4i into mn-diensional set (m 2 in Fig. 2) which contains index of top-m values of g--
2r |r|
Σ RED
(g. pa)² +
2r
Σ EDD
(q: pa))
3.2.3 Denoised Neighborhood Aggregation
(7)
After mining reliable neighbors, we perform De- noised Neighborhood Aggregation (DNA) to pull queries and their neighbors closer by extending the traditional contrastive loss (Oord al., 2018) to the form with multiple positive keys:
where the symbol - indicates equal up to a multiplicative and/or an additive constant. μís
[. Σ hj is the center of query's 's neighbors. cy
1 = because of normalization, and || is constant since the neighbor embedding hy is from the queue without gradient.
expllt h//t)
Interpreting DNA from the generalized EM per- spective. If we consider the centers of the neigh- bors as hidden variables, we can interpret the DNA framework from the generalized EM perspective. At the E-step, we fix model parameters to find the hidden variables by retrieving and refining the neighbor set Sy:
LDNA
Σ
log Σ explal he/T)
D
S
hyES,
HEEM
hs/t)
D
K,ES
(Aligrement)
log explet hx/T)
{ui10, 9. Mii
h
(8)
(Unkformity)
S
(6)
hyES.
At the M-step, we fix the hidden variables { to optimize model parameters
where qi is the L2 normalized query embedding and hy is the L2 normalized key embedding from the queue M. Then we train the model with the loss LDNA and the cross-entropy loss LCE with the coarse-graines labels to learn compact cluster embeddings for FCDC.
argmni #
E(: mi)² WED
(9)
Since more accurate neighbors can boost repre- sentation learning and better representation learn- ing can help retrieve more accurate neighbors, our framework can iteratively bootstrap model per- formance on representation learning and neighbor- hood retrieval. In addition to the intuitive expla- nation, we also verify the intuition through experi- ments (Sec. 5.4).
3.2.4 Theoretical Analysis
Analysis for LDNA- The loss LDNA can be di- vided into two parts (Wang and Isola, 2020): Align- ment to pull queries and their neighbors closer, and Uniformity to make samples uniformly distributed in hyper-sphere. Then will prove that the Align- ment term in LDNA is equivalent to clustering loss that makes the query converge to the center of its neighbors, which can help to learn more com- pact cluster representations for fine- -mn-ind cate-
4
Experiments
4.1
Experimental Settings
4.1.1 Datasets
We conduct experiments on three benchmark
datasets. CLINC (Larson et al., 2019) is an intent

---

Dataset
ICI
IFF
# Train
#'est
where de the prediction from clustering and ya the ground-truth label, N is the number of sam- ples, P(-) is the permutation map function from Hungarian algorithm (Kuhn, 1955).
CLINC
10
150
18,000
1,000
WOS
7
33
8,362
2,420
HWU64
18
64
8,954
1,031
Table Statistics benchmark datasets. number of samples. ICI: number of coarse-graie graieed categories. IFFI: number of fine-grained categories.
4.2.1 Implementation Details
We use the pre-trained BERT-base model (Devlin et al., 2018) as our backhone with the learning rate 5e 5e-5 weight decay and 1.0 gradient clipping, For hyper- parameters, the batch size for pretraining, training and testing is set to 64. Epochs for pretraining and training are set to 100 and 20, respectively. The temperature is set to 0.07. The number of neigh- bors is set 1200, 120. 250) for the dataset CLINC, HWU64 and WOS, respectively. The di- mension for Rank Statistic Constraint set to 5. The momentum factor a is set to 0.99. For com- pared methods, we use the same BERT model as ours to extract features and adopt in their original papers for a fair comparison.
detection dataset from multiple domains. WOS (Kowsari et al., 2017) is paper classification dataset. HWU64 (Liu et al., 2021) is assis- tant query classification dataset. Statistics of the datasets are shown in Table
We use the AdamW optimizer with 0.01
4.1.2 SOTA Methods for Comparison
We compare our model with following methods. Baselines: BERT (Devlin et al., 2018) without fine- tuning and BERT under coarse-grained supervi- sion. Self-training Methods: DeepCluster (Caron et al., 2018) and DeepAligmed (Zhang et al., 202 Contrastive based Methods: SimCSE (Gao et al., 2021), Ancor (Bukchin et al., 2021), Delete (Wu al., 2020), Nearest-Neighbor Contrastive Learn- ing (NNCL) (Dwibedi al., 2021b), Contrastive Learning with Nearest Neighbors (CLNN) (Zhang et al., 2022b) and Weighted Seff-Contrastive Learn- ing (WSCL) (An et al., 2022a). We also investigate some variants by adding cross-entropy loss (+CE).
4.3 Result Analysis
Comparison results of different methods are shown in Table 2. From the results, we can get following observations. (1) Our model outperforms the com- pared methods across all evaluation metrics and datasets, which clearly shows the effectiveness of our model. we attribute the better performance of our model to the following reasons. Firstly, our model can model semantic structures of data learn more compact cluster representations by ag- gregating information from neighbors. Secondly, our model can retrieve reliable neighbors to boost the quality of representation learning with three (10) filtering principles. Thirdly, the previous two steps can boost performance of each other in an EM man- ner, which can progressively bootstrap the entire model performance. (2) Baselines and self-training methods perform badly on the FCDC task since they rely on abundant fine-grained labeled data to train their models, which are not available under the FCDC setting. (3) Contrastive-basse methods perform better than baselines above since they can acquire fine-grained knowledge grained supervision. However, these methods sim- ply treat each instance single class but ignore semantic similarities between different instances, which may prevent them from learning compact representations for subsequent clustering.
4.2 Evaluation Metrics
To evaluate the quality of the discovered fine- grained clusters, we use two broadly used evalua- tion metrics: Adjusted Rand Index (ART) (Hubert and Arabie, 1985) and Normalized Mutual Infor- mation (NMT) (Lancichinetti et al., 2009):
- - E(R) max max(RI- E(RI)
ARI
NMI 2x I(r:n)
(11)
H()) H(y)
where RI is the rand index and E(RT) is the ex- pectation of RT. ÿ is the prediction from clustering and y is the ground truth. I [ v) is the mutual information between ÿ and y. (T) and Hvv) rep- resent the entropy of ý and y. respectively.
without fine-
To evaluate the classification performance of models, we use the metric clustering accuracy (ACC):
ACC L1=1 H(P())) = :i
(12)
N

---

CLINC
WOS
HWU64
Methods
ACC
ARI
NMI
ACC
ARI
NMI
ACC
ARI
NMI
BERT
34.37
17.61
64.75
31.97
18.36
45.15
33.52
17.04
56.90
BERT + CE
43.85
32.37
78.58
38.29
36.94
64.72
37.89
33.68
74.63
DeepCluster 26.40
12.51
61.26
29.17
18.05
43.34
29.74
13.98
53.27
DeepAligned
29.16
14.15
62.78
28.47
15.94
43.52
29.14
12.89
52.99
DeepCluster + CE
30.28
13.56
62.38
38.76
35.21
60.30
41.73
27.81
66.81
DeepAligned + CE
42.09
28.09
72.78
39.42
33.67
61.60
42.19
28.15
66.50
NNCL
17.42
13.93
67.56
29.64
28.51
61.37
32.98
30.02
73.24
CLNN
19.96
14.76
68.30
29.48
28.42
60.99
37.21
34.66
75.27
SimCSE
40.22
23.57
69.02
25.87
13.03
38.53
24.48
8.42
46.94
Ancor CE
44.44
31.50
74.67
39.34
26.14
54.35
32.90
30.71
74.73
Ancor
45.60
33.11
75.23
41.20
37.00
65.42
37.34
34.75
74.99
Delete
47.11
31.28
73.39
24.50
11.68
35.47
21.30
6.52
44.13
Delete + CE
47.87
33.79
76.25
41.53
33.78
61.01
35.13
31.84
74.88
SimCSE + CE
52.53
37.03
77.39
41.28
34.47
61.62
34.04
31,81
74.86
WSCL
74.02
62.98
88.37
65.27
51.78
72.46
59.52
49.34
79.31
Ours
87.66
81.82
94.69
74.57
63.30
76.86
70.81
59.66
83.31
Improvement
+13.64
+18.84
+6.32
49.30
+11.52
+4.40
+11.29
+10.32
+4.00
Table 2: Model performance (%) on the FCDC task. Average results over runs are reported. Some results are cited from An (2022a).
5 Discussion
Model
ACC
ARI
NMI
5.1 Ablation Study
NNCL
32.98
30.02
73.24
Multi. Neighbors
64.89
52.88
81.14
We investigate the effectiveness of difffrent com-
Coarse Labels
68.19
55.95
81.90
ponents of our model in Table 3. From the table we can draw following conclusions. (1) Traditional Nearest-Neighbor Contrastive Learning (NNCL) performs badly the FCDC task, which is be- cause NNCL ignores semantic similarities between samples and fails to retrieve reliable neighbors. (2)
Label
68.67
57.08
81.99
Reciprocal
69.42
58.16
82.99
Rank (Ours)
70.81
59.66
83.31
Table 3: Results (%) different model variants on the HWU64 dataset. "+' means that add the component to the previous model.
Adding multiple neighbors as positive keys (Eq. 6) significantly improves model performance since can help to Icarn compact cluster representations for fine-gras gree-gr categories (Sec. 3.2.4). (3) Adding coarse-grained supervision with cross entropy loss also boost model performance since it can con- tribute to representation learning. (4) Adding dif- ferent filtering principles (Label, Reciprocal and Rank) can also improve model performance since they are responsible to retrieve reliable neighbors for better representation learning.
bors have the same fine- fine-grained labels) of the re- trieved neighbors in Table Specifically, (I) re- trieving neighbors without any filtering (k-NN) can yield only 50% retrieval accuracy, which can signif- icantly affect subsequent learning since those false neighbors provide wrong supervision signal for rep- resentation learning. (2) Adding the coarse -raiined label constraint (Label) improve the retrieval accuracy slightly since k-NN has also learned the gearse-grained knowledge through pretraining. (3) Adding the reciprocal constraint (Reciprocal) can make huge improvement retrieval accuracy since this constraint can capture bidirectional relation-
5.2 Accuracy of Selected Neighbors
To investigate the effect of three filtering princi- ples on the accuracy the retrieved neighbors, we report the accuracy (i.e., the query and neigh-

---

Accuracy of retrieved neig heigh
2.00
90-
number of heligh
1.75
80-
1.50
Metrics (%6
70-
1.25
60
1.00
50-
0.75
ACC
0.0
ARI
NMI
30
14
Epoch
Epoch
Epoch
(a) Accuracy selected neighbors.
(b) The number selected neighbors.
(c) Model performance during training.
Figure 3: Analysis of the quality of representation learning and neighborhood retrieval.
Model
WOS
HWU64
CLINC
k-NN
49.39
49.65
48.08
+Label
50.43
50.60
48.24
85
+Reciprocal
67.57
65.00
66.78
Metrics
+Rank
77.34
66.44
67.28
Improvement
+27.95
+16.79
+19.20
70
ACC
Table Accuracy (%) of retrieved neighbors.
30
60
120
150
180
ships between queries and the retrieved neighbors. (4) Adding the rank statistic constraint (Rank) can also improve the retrieval accuracy. After pretrain- ing, samples with the same coarse-grained labels but different fine-grained ones be aggregated together, leading to retrieving false neighbors. The rank statistic constraint can alleviate this problem by ignoring noisy components in feature embed- dings and only considering important ones that con- tain fine-grained information, where the important components are selected by ranking statistics.
Figure 4: Influence of the number of neighbors k.
trieval are performed alternately bootstrap both of their performance.
5.4 EM Validation
To validate the intuitive EM explanation about our framework (Sec. 3.2.4), we further visualize the curve of model performance during training on the CLINC dataset in Fig. 3(c). From the figure we can that our model gets better and better per- formance during training, which indicates that the quality of representation learning is gradually im- proved. Combined with the improved quality and quantity of the selected neighbors Fig. 3(a) and 3(b), we can draw the conclusion that our frame- work can bootstrap model performance on repre- sentation learning and neighborhood retrieval itera- tively through the generalized EM perspective.
5.3 Quantity and Quality Trade-off'
While the filtering principles can increase the qual- ity of neighbors by filtering out noisy ones, they can also reduce the number retrieved neighbors. To investigate the trade-off between quantity and quality, we plot the curve of accuracy of selected neighbors (Fig. 3(a)) and the number of selected neighbors (Fig. 3(b)) on the CLINC dataset. From the figure can that our model can retrieve and more neighbors with increasing accuracy during training, which means that our model can reach trade-off between quantity and quality. We attribute this advantage to our learning paradigm where representation learning and neighborhood re-
5.5 Influence of Number of Neighbors
We investigate the influence of the number of neigh- bors on model performance the CLINC dataset in Fig. From the figure we can see that too large or too small & can lead to poor model performance. we can also see that our model gets the best per-

---

92
ACC
Metrics (%)
90
ARI
88
86
(a) WSCL
(b) DNA
84
82
Figure 6: The t-SNE visualization of embeddings.
100
400
600
768
Figure 5: Influence of the number of rank dimensions.
6 Conclusion
In this paper, we propose Denoised Neighbor- hood Aggregation (DNA), a self-supervised learn- ing framework that iteratively performs represen- tation learning and neighhorhood retrieval for fine- grained category discovery. We further propose three principles to filter out false-positive neigh- bors for better representation learning. Then we interpret our model from a generalized EM per- spective and thcoretically justify that the learning objective of our model is equivalent to clustering loss, which can encode semantic structures of data to form compact clusters. Extensive experiments three benchmark datasets show that model can retrieve more accurate neighbors and outper- form state-of-the-a models by large margin.
formance when is approximately equal to the number of samples in each fine-grained category (e.g., 120 for the CLINC dataset). This is in line with our analysis in Sec. 3.2.4, since samples will gather into the center of fine-grained clusters in the ideal situation.
5.6 Influence of Number of Rank Dimensions
We investigate the influence of the number rank dimensions m on the CLINC dataset in Fig. 5. From the figure we can see that our model is in- sensitive to changes in m, since we only use the rank statistic constraint to select reliable neighbors for the first epoch of training. This is because rank statistic constraint strong constraint, which is most effective when the model performance poor. And during training, our model can select more accurate neighbors, so we want to retrieve more neighbors to improve model's ' generalization abil- ity. Furthermore, since sorting is come-connuming process, performing rank statistic constraint only at the beginning of training save much time and balance effectiveness and efficiency.
Limitations
Even though the proposed Denoised Neighborhood Aggregation (DNA) framework achieves superior performance the FCDC task, it still faces the following limitations. Firstly, DNA requires addi- tional memory to store a queue for neighborhood trieval and representation learning. Secondly, even though the three filtering principles can help to select more accurate neighbors, they require addi- tional time to post-process the retrieved neighbors than traditional k-NN methods.
5.7 Visualization
A Acknowledgments
We visualize the learned embeddings of WSCL (An et al., 2022a) and our model through t-SNE on the HWU64 dataset Fig. 6. From the figure we can see that our model can separate different coarse- grained categories effectively. In the meanwhile, our model can maintain separability within each coarse-grained categories to facilitate the subse- quent fine-grained category discovery. In summary, our model can better control both inter-class and intra-class distance between samples facilitate the FCDC task than previous instance-level con- trastive learning methods.
This work was supported by National Key Research and Development Program of China 22022DD0111022 National Natural Science Foun- dation of China (62293551, 62177038, 62277042, 62 17702,2 61721002, 61937001, 62377038). In- novation Research Team of Ministry of Educa- tion [IRT_7R886 Project of China Knowledge Centre for Engineering Science and Technology, "LENOVO-XITU" Intelligent Industry Joint Labo- ratory Project

---

References
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson. Pierre Sermanet, and Andrew Zisserman. 2021b. With a little help from my friends: Nearest-neighhor contrastive learning of visual representations. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pages 9588-9597.
Matthew Almeida, Yong Zhuang, Wei Ding, Scott E Crouter, and Ping Chen. 2021. Mitigating class- boundary lahel uncertainty to reduce both model bias and variance. ACM Transactions on Knowledge Dis- covery from Data (TKDD), 15(2)1-18.
Jerome H Friedman. 1994. An overview of predictive learning and function approximation. Springer.
Wenbin An, Feng Tian, Ping Chen, Siliang Tang, Qinghua Zheng, and YinnYing Wang. 2022a. Fine- grained category discovery under coarse-grained su- pervision with hierarchical weighted self-contrastive learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simese: Simple contrastive learning of sentence em- beddings. arXiv preprint arXiv: (ete22104088822
Kai Han, Sylvestre- Alvise Rebuffi, Sebastien Ehrhardt, Andrea Vedaldi, and Andrew Zisserman. 2020. Au- tomatically discovering and learning new visual categories with ranking statistics. arXiv preprint arXiv:2002.05714.
Wenbin An, Feng Tian, Ping Chen, Qinghua Zheng, and Wei Ding. 2023. New user intent discovery with robust pseudo label training and source domain joint- training. IEEE Intelligent Systems.
Wenbin An, Feng Tian, Qinghua Zheng, Wei Ding, QianYing Wang, and Ping Chen. 2022b. General- ized category discovery with decoupled prototypical network. arXiv preprint arXiv:2211.1. 151155.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum contrast for un- supervised visual representation learning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729-9738.
Sarkhan Badirli, Christine Johanna Picard, George Mohler, Frannie Richert, Zeynep Akata, and Murat Dundar. 2023. Classifying the unknown: Insect iden- tification with deep hierarchical bayesian learning. Methods in Ecology and Evolution.
Lawrence Hubert and Phipps Arabie. 1985. Comparing partitions. Journal of classification, 211):193224 218.
Kamran Kowsari, Donald E Brown, Mojtaba Hei- darysafa, Kiana Jafari Meimandi, Matthew S Gerber, and Laura E Barnes. 2017. Hdltex: Hierarchical deep learning for text classification. In Machine Learning and Applications (ICMLA), 2017 16th IEEE Interna- rional Conference on. IEEE.
Guy Bukchin, Eli Schwartz, Kate Saenko, Ori Shahar, Rogerio Feris, Raja Giryes, and Leonid Karlinsky. 2021. Fine-grained angular contrastive learning with coarse labels. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Patlern Recognition, pages 8730-8740.
Harold W Kuhn. 1955. The hungarian method for the assignment problem. Naval research logistics quar- terly, 2:83-97.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. 2018. Deep clustering for unsuper- vised learning of visual features. In Proceedings of the European Conference Computer Vision (ECCV), pages 149.
Andrea Lancichinetti, Santo Fortunato, and János Kertész. 2009. Detecting the overlapping and hi- erarchical community structure in complex networks. New journal of physics, 1 11())033015.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. simple framework for contrastive learning of visual representations. In In- ternational conference machine learning, pages 1597-1607. PMLR.
Stefan Larson, Anish Mahendran, Joseph Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K Kummerfeld, Kevin Leach, Michael A Laurenzano, Lingjia Tang, et al. 2019. An evalua- tion dataset for intent classification and out-of-scope prediction. arXiv preprint 190.:.:99900002....
Jacob Devlin, Ming-Wei Chang, Kenton Lec, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arxiv preprint 181.0180000400:
Junnan Li, Pan Zhou, Caiming Xiong, and Steven CH Hoi. 2020. Prototypical contrastive learning of unsupervised representations. arXiv preprint DrXiv:2005.04900
Alexey Dosovitskiy, Jost Tobias Springenberg. Martin Riedmiller, and Thomas Brox. 2014. Discrimina- tive unsupervised feature learning with convolutional neural networks. Advances in neural information processing systems, 27:766-774.
Xingkun Liu, Arash Eshghi, Pawel Swietojanski, and Verena Rieser. 2021. Benchmarking natural language understanding services for building conversational agents. In Increasing Naturalness and Flexibility in Spoken Dialogue Interuction. Springer.
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. 2021a. With a little help from my friends: Nearest-neigghor contrastive learning of visual representations. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pages 9588-9597.
Dheeraj Mekala, Varun Gangal, and Jingho Shang. 2021. Coarse2fine: Fine-grained text classification on coarsely-grained annotated data, arXiv preprint arXiv:2109.10856

---

Manish Mumikar, Sushil Shakya, and Aakash Shrestha. 2019. Fine Fine-grained grained sentiment classification using bert. In 2019 Artificial Intelligence for Transforming Business and Sociery (AITB), volume pages 1-5. LEEE.
Hanlei Zhang, Hua Xu, Ting-En Lin, and Rui Lyu. 2021a. Discovering new intents with deep aligned clustering. In Proceedings of the AAAI Conference on Artificial Intelligence.
Hanlei Zhang, Hua Xu, Ting-En Lin, and Rui Lyu. 2021b. Discovering new intents with deep aligned clustering. In Proceedings of the AAAI Conference on Artificial Intelligence.
Meike Nauta, Ron Van Bree, and Christin Seifert. 2021. Neural prototype trees for interpretable fine-grained image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 1993-11493..
Yuwei Zhang, Haode Zhang, Li-Ming Zhan, Xiao-Ming Wu, and Albert Lam. 2022a. New intent discov- ery with pre- tre-training and contrastive learning. arXiv preprint arXiv:2205. 129.1
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint
Yuwei Zhang, Haode Zhang, Li-Ming Zhan, Xiao-Ming Wu, and Albert Lam. 2022b. New intent discov- ery with pre- training and contrastive learning. arXiv preprint arXiv:2205. 12944
Danfeng Qin, Stephan Gammeter, Lukas Bossard, Till Quack, and Luc Van Gool. 2011. 1. Hello neighbor: Accurate object retrieval with k-reciprocal nearest neighbors. In CVPR 1, pages 777-784. IEEE.
Zhun Zhong, Enrico Fini, Subhankar Roy, Zhiming Luo, Elisa Ricci, and Nicu Sebe. 2021. Neighborhood contrastive learning for novel class discovery. In Pro- ceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 10867-10875.
Walter J Scheirer, Lalit P Jain, and Terrance Boult. 2014. Probability models for open set recognition. IEEE transactions on pattern analysis and machine intelligence, 36(1 1:22222-3224.
Varsha Suresh and Desmond C Ong. 2021. Not all negatives are equal: Label-aware contrastive loss for fine grained text classification. arXiv preprint arXiv:2109.05227.
Roopal Vaid, Kartikey Pant, and Manish Shrivastava. 2022. Towards fine-grained classification of climate change related social media text. In Proceedings of the 60th Annual Meeting of the Association for Com- putational Linguistics: Student Research Workshop, pages 434 443.
Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zis- serman. 2022. Generalized category discovery. In Proceedings of the IEEE/CVF Conference Com- puter Vision and Pattern Recognition, pages 7492- 7501.
Tongzhou Wang and Phillip Isola. 2020. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. International Conference on Machine Learning, pages 9929-9939. PMLR.
Xiu-Shen Wei, Yi-Zhe Song, Oisin Mac Aodha, Jianxin Wu, Yuxin Peng, Jinhui Tang, Jian Yang, and Serge Belongie. 2021. Fine-grained image analysis with deep learning: A survey. IEEE transactions pat- tern analysis and machine intelligence, 44(12):8927- 8948.
Zhuofeng Wu, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. 2020. Clear: Contrastive learning for sentence representation. arXiv preprint arXiv:2012.F 15466.
Qing Yu, Daiki Ikami, Go Trie, and Kiyoharu Aizawa. 2022. Self-labeling framework for novel category discovery over domains. In Proceedings of the AAAI Conference on Artificial Intelligence.


Reviews:
[{"review_id": "NrMIuCUyQm", "reviewer": "Reviewer_1nk5", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive comments and suggestions. The responses to your concerns are listed below. \n\n[Q1] What is the definition of \"compact\" representation? Why more compact cluster representations mean more separable fine-grained categories?\n\n[R1] As analyzed in Section 3.2.4, we define ‘compact’ as samples with the same category are compactly clustered into the center of category and away from samples from different categories, which means smaller intra-class distance and larger inter-class distance. Since samples distributed around decision boundaries are easily misclassified into other categories, distributing samples near the category center compactly can avoid overlapping decision boundaries of different categories and make these categories more separable. Similar observations have also been made by other researchers such as the phenomenon of Neural Collapse [1].\n\n\n[Q2] How to solve the situation that no neighbors left after filtering by three principles?\n\n[R2] We solve this problem by ‘back off’ strategy, which means that we can  implement only parts of the filtering principles rather than all of them  to make sure at least some neighbors are left. Empirically we found that the case of no neighbors is unlikely to happen, especially after the first epoch as the number of neighbors is usually large (Fig. 3(b) in the paper). We will add this detail in the revised version.\n\n[Q3] The features in query and the features in queue are extracted from different encoder, why the similarity between them can be directly computed?\n\n[R3] Since features in the queue are updated during training, we use the slowly changing momentum encoder to update features in the queue to avoid these features to differ too much or sudden when neighborhood retrieval (before training) and representation learning (during training), especially at the last few iterations of an epoch, where the momentum encoder can make the training more stable. Furthermore, both encoders are initialized identically and the momentum encoder is updated according to the parameters of the encoder (Eq. 1 in the paper), so the difference between the two encoders is minimal. The momentum encoder techniques have been proven effective and widely used in both CV [1] and NLP [2]. Furthermore, we also perform experiments to validate the effectiveness of the momentum encoder.\n| Dataset | w/ momentum | w/o momentum |\n| ---- | --- | --- |\n| CLINC | {ACC: 87.66, ARI: 81.82, NMI: 94.69} | {ACC: 86.44, ARI: 80.22, NMI: 93.26} |\n| WOS |  {ACC: 74.57, ARI: 63.30, NMI: 76.86} | {ACC: 74.28, ARI: 61.77, NMI: 75.90} |\n| HWU64 | {ACC: 70.81, ARI: 59.66, NMI: 83.31} | {ACC: 68.87, ARI: 57.86, NMI: 82.46} |\n\n[Q4] In Table 2, Ancor -> Anchor.\n\n[R4] Thanks for pointing out this typo, we will correct it in the revised version.\n\nReferences\n\n[1] Kothapalli V, Rasromani E, Awatramani V. Neural collapse: A review on modelling principles and generalization[J]. arXiv preprint arXiv:2206.04041, 2022.\n\n[2] He K, Fan H, Wu Y, et al. Momentum contrast for unsupervised visual representation learning[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020: 9729-9738.\n\n[3] An W, Tian F, Chen P, et al. Fine-grained Category Discovery under Coarse-grained supervision with Hierarchical Weighted Self-contrastive Learning[C]//Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022: 1314-1323.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper is well-written and is easy to follow.\n- Compared with SOTA methods, the experimental results of the proposed framework show large improvements, which demonstrates the effectiveness of DNA.\n- The proposed three principles to filter out false-positive neighbors are insightful.", "reasons_to_reject": "- In lines 071-074, what is the definition of \"compact\" representation? Why more compact cluster representations mean more separable fine-grained categories?\n- How to solve the situation that no neighbors left after filtering by three principles?\n- The features in query and the features in queue are extracted from different encoder, why the similarity between them can be directly computed?\n- In Table 2, Ancor -> Anchor", "questions_to_authors": "", "paper_topic_and_main_contributions": "- This paper focuses on the task of FCDC.\n- This paper proposes a novel framework for FCDC, called DNA.\n- The proposed framework is interpreted from a generalized EM perspective, and is proved that the learning object is equivalent to a clustering loss.\n- The experimental results show that the proposed framework outperforms other methods on three benchmark datasets.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VgJvpKRLnK", "reviewer": "Reviewer_Mz1B", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive comments and suggestions. The responses to your concerns are listed below.\n\n[Q1] The paper's experiments should aim to conduct a broader range of ablation studies on multiple datasets. \n\n[R1] Thanks for your suggestions, we only report results on one dataset due to the page limit. We have added ablation studies on other two datasets. The results are listed below.\n\n**CLINC Dataset**\n| Model | ACC | ARI |NMI |\n| --- | --- |  --- |  --- |\n| NNCL |                   17.42 | 13.93 | 67.56 |\n| \\+ Multi. Neighbors |       85.31 | 78.33 | 92.76 |\n| \\+ Coarse Labels  |        85.69 | 80.23 | 93.08 |\n| \\+ Label  |               86.89 | 81.53 | 94.13 |\n| \\+ Reciprocal  |           87.46 | 81.29 | 94.09 |\n| \\+ Rank (Ours)  |          87.66 | 81.82 | 94.69 |\n\n**WOS Dataset**\n| Model | ACC | ARI |NMI |\n| --- | --- | --- | --- |\n| NNCL |                    29.64 | 28.51 | 61.37 |\n| \\+ Multi. Neighbors |        71.02 | 61.32 | 75.18 |\n| \\+ Coarse Labels |           72.23 | 62.02 | 75.89 |\n| \\+ Label |                  72.85 | 61.50 | 75.89 |\n| \\+ Reciprocal |              73.93 | 62.41 | 76.61 |\n| \\+ Rank (Ours) |             74.57 | 63.30 | 76.86 |\n\nFor other experiments (e.g., Fig 3, 4, 5), we have also performed experiments on other two datasets, and the results demonstrate the similar trend as the results reported in the paper, which can validate the conclusions drawn in our paper and show effectiveness and generalizability of our model. However, we cannot show these images here because of the textual constraints of openreview platform. We will add these details and conclusions in the revised version.\n\n[Q2] The paper's visual experiments should incorporate control groups. \n\n[R2] Thanks for your suggestions, we only report our visual results due to the page limit. We have added visual experiments compared to existing SOTA approaches (WSCL and SimCSE + CE). Compared to these methods, our model can learn more separable fine-grained clusters, which is consistent with the clustering metrics in Table 2. Limited by the openreview platform, we cannot show these images during rebuttal, but we will include these control groups in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The paper introduces a fresh perspective by advocating the modeling of semantic structures in data for learning more compact cluster representations. \n2) The proposed Denoised Neighborhood Aggregation framework presents a novel self-supervised approach for capturing semantic similarities between data points and aggregating information from neighboring data points. \n3) The paper provides a solid theoretical foundation by interpreting the proposed framework from a generalized Expectation-Maximization perspective. The theoretical proof, which establishes the equivalence of the learning objective to a clustering loss, adds credibility to the proposed approach and its potential impact on NLP research.\n4) The FCDC task and representation learning are crucial components of many NLP applications. By improving the efficiency and accuracy of clustering methods, the paper's contributions can have broad applications, such as document clustering, topic modeling, sentiment analysis, and information retrieval, which are of utmost importance in NLP research and applications.\n5) The proposed principles for filtering out false neighbors and the theoretical interpretation from an EM perspective can serve as a basis for further research and inspire the development of new clustering and representation learning techniques in NLP.", "reasons_to_reject": "1) The paper's experiments should aim to conduct a broader range of ablation studies on multiple datasets. Currently, the experiments seem to be limited to only one or a few datasets, which may lead to overfitting and limit the generalizability of the proposed approach to other real-world scenarios.\n2) The paper's visual experiments should incorporate control groups. Presenting results without control groups may lead to difficulties in interpreting the effectiveness of the proposed method compared to existing or alternative approaches. The inclusion of control groups helps establish a solid basis for making valid conclusions.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a novel approach to address the problem of learning compact cluster representations for the Fully Connected Dense Clustering task. The paper addresses the problem of learning compact cluster representations for the FCDC task. It proposes a novel framework, Denoised Neighborhood Aggregation, and provides a theoretical foundation for its effectiveness. The experimental evaluations demonstrate that the proposed method outperforms existing approaches in both the FCDC task and neighbor retrieval accuracy, which further validates the theoretical claims and establishes the paper's contributions to the field.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "TZe2i2TAp7", "reviewer": "Reviewer_Sqwo", "summary": "", "questions": "A: What is the impact of the queue size and momentum update coefficient on model performance?\nB: Can authors try to validate the argument in L62-64 with some theoretical or experimental  evidence? \"Despite the improved performance, these instance-based methods fail to encode cluster-level semantic structures of data\"\nC: Can you provide more ablation results to demonstrate the individual impact of the 3 filtering principles?\nD: Could you please elaborate why \"uniform\" part in objective function is important?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive comments and suggestions. The responses to your main concerns are listed below.\n\n[Q1] The proposed ideas like loss function, dynamic queue and nearest neighbor help, are updated yet incremental improvements over prior work like nearest neighbor contrastive learning. That being said, it is good to see successful implementations from cross-domains (CV to NLP).\n\n[R1] Our model is based on the nearest neighbor contrastive learning, and our novelty lies in following three aspects.\n+ We analyze the problems of existing FCDC methods and extend the traditional nearest neighbor contrastive learning to multiple neighbors to solve the problem, and the multiple-neighbor strategy is effective for FCDC (please refer to Table 3 in the paper).\n+ We theoretically prove that the learning objective of our multiple-neighbor contrastive learning is equivalent to a clustering loss, which provides a theoretical foundation for why our model can learn compact cluster representations.\n+ We propose three principles to filter out false neighbors for better representation learning, where the quality of neighbors is not addressed by pervious methods.\n\n[Q2] Additional memory is required to store the queue for neighborhood retrieval. Filtering principles increase training time.\n\n[R2] Our model can minimize the impact of memory usage by maintaining a smaller queue without losing much model performance (please refer to [R4]). As for training time, we report the compared training time in Table 6 in our paper, and the additional training time is minimal. And our model can further reduce the training time by updating neighborhood relations every five epochs and get comparable training efficiency as previous methods (85.2 s/epoch vs. 67.0 s/epoch).\n\n[Q3] More details on hyper-parameter choice/tuning would be helpful to better analyze the sensitivity of the model to different parameter choices, like k , rank dimensions etc.\n\n[R3] We included experiments and analysis of some hyperparameters in our original paper (e.g., k in Section 5.5 and Fig. 4, rank dimensions in Appendix A.5 and Fig. 7), as well as the impact of queue size and momentum coefficient in [R4] below.\n\n\n[Q4] What is the impact of the queue size and momentum update coefficient on model performance?\n\n[R4] For the impact of queue size, we conducted experiments with different ratio of data on the HWU64 dataset, and the results are shown below.\n\n| Ratio | ACC | ARI | NMI |\n| ---- | --- | --- | --- |\n| 30% | 65.16 | 55.38 | 81.98 |\n| 50% | 68.52 | 57.93 | 82.31 |\n| 70% | 70.62 | 60.00 | 83.01 |\n| 90% | 69.55 | 58.36 | 82.61 |\n| 100% | 70.81 | 59.66 | 83.31 |\n\nFrom the results we can see that our model can get comparable performance even with a smaller queue.\n\nWe also conducted experiments for momentum coefficient on the HWU64 dataset, and the results are shown below.\n\n| Coefficient | ACC | ARI | NMI |\n| ---- | --- | --- | --- |\n| 0 (w/o momentum) | 68.87 | 57.86 | 82.46 |\n| 0.3 | 70.16 | 59.96 | 83.21 |\n| 0.5 | 69.98 | 59.84| 83.28 |\n| 0.7 | 69.56 | 59.99 | 84.40 |\n| 0.9 | 68.95 | 58.75 | 82.55 |\n| 0.99 | 70.81 | 59.66 | 83.31 |\n\nFrom the results we can see that our model is insensitive to the momentum coefficient, which can show the robustness of our model.\n\n\n[Q5] Can authors try to validate the argument in L62-64 with some theoretical or experimental evidence? \"Despite the improved performance, these instance-based methods fail to encode cluster-level semantic structures of data\"\n\n[R5] We can validate the argument with both theoretical and experimental evidence.\n+ Theoretically, [1] proves that compared to instance features, class centers (prototypes) have  more mutual information with the class labels, which means that pulling samples closer to class centers (prototypes) can help to capture cluster-level semantics between data with the same fine-grained category and learn better cluster representations. Furthermore, our theoretical analysis in Sec. 3.2.4 also proves that our model can capture semantic structures between data by pulling samples with high semantic similarity closer and clustering these samples to form compact clusters.\n+ Experimentally, we have conducted experiments to validate the argument. Since good feature representations require samples of the same category to have high semantic similarity and samples of different categories to have low semantic similarity, we define a score as average semantic similarity between samples from the same fine-grained category **minus** average semantic similarity between samples from different fine-grained categories. A higher score indicates that the model can better capture semantic similarities between data. The results on three datasets are shown below.\n\n| Dataset | WSCL | Ours |\n| ---- | --- | --- |\n| WOS   | 0.4090 | 0.5564 |\n| CLINC  | 0.5676 | 0.7197 |\n| HWU64 | 0.5732 | 0.5897 |\n\nFrom the results we can see that our model gets higher scores on all datasets, which means that our model can better capture semantic relationships between data compared to previous SOTA methods.\n\n\n[Q6] Can you provide more ablation results to demonstrate the individual impact of the 3 filtering principles?\n\n[R6] For ablation study in Table 3, we conducted individual impact experiments. We define the multi-neighbor contrastive learning with coarse supervision as the base model.\n\n| Model | ACC | ARI | NMI |\n| ---- | --- | --- | --- |\n| Base             | 68.19 | 55.95 | 81.90 |\n| Base + Label      | 68.67 | 57.08 | 81.99 |\n| Base + Reciprocal  | 68.81 | 57.64 | 82.35 |\n| Base + Rank       |70.11 | 60.10 | 83.22 |\n\nFrom the results we can see that individual filtering principle can also improve our model performance, which can show the individual effectiveness of our filtering principles.\n\nFor accuracy of retrieved neighbors in Table 4, we also conducted individual impact experiments.\n\n| Model | WOS | HWU64 | CLINC |\n| ---- | --- | --- | --- |\n| k-NN             | 49.39 | 49.65 | 48.08 |\n| k-NN + Label      | 50.43 | 50.60 | 48.24 |\n| k-NN + Reciprocal  | 66.69 | 65.41 | 66.32 |\n| k-NN + Rank        |72.10 | 60.12 | 55.22 |\n\nFrom the results we can see that individual filtering principle can also improve the accuracy of retrieved neighbors, which means that our filtering principles are effective. Overall, our filtering principles are effective, even when using the individual filtering principle.\n\n\n[Q7] Could you please elaborate why \"uniform\" part in objective function is important?\n\n[R7] The \"uniform\" part corresponds to negative keys in contrastive learning. As theoretically and experimentally demonstrated in [2], the \"uniform\" part is responsible for preserving maximal information of data and making features uniformly distributing on the feature hypersphere, which is vital for learning good representations. Without this term, all sample features will collapse to a single point in the feature hypersphere to maximize the alignment part, which can lead the model to fail to learn any useful representations.\n\n[Q8] Missing References\n\n[R8] Thanks for your suggestions, we will add these references in the revised version.\n\n\n\nReferences\n\n[1] Li J, Zhou P, Xiong C, et al. Prototypical Contrastive Learning of Unsupervised Representations[C]//International Conference on Learning Representations. 2020.\n\n[2] Wang T, Isola P. Understanding contrastive representation learning through alignment and uniformity on the hypersphere[C]//International Conference on Machine Learning. PMLR, 2020: 9929-9939.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Authors tackle an important and practical problem of discovering fine-grained categories from coarsely labeled data.\n•\tThe three principles for filtering false neighbors are reasonable and effective as evidenced by improved neighbor accuracy and reflected performance boost on clustering metrics.\n•\tTheir simple theoretical analysis provides good justification and intuition behind the objective function for alignment part. Interpretation as generalized EM is insightful.\n•\tStrong experimental results across three datasets demonstrate effectiveness of the model, significantly outperforming prior state-of-the-art.\n•\tThe problem being addressed is relevant, with applications in areas like computer vision and NLP.\n", "reasons_to_reject": "•\tThe proposed ideas like loss function, dynamic queue and nearest neighbor help, are updated yet incremental improvements over prior work like nearest neighbor contrastive learning. That being said, it is good to see successful implementations from cross-domains (CV to NLP)\n•\tAdditional memory is required to store the queue for neighborhood retrieval. Filtering principles increase training time.\n•\tMore details on hyper-parameter choice/tuning would be helpful to better analyze the sensitivity of the model to different parameter choices, like k , rank dimensions etc.", "questions_to_authors": "A: What is the impact of the queue size and momentum update coefficient on model performance?\nB: Can authors try to validate the argument in L62-64 with some theoretical or experimental  evidence? \"Despite the improved performance, these instance-based methods fail to encode cluster-level semantic structures of data\"\nC: Can you provide more ablation results to demonstrate the individual impact of the 3 filtering principles?\nD: Could you please elaborate why \"uniform\" part in objective function is important?", "paper_topic_and_main_contributions": "The paper proposes a new self-supervised learning method called Denoised Neighborhood Aggregation (DNA) for fine-grained category discovery from coarsely labeled data. The main contributions are:\n•\tModeling semantic structures of data instead of instance-level learning to learn compact cluster representations for fine-grained category discovery.\n•\tIntroducing 3 principles (label, reciprocal, rank constraints) to filter false neighbors for a more accurate representation for the  query.\n•\tProviding a theoretical analysis to show the learning objective mimics the clustering loss to form compact clusters.\n•\tDemonstrating significant performance boost on 3 benchmark datasets, with improved neighbor accuracy over state of the art methods.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Scheirer et. al. (2016). Statistical methods for open set recognition\n\nBadirli et. al. (2023) Classifying the unknown: Insect identification with deep hierarchical Bayesian learning", "Typos_Grammar_Style_and_Presentation_Improvements": "The paper is well written and easy to follow."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper delves into the problem of fine-grained category discovery and introduces denoised neighborhood aggregation, a self-supervised framework aimed at encoding semantic structures into the embedding space. The paper provides both theoretical analysis and empirical experiments to demonstrate the effectiveness of the proposed model.

The consensus among all reviewers is that this paper exhibits strong soundness and excitement. However, there are several concerns that need further attention from the authors, including:

1. Definition Clarification: Some term definitions are found to be confusing and would benefit from clearer explanations.
2. Boundary Settings: For example, how to solve the situation without any neighbors. The authors can give more explanations.
3. Ablation Study: A broader range of ablation studies on multiple datasets should be conducted to make the experiments more convincing.
4. Hyper-parameter Settings: The paper lacks sufficient details regarding the choice of hyper-parameter settings, which should be provided for transparency.

In conclusion, this paper offers strong quality and novelty, with the potential to contribute significantly to the research community. To further improve its quality, I recommend that the authors diligently address the concerns raised by the reviewers during their final revision.
Decision: Accept-Main