Improved Training of Deep Text Clustering
Authors: Zonghao Yang
Wenpeng Hu
Yushan Tan
Zhunchen Luo
Venue: EMNLP 2023 Findings (2023)
Keywords: Text Clustering
Deep Clustering

Abstract: The classical deep clustering optimization methods basically leverage information such as clustering centers, mutual information, and distance metrics to construct implicit generalized labels to establish information feedback (weak supervision) and thus optimize the deep model. However, the resulting generalized labels have different degrees of errors in the whole clustering process due to the limitation of clustering accuracy, which greatly interferes with the clustering process. To this end, this paper proposes a general deep clustering optimization method from the perspective of empirical risk minimization, using the correlation relationship between the samples. Experiments on two classical deep clustering methods demonstrate the necessity and effectiveness of the method. Code is available at https://github.com/yangzonghao1024/DCGLU.

Improved Training of Deep Text Clustering
Zonghao Yang
Wenpeng Hu Yushan Tan
Zhunchen Luo
Information Research Center of Military Science, PLA Academy of Military Science
(yangzonghao1 1024, tanys@166)@163.. com
wenpeng. hu@pku. edue. cn, zhunchenluo@gmail.
Abstract
specific supervision information. We call gen- eralized supervision in clustering which imple- mented through generalized labels in most cases. The generalized labels usually have different gener- ation method according to different deep clustering al algorithm. We will elaborate them in Section 2. Here, we argue that the generalized labels are full of noise due to the performance limitation of clus- tering model and estimation methods especially the beginning of clustering progress, which will disturb the recognition of clustering group and in- fluence the clustering result. To this end, inspired by Positive and unlabeled Learning (PU learning), we propose a novel method to reduce the impact of noise in generalized labels, namely Deep Clus- tering optimization from Generalized Labeled and Unlabeled data (DCGLU for short).
The classical deep clustering optimization methods basically leverage information such as clustering centers, mutual information, and distance metrics construct implicit general- ized labels to establish information feedback (weak supervision) and thus optimize the deep model. However, the resulting generalized labels have different degrees of errors in the whole clustering process due to the limitation of clustering accuracy, which greatly interferes with the clustering process. To this end, this paper proposes general deep clustering optimization method from the perspective empirical risk minimization, using the correlation relationship between the samples. Experiments on two classical deep clustering methods demonstrate necessity effec- tiveness of the method. Code is available at http::/ggihub..cmyyaaggnnggggghi 1024/DCGL.U
The proposed DCGLU divides the clustering samples into high confidence samples and low con- fidence samples according to the confidence level of the samples been correctly clustered. Obviously. the generalized labels of low confidence samples tend to have more errors (as the decision confidence usually related to accuracy (Hu et al., 2020)). In this case, we propose to regard the low confidence samples unlabeled samples and the high confi- dence samples labeled samples, then learning the model in the same fashion with PU learning to reduce the errors in generalized supervision and thus improve the clustering performance.
1 Introduction
Cluster analysis plays role in machine learning and data mining as can classify data into differ- ent groups in an unsupervised way. Over the past decades, a large number of clustering methods with shallow models have been proposed (Ren et al., 2019; Comaniciu and Meer, 2002: Ren et al., 2018; Bishop and Nasrabadi, 2006; Ren al., 2017; Cai al., 2013; Huang al., 2021). The performance of them on the complex data limited due to the poor power feature learning.
Overall, our contributions can be summarized as follows:(1) proposes a new problem that exists in most deep clustering approaches and formulate it as problem of generalized supervision with general- ized label, which will facilitate the research of deep clustering; (2) proposes general deep clustering optimization method, which be leveraged reduce the impact of noise in clustering; (3) experi- ments show the proposed DCGLU can improve the clustering features and clustering results strong baselines.
Recently, deep learning based clustering ap- proach (referred to deep clustering) aims at effec- features from data and performing clustering with learned features simultaneously (Chang al., 2017; Albert et al., 2022; Ronen et al., 2022).
tively extracting more
After in-depth research, we find that most deep clustering models are usually optimized through
Corresponding authors.

---

DOGLU Medule
rithm. Hence, the generated target distribution P can be considered as the distribution of the gen- eralized labels Y" of the samples. The process of fine linettunig the deep clustering model is the process of generalized label exploitation, which can be de- scribed formally by the experience risk Rs(f):
7,7 mea. CD Gn
Pre Pre-triined Mocel
Firedlans
it tooe
Labeled Comple
Unlabedd Gardies
Festure Vector
Bady Empprtton
Clustor Mocule
Gel Comfidenc level
Selt Aesggmem
Corldereen
Ra(f) 1 Σ (f
(I)
Figure 1: Architecture overview of DCGLU.
where is the distance metric function of fxx and the generalized label y', (24, 1/() ~ P.
2 Problem Formulation
Given the unlabeled text dataset D - {z;, N.. where represents the tth text in the cor- pus, our goal is to cluster N samples into & differ- ent classes by unsupervised method. Each sam- ple in D is represented by feature vector. We use the pre-trained language model Bert (Devlin et al., 2018) to obtain the representation & = BERT (2:) of each text.
3 Method
The noises in generalized label Y' result in error propagation problem. A common used method for the problem is instance weighting (Lison and Bibauw, 2017): R₂(f)
1 - W;- L (ʃ (x:) + v/). wi is the importance estimation weight of each sample. Clearly, there are two main challenges for instance weighting, (1) it reduces the impact of noise while igmoring a large amount of information in data with small wy; (2) the es- timation of w, is very tricky due to the accuracy limitation of unsupervised clustering.
n
As discussed in Section 1, the deep model in deep clustering methods is usually optimized by specific supervised information. We refer the specific supervision to generalized supervision, which is basically realized by generalized labels. The generalized label generation method varies among different clustering methods !: 1) The clus- ter center used in K-means-based methods (Yang et al., 2017: Xie et al., 2016) can be considered as the generalized label; 2) Spectral clustering correla- tion methods (Chang et al., 2017; Yang et al., 2019) use the distance similarity matrix between sam- ples to minimize the similarity distance between samples, the similarity between samples regarded as generalized labels; 3) Gaussian Mix- ture Model methods (Ronen et al., 2022) assume that the samples in each class obey an indepen- dent Gaussian distribution. The center of the Gaus- sian distribution can be regard as the generalized label; 4) (Caron et al., 2018) use the clustering assignment results as pscudo-label information to optimize feature generation. Thus clustering as- signment results are the generalized labels.
For challenge (2), we propose to use the con- fidence of samples been correctly clustered (easy estimation, see appendix A.1) to measure the qual- ity of generalized labels. Note that the confidence cannot replace noise estimation as the generalized labels tend to have little errors with high confidence but the contrary is not necessarily true in most cases. To solve the above problem and challenge (1), we regard the samples with low confidence as the unla- beled data as we cannot make sure the correctness of their generalized labels. 2 Therefore, the samples are divided into high-confidence labeled samples and unlabeled (low confidence) samples according to their confidence levels by hyper-paramete, t. Then we adapt PU learning (Liu et al., 2002) to our case for model optimization. Here we directly provide the final adaptation formula, for detailed
be
Formally, deep clustering algorithm will gen- erate target distribution P, which is used to in- dicate the probability distribution of samples of the same category, and optimize the clustering model through stochastic gradient descent algo-
*According maximum entropy principle, optimal treatment for samples with unknown distribution to make no assumptions, in which case the probability distribution is the most uniform and the risk prediction is minimal.
please note that the generalized label is not necessarily a discrete classification label, but also a continuous measure- ment, such as distance
*We set f to 0.8 in the experiments which performs well in different data sets and for different models.

---

Dataset
SNIPS
DBPedia
Method
NMI
ARI
ACC
NMI
ARI
ACC
KM
71.42
67.62
84.36
67.26
49.93
61.00
AC
71.03
58.52
75.54
65.63
43.92
56.07
SAE-KM KM
78.24
74.66
87.88
59.70
31.72
50.29
DEC
84.62
82.32
91.59
53.36
29.43
39.60
DCN
58.64
42.81
57.45
54,54
32.31
47.48
BERT-KM
52.11
43.73
70.29
60.87
26.6
36.14
DAC
75.51
4.93
67.13
7.10
77.17 6.61
72.73 + 3.57
53.01 + 5.04
63.27 + 5.74
DAC*
79.92 + 4.40
70.67 + 6.89
79.64 + 6.22
74.25 : 2.57
56.93 +: 4.36
66.90 +: 4.96
DeepDPM
83.69
2.62
79.48 5.80
88.35 + 6.10
69.17 + 4.33
18.02 + 6.68
58.30 + 6.52
Dee
85.15 0.10
82.93 0.12
92.07 0.05
72.14 + 2.11
$2.84 1 3.21
62.36 + 4.54
Table 1: The clustering results on two datasets, where with denotes the result after introducing DCGLU. SNIPS and DBPedia baseline results are obtained from unsupervised section (Lin al., 2020). All reported results are in percentages. DAC well DeepDPM results averaged from the source code third-party quality code under 10 runs dev.).
derivation process, please refer Appendix A.2:
of DCGLU. We keep our primary experimental setup consistent with what DeepDPM' and DAC reported their original paper. For more details of datasets, baselines and implement settings please refer to Appendix B.
Rf)(f) =p Rf) max -
R/() Σ (x5)
"pp
--1
(2)
R5( 1 Σ (19)
4.2
Compared with strong baselines
Table shows the clustering results of strong base- lines and the effectiveness of DCGLU, We can draw the following observations.
RC)) E (xF()))
where yPP the multi-class generalized label corre- sponding to the high confidence sample; in general, y can be expressed as y/d - yL: Zp is the prior probability of positive samples.
Firstly, the proposed DCGLU has significant im- provement two different kinds of deep cluster- ing methods DAC and DeepDPM (with p-value 0.01 on paired t-test). The consistency improve- ment of multiple algorithms, evaluation metrics and datasets indicates that DCGLU has better ef- fectiveness and universality.
Overall, give the clustering optimization algo- rithm in the Algorithm in appendix A.2, and give the architecture of DCGLU in Figure Clearly, DCGLU realizes decoupling from the clustering model, therefore it can be applied to most cluster- ing algorithms.
Secondly, we can see that 1) on the SNPIS dataset, with the help of DCGLU, DeepDPM suc- ceeds in significantly outperforming the strongest baseline system (DEC) participating in the compar- ison (with p-value 0.01 on paired t-test); 2) on the DBPedia dataset, DCGLU makes DAC as well as DcepDPM, the top two powerful systems, get better results. This indicates that the error propaga- tion problem in generalized supervision described in this paper has large impact on the performance of deep models and is prevalent (even in very strong clustering systems), while DCGLU can deal with the problem and get better results.
4 Experiments
4.1 Experiment Setup
We apply the DCGLU method on two strong deep clustering method: DAC (Chang et al., 2017) and DeepDPM (Ronen et al., 2022), and then evalu- ate them on two publicly available text datasets: SNIPS (Coucke et al., 2018 8) classes) and DBPe- dia (Lin et al., 2020) (14 classes). We further take K-Mcans(KM) (MacQucen, 1967) and agglomer- ative clustering (Gowda and Krishna, 1978), SAE-KM, BERT-KM. Deep embedding cluster- ing(DEC) (Xie al., 2016), Deep clustering net- work(DCN) (Yang et al., 2017), etc as the base- lines into comparison to show the competitiveness
In Section 3, we discussed that the instance weighting method can mitigate the noise problem in generalized labels, unfortunately is difficult to

---

estimate the noise of generalized labels accurately in the clustering algorithm. This paper uses con- fidence to select high confidence labeled samples and unlabeled samples. In this case, can we use confidence as weight to mitigate the noise prob- lem? The answer is yes but we cannot get better clustering results in experiments. Taking DAC al- gorithm as example, compared with the orig- inal DAC algorithm, instance weighting (regard the confidence level the instance weight) de- creases 3.31,3. 3.31,3.79,2.63 percentage points for NMI, ARI, and ACC metrics on SNIPS dataset, and 7.19,6.59,6.16 percentage points for the three met- rics on DBPedia dataset, respectively. This is be- cause the generalized labels with low confidence level are not necessarily noise, so the instance weighting using confidence level will ignore lot of relevant information, which makes the clustering effect appear a certain degree of degradation.
tends be poor when the extracted sample features are poor. In recent years, rescarchers (Xic et al., 2016; Yang et al., 2017) have combined deep neural networks to focus their work in deep clustering by feature clustering jointly. (Chang et al., 2017) pro- posed Deep Adaptive Cluster(DAC), which trans- forms the clustering problem into a binary pairwise- classification framework to determine whether sam- ples belong the same class. (Hadifar et al., 2019) proposed learn discriminative features from both an autoencoder and a sentence embedding, and then update the weights of the encoder network using assignments of clustering algorithm as su- pervision. (Zhang et al.. 2021) proposed Support- ing Clustering with Contrastive Learning (SCCL) to leverage contrastive learning to promote better separation. (Ronen et al., 2022) proposed Deep Dirichlet Process Mixpurr(DeepDPM) to adapt changes in the k-values of clustering categories by dynamically performing split-and-merg oper- ations. In this paper, we focus on the influence of noise in clustering which is not explicitly investi- gated before.
4.3 More evaluations
We evaluate the representation ability of DCGLU as learning better clustering features is very im- portant for deep clustering algorithm (Caron et al., 2018). Good representation of clustering indicate the clustering results can be further improved by fine-tuning or distance-based clustering methods. etc. In the experiments, we leverage K-Means and DEC as the post clustering method based on the features learned by DCGLU, and get consistency and significant improvements on the two datasets (with p-value < 0.01 on paired (-test), which indi- cates DCGLU can learn better representations. For more details please see Appendix C.I.
PU learning, Positive-unlabeled (PU) learning has been studied for a long time (De Comité et al., 1999; Du Plessis et al., 2014, 2015; Kiryo et al., 2017). For PU learning (Liu et al., 2002) is de- fined follows: given set of positive samples P and a set of unlabeled samples U, which contain hidden positive and negative samples, construct a binary classifier to classify the samples. PU learn- ing is used in many natural language processing applications, (Xia et al., 2013) combined instance selection and instance weighting to apply PU learn- ing to cross-domain sentiment classification; (Peng et al., 2019) explored the way to perform named entity recognition (NER) using only unlabeled data and named entity dictionaries; (He al., 2020) presented method to improve the performance of distant supervision relation extraction with PU Learning.
The Tp in Equation is the key in DCGLU, we conducted a detailed analysis to guide its setting in specific applications. Fortu- nately, we found that using a constant Tp (Tp = 0.5) can already achieve good results, which greatly reduces the conditions of DCGLU applications. For more details please see Appendix C.2.
5 Related Work
In this paper, inspired by PU learning, we pro- pose a novel method (DCGLU) to improve the performance of text clustering, which 1) transform the multi-claasificatioon problem in clustering into binary classification; 2) establish a binary classifi- cation selector; 3) from the perspective of empirical risk minimization, dig the correlation between gen- eralized labeled and unlabeled samples. Clearly, there are some major differences between PU learn- ing and DCGLU.
Deep clustering. Researchers have proposed many clustering methods, including center-of- mass-based clustering (MacQueen, 1967), density- based clustering (Ester et al., 1996; Comani- ciu and Meer, 2002), agglomerative clustering (AG) (Gowda and Krishna, 1978) and so on, but these traditional clustering methods fix sample fea- tures, and the clustering performance of the model

---

6 Conclusion
Dorin Comaniciu and Peter Meer. 2002. Mean shift: A robust approach toward feature space analysis. IEEE Transactions pattern analysis and machine intelli- gence, 24(5):603-619.
In this paper, we propose the concept of general- ized supervision and generalized labels in cluster- ing which can help to study the impact of noise in clustering and thus improve the performance of clustering. Based the generalized supervi- sion and labels, we propose deep clustering op- timization method, namely Deep Clustering opti- mization from Generalized Labeled and Unlabeled data (DCGLU) which can be leveraged to most deep clustering methods of text. As we discussed in related works, DCGLU different from existing clustering and clustering optimization approaches. The experimental results demonstrate the necessity and effectiveness of the method.
Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Calta- girone, Thibaut Lavril, al. 2018. Snips voice plat- form: an embedded spoken language understanding system for private-by-desige voice interfaces. arXiv preprint arXiv: 1805. 10790.
Francesco De Comité, François Denis, Rémi Gilleron, and Fabien Letouzey. 1999. Positive and unlabeled examples help learning. In International confer- ence on algorithmic learning theory, pages 219-230. Springer.
Arthur P Dempster, Nan M. Laird, and Donald B Rubin. 1977. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1):1-22
Acknowledgement
We would like to express gratitude to the anony- mous reviewers for their kind comments. This work was supported by National Natural Science Foun- dation of China (N0.62206308, No.61976221).
Jacob Devlin, Ming-Was Beeg-Weii Chang. Kenton Lee, and
Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint 1xX.v:880.008800
Marthinus Du Plessis, Gang Niu, and Masashi Sugiyama. 2015. Convex formulation for learning from positive and unlabeled data. In International conference machine learning, pages 1386-1394. PMLR.
Limitations
Due to the limitations imposed by the data repre- sentation methods, in future work, we will attempt apply our method to image data.
Marthinus C. Du Plessis, Gang Niu, and Masashi Sugiyama. 2014. Analysis of learning from positive and unlabeled data. Advances in neural information processing systems, 27.
References
Paul Albert, Eric Arazo, Noel E O'Connor, and Kevin McGuinness. 2022. Embedding contrastive unsuper- vised features to cluster in-and out-of-distribution noise in corrupted image datasets. In European Conference on Computer Vision, pages 402 419. Springer.
Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, al. 1996. A density-based algorithm for dis- covering clusters in large spatial databases with noise. In kdd, volume 96, pages 226-231.
K Chidananda Gowda and G Krishna. 1978. Agglomer- ative clustering using the concept of mutual nearest neighbourhood. Pattern recognition, 10(2):055-122.
Christopher M Bishop and Nasser M Nasrabadi. 2006. Pallern recognition and machine learning, volume 4.
Springer.
Amir Hadifar, Lucas Sterckx, Thomas Demeester, and Chris Develder. 2019. self-training approach for short text clustering. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepLANLP-20) 22099), pages 199.
Xiao Cai, Feiping Nie, and Heng Huang. 2013. Multi- view k-means clustering on big data. In Twenty-Thin Third International Joint conference on artificial intelli- gence.
Zhengqiu He, Wenliang Chen, Yuyi Wang, Wei Zhang, Guanchun Wang, and Min Zhang. 2020. Improving neural relation extraction with positive and unlabeled learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages 7934-7934.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. 2018. Deep clustering for unsuper- vised learning of visual features. In Proceedings of the European conference on computer vision (ECCV). pages 132-149.
Wenpeng Hu, Ran Lc, Bing Liu, Jinwen Ma, Dongyan Zhao, and Rui Yan. 2020. Translation vs. dialogue: A mod-
Jianlong Chang. Lingfeng Wang, Gaofeng Meng, Shim- ing Xiang, and Chunhong Pan. 2017. Deep adaptive image clustering. In Proceedings of the IEEE in- ternational conference on computer vision, pages 5879-5887.
comparative analysis of sequence-to-sequeme eling. In Proceedings of the 28th International Con- ference on Computational Linguisties, pages 4111- 4122.

---

Zongmo Huang, Yazhou Ren. Xiaorong Pu, and Lifang He. 2021. Non-linear fusion for self-paced multi- view clustering. In Proceedings of the 29th ACM International Conference on Multimedia, pages 3211- 1- 3219.
Meitar Ronen. Shahaf E Finder, and Oren Freifeld. 2022. Deepdpm: Deep clustering with an unknown number of clusters. In Proceedings of the IEFF/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 9861-9870.
Ryuichi Kiryo, Gang Niu, Marthinus C Du Plessis, and Masashi Sugiyama. 2017. uositive-unlabeled learn- ing with non- non-negative risk estimator. Advances in neural information processing systems, 30.
Jianlin Su, Jiarun Cao, Weijie Liu, and Yangyiwen Ou, 1021. Whitening sentence representations for bet- ter semantics and faster retrieval. arXiv preprint arXiv:2103.15316
Harold W Kuhn. 1955. The hungarian method for the assignment problem. Naval research logistics quar- terly, 2(1-2):83-97.
Rui Xia, Xuclei Hu, Jianfeng Lu, Jian Yang, and Chengqing Zong. 2013. Instance selection and in- stance weighting for cross-domain sentiment classifi- cation via pu learning. In Twenty-Thiie International Joint Conference Artificial Intelligence.
Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef, Sören Auer, et al. 2015. Dbpedia-a large-scale, mul- tilingual knowledge base extracted from wikipedia. Semantic web, 1725
Junyuan Xie, Ross Girshick, and Ali Farhadi. 2016. Unsupervised deep embedding for clustering analy- sis. In International conference on machine learning, pages 478 487. PMLR.
Ting-Fan -nn Lin, Hua Xu, and Hanlei Zhang. 2020. Dis- covering new intents via constrained deep adaptive clustering with cluster refinement. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages 1660-837.
Bo Yang. Xiao Fu, Nicholas D Sidiropoulos, and Mingyi Hong. 2017. Towards t-means--rientee spaces: Si- multaneous deep learning and clustering. In interma- tional conference on machine learning, pages 3861- 3870. PMLR.
Pierre Lison and Serge Bibauw. 2017. Not all dialogues are created equal: Instance weighting for neural conversational models. arXiv preprint
Yang, Cheng Deng, Feng Zheng, Junchi Yan, and Wei Liu. 2019. Deep spectral clustering using dual autoencoder network. In Proceedings of the IEEE/CVF conference on computer vision and pat- tern recognition, pages 4066-4075.
arXiv:1704.08966.
Bing Liu, Wee Sun Lee, Philip S Yu, and Xiaoli Li. 2002. Partially supervised classification of text documents. In ICML, volume 2, pages 387-394. Sydney, NSW.
Dejiao Zhang. Feng Nan, Xiaokai Wei, Shangwen Li, Henghui Zhu, Kathleen McKeown, Ramesh Nalla- pati, Andrew Arnold, and Bing Xiang. 2021. Sup- porting clustering with contrastive learning, arXiv preprint arXiv:2 100..3553..
MacQueen. 1967. Classification and analysis of mul- tivariate observations. In 5th Berkeley Symp. Math. Statist. Probability, pages 281 197-
Minlong Peng, Xiaoyu Xing. Qi Zhang, Jinlan Fu, and Xuanjing Huang. 2019. Distantly supervised named entity recognition using positive-unlabeled learning. arXiv preprint 1906v 013788.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word rep- resentation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Yazhou Ren, Carlotta Domeniconi, Guoji Zhang, and Guoxian Yu. 2017. Weighted-ohject ensemble clus- tering: methods and analysis. Knowledge and Infor- mation Systems, 51()):661-68)
Yazhou Ren, Xiaohui Hu, Ke Shi, Guoxian Yu, Dezhong Yao, and Zenglin Xu. 2018. Semi-supervised den- peak clustering with pairwise constraints. In Pacific Rim International Conference on Artificial Intelli- gence, pages 837-850. Springer.
Yazhou Ren, Uday Kamath, Carlotta Domeniconi, and Zenglin Xu. 2019. Parallel boosted clustering. Neu- mccmmuting, 351:87-100.

---

A Detailed Explanations
and unlabeled (low confidence) samples are con- sidered as unlabeled samples, thus:
A.1 Method of Confidence Estimation
Compared with the estimation of generalized la- bels, the method of obtaining confidence is rela- tively simple. For examples, in the method based the cluster center, the closer the sample is to the cluster center, the higher the possibility of belong- ing to the center, so the confidence level can be acquired by the distance; in the method based on the similarity between samples, the higher the sim- ilarity between samples, the higher the possibility of the sample pair belonging to the same category, thus the confidence level can be acquired by the similarity between samples.
R5( Σ (((
(7)
Rpp
R5 (f) 1 "y R2
M (ff
(8)
where.y is the multi-class generalized label cor- responding to the high confidence sample; Eq. indicates that the high confidence sample does not belong to the loss of the currently assigned gen- eralized label, in general, v/" can be expressed y vff. Based on the above development method, Ry (J) can be extended to:
A.2 Adapting PU Learning to Deep Clustring
The
According the problem of learning from unla- beled data discussed Section 3, we find that PU learning has more in-depth research, which can pro- mote the identification of samples with similar char- acteristics of positive data among unlabeled sam- plcs. In the scenario of clustering, high-confidence labeled samples and unlabeled (low confidence) samples can be regard as the positive samples and unlabeled samples in PU learning respectively. The classical PU learning (Du Plessis al., 2014) relies on the binary unbiased risk estimator to mine the correlation between unlabeled samples and positive samples:
(T)
Tha E (x)))
(9)
However, there is no reliable multi-class gen- cralized label for low confidence samples. Simi- larly, based on the maximum entropy principle, we assume that the generalized label y/P of each un- labeled (low confidence) sample obeys uniform distribution, i.e., the unlabeled sample expects the minimum under all category labels, so for the unla- beled sample, we transform (q. into:
1
R (f)
Tu Σ (f(()))) (10)
Rpm k5 x,) (T)
where f (xx") is the mean of the predicted probabil- ities of unlabeled (low confidence) samples over all categories. The extension method, including Eq. (7), (8), and Eq.(10), can be approximated as way to complete the learning of high confidence (4) categorized samples and uncategorized (low confi- dence) samples by transforming the multi-category task split into multiple binary categories. Further, (5) to prevent overfitting we use the non-negative risk estimator (Kiryo et al., 2017), which transforms (3) into:
(3)
where Tp is the prior probability of positive sam- ples, Among Eq. (3),
1
(f)
cf (zf))++)
1 Tpp
RT( = 7p 2=1
Σ
(x)))
1 Σ (((())))))
(6)
nu
R n. (F) max (0. (,, Rn)
(11)
where {+1, }} corresponds to the labels of posi- tive and unlabeled samples, respectively.
By substituting (7, (q. and (q. into Eq.(1 the loss function of the optimized deep clustering model proposed in this paper can be obtained, which is called Deep Clustering opti- mization from Generalized Labled and Unlabeled learning algorithm(DCGL) It can be added as a regularization term into the currently commonly
Here we introduce PU learning deep cluster- ing help use the generalized labels and extend it to multiple classes to adapt for the application scenario of the clustering, in which high confidence labeled samples are considered as positive samples

---

used deep clustering algorithm, with good general- ization ability:
DBpedia contains 14 non-overlapping
classes of ontology selected from DBPedia 2015 (Lehmann et al., 2015). We follow (Lin et al., 2020), which contained 1,000 samples each classes
Lotal = Lon + A (f)
(12)
Where Can is the original loss of deep clustering algorithm, À is the blend factor, and (f) is the loss function of deep clustering optimization model proposed in this paper.
B.2 Baseline and Evaluation Metrics
We follow (Lin et al., 2020), and chose the baseline of the unsupervised part for comparison:
Algorithm DCGLU
KM and AG. K-means(KM) (MacQueen, 1967) and Agglomerative Clustering(AC) (Gowda and Krishna, 1978) are classical clustering algorithms. and here we represent the sentences with the av- craged pre- pre-trained 300-dimensional word embed- dings from GloVe (Pennington et al., 2014).
Input: Dataset X' {a:) xy is the 4th text in X.
Parameter: Clustering model parameter 0, hyppe-parametent confidence threshold (0 S and positive class-prior probability
Output: Cluster label e, for each x;
1: Initialization
the cluster network parameters; number of training steps
3: Sample butch Nr from X, k indicates the batch id;
We encode the sentences with the stacked autoencoder (SAE), and then execute k- means.
SAE-KM.
4: Input to clustering model f. Get probability of sample pairs samples p:
5: if & then
6: Get high confidence samples X& from 2t as the positive samples A?": Calculate the empirical risk of the positive samples
BERT-KM. We encode the sentences by BERT (Devlin et al., 2018), and then execute k-means.
7:
RØ (J') and RG (7) and (8)
8:
else
DEC. Deep embedding clustering(DECC) (Xie et al., 2016) learns the mapping from data space to low-dimensional feature space, and uses distribution iteration to fine-tune clustering model.
9:
Get low confidence samples x from N as the unlaheled samples X2;; Calculate the empirical risk of the unlabeled sam-
10:
ples R. (f)/Eq,((1)
11:
end if
12:
if R. (f) - -T RT (f) 2 0 then
DCN. Deep clustering network(DCN) (Yang et al., 2017) follows the idea of DEC and adds regularization term in the optimization process. As optimization strategy, in order show the performance of our algorithm, we select two advanced deep clustering algorithms of differ- ent kinds as the main methods to test DCGLU: Deep Dirichlet Process Mixture (DeepDPM) (Ro- nen et al., 2022) and Deep Adaptive Clustering (DAC) (Chang et al., 2017). DeepDPM is a deep nonparametric clustering method that can adapt to k-value changes by dynamically adjusting the split and merge operations. DAC transforms the clus- tering problem into binary classiisiilssiiiaatiionn framework to judge whether the samples belong to the same category.
13:
gradient VoLwi + V (ff
14:
else
15:
Set gradient VoLai + Vo(Tp Rp (f) (f))
16:
end if
17:
Update by external SGD-like stochastic optimiza- tion algorithm:
18: end for
19: for all x; E X do
20: 1, = (x,);
21: arg maxa (l..):
22: end for
B Experiment Setup
B.1 Datasets
We conduct experiments on two publicly available text datasets. The detailed statistics are shown in Table 2.
To evaluate the experimental results, we choose three common clustering measures: Normalized Mutual Information (NMI), Adjusted Rand Index (ARI), and clustering accuracy (ACC). To calcu- late clustering accuracy, we use the Hungarian al- gorithm (Kuhn, 1955) to find the best alignment between the predicted cluster label and the ground-
SNIPS It derives from (Coucke et al., 2018) and is personal voice assistant dataset containing 14,484 voices, divided into seven categories alto- gether.

---

Dataset
Classes
#Training
#Validation
#Test
Vocabulary Size
Length(Avg)
SNIPS
7
13084
700
700
11971
9.03
DBPedia
14
12600
700
700
45077
29.97
Table 2: Statistics of SNIPS and DBPedia datasets. # indicates the total number of sentences.
truth label. The higher the score of all metries, the better the clustering performance.
distance-based clustering methods, etc. To explore more deeply the effect of DCGLU on clustering
features, we explored the experimental results different variants of the DAC algorithm, which DAC-KM clusters the embedded features Iearned by DAC using the K-Means model, and DAC-DEC combines the idea of DEC (Xie al., 2016) model to fine-tune and improve the clustering assignment by expectation maximization iterations.
B.3 Implement Details
For both DeepDPM and DAC, we use the same BERT model to get feature vectors text data, but as the adaptation of DeepDPM is limited for high- dimensional features, in accordance with their sug- gestions, we use Bert- Sort-Whiteen Whitening (Su et al., 2021) to reduce the dimension of text features. Text features H is reduced from 768 to 64 dimensions. For fairer comparison, we maintain the original dataset set- tings of (Ronen al., 2022) and (Lin al., 2020). Finally, we report the average results of each algo- rithm over ten runs.
As can be seen from Table 3 of the experimental results, each method has certain improvement on both datasets after combining the DCGLU op- timization method, which is more obvious the DBPedia dataset, which indicates that the DCGLU method able to optimize the clustering effect of the DAC algorithm itself, in addition to obtaining better clustering features. Note that DCGLU has limited improvement on the SNPIS dataset, and we believe this is because the performance improve- ment on the clustering representation cannot be efficiently transferred to the final clustering perfor- mance after post-processing, therefore the limited improvement can already indicate that DCGLU has some improvement on the clustering features.
DeepDPM is divided into clustering module and subclustering module. The model is optimized by introducing new loss caused by Expectation- Maximization (EM) (Dempster et al., 1977) in Bayesian GMM, where every E-step is followed by standard M-step. However, we believe that the probability of E-step generation is too incredible, and there are many misdivided samples. Therefore, we sharpen the probability of E-step generation, se- lect high confidence labeled samples and other sam- ples as unlabeled samples to optimize the clustering effect. DAC transforms the clustering problem into framework to deter- mine whether the samples belong the same class, we optimize the clustering effect by treating two samples with high similarity as high-confidence la- beled sample pairs and those with low similarity as unlabeled sample pairs. Our experiments conduct on pytorch version 1.1 and 1.0.1 respectively. All experiments were performed on the NVIDIA Ge-Foree RTX-2080Ti Graphical Card with 10G graphical memory.
C.2 Effect of Tp
binary
In the PU learning domain, Tp denotes that the prior of positive samples be estimated by the ratio of positive samples the data, however, in DCGLU, the accuracy and recall of high confidence labeled sample estimates in the generalized labels of most deep clustering algorithms keep changing as the clustering algorithm learns(generally. it will be pro- moted first and then remain stable). This would result in Tp changing dynamically throughout the training process and thus difficult to estimate. For- tunately, we found that using a constant тp can al- ready achieve good results, which greatly reduces the conditions of DCGLU applications.
C Experimental Results
C.1 Effect on clustering features
The effects of different Tp the experimen- tal results are shown in Figure 2, where Vp {0.1, 0.3, 0.5, 0.7, 0.9}. It can be seen that for dif- ferent Tp. the three evaluation metrics of the two data sets show the same trend of change, and the optimal effect is reached at Tp = 0.5. When Tp
Learning better clustering features is one of the important directions of deep clustering algorithm research (Caron et al., 2018), on which the cluster- ing effect can be further improved by fine-tuning "htts://pytorch.org

---

Dataset
SNIPS
DBPedia
Method
NMI
ACC
NMI
ARI
ACC
DAC
75.51 +:
67.13 + 7.10
77.17 1 6.61
72.73 = 3.57
53.01 + 5.04
63.27 : 5.74
DAC*
79.92 4.40
70.67 = 6.89
79.64 6.22
74.25 + 2.57
56,93 + 4.36
66.90 = 4.96
KM
86.65 3.35
83.14
01.49
84.93 2.45
73.74 4.94
80.27 + 5.46
DAC-KM
87.60 1.64
84.70 2.06
92.52 + 1.18
85.26 + 2.53
74.34 5.34
82.37 + 5.95
DAC-DEC
87.17 2.87
83.79 4.46
91.78 + 3.02
86.59 = 2.42
74.74 + 4.68
81.18 + 5.31
DAC-DEC"
87.52 + 1.49
84.59 + 1.78
92.44 + 1.00
86.52 + 2.69
76.69 + 5.23
83.14 + 5.77
Table The clustering results for variants of DAC.
Method-Datasee
DCGLU Time
Base Time
Proportion(DCGLU Time/Base Time)
DAC-Snips
10.74s
829.22s
1.29
DAC-DBPedia
10.30s
1193.92s
0.86
DecpDPM-Snips
104.60s
1832.19s
5.70
DeepDPM-DBPedia
102.29s
2855.89s
3.58
Table Experimental results of efficiency cost DCGLU
DBPedia
We conduct runtime complexity experiment, and Table 4 shows the run time. "Proportion(DCGLUU Time/Base Time)" indicates the proportion of the calculation time of DCGLU in the entire program running time. As can be seen from the table, our method does not bring much additional time over the entire run time. This is because DCGLU has a time complexity of O(N), and at the same time, the additional spatial complexity brought by DCGLU is O(N).
70
Fercentage
60-
NMI
ARI
40
0.3
0.5
SNIPS
80
Porcentage
70
NMI
ARI
ACC
50
0.1
0.3
0.5
0.7
0.9
Figure 2: Experimental results of optimization of DAC algorithm given Tp E {0.1,
0.3, 0.3, 0.5, 0.9).
is overestimated, there is a significant decrease in the experimental results, while the experimental results tend to be stable when Tp is small, so we suggest not to use a larger Tp- This is because in the clustering process, there are still a small number of errors in the high confidence labeled samples. Hence, under the condition of ensuring performance, smaller xp should be used to reduce the probability of introducing errors in DCGLU, so to improve model stability and clustering effect.
C.3
Effect of time efficiency
DCGLU does not increase the complexity of the primary clustering algorithm in terms efficiency.


Reviews:
[{"review_id": "WtLtD36Bem", "reviewer": "Reviewer_xqXP", "summary": "", "questions": "1.\tIn lines 139 to 142, what does the ‘contrary’ mean? Is it for low confidence samples?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for you constructive comments and generous supports! For your questions, we have the following replies:\n\n> Q1. In lines 139 to 142, what does the 'contrary' mean? Is it for low confidence samples?\n> \n\nA1. The word \"contrary\" means \"conversely\" in our paper. In general, we cannot prediction the labels of the samples other than high-confidence samples as the current model is confused about them. But this does not mean that these samples are all negative samples, as there may be positive samples inside (just because the current model does not have the ability to correctly recognize them). So we consider it as an unlabeled sample.\n\n> Q2. The specific problems in the existing methods have not been fully discussed.\n> \n\nA2. Our paper discusses two classical deep clustering algorithms, GMM and spectral clustering. We found that many clustering algorithms contain generalized supervised information, such as clustering center, distance between samples, etc. We believe that the strategy of directly fitting high and low confidence samples in clustering algorithms is not reliable, so we combined PU learning to process low confidence samples as unlabeled samples.\n\n> Q3. The presentation of the article is not very clear.\n> \n\nA3. About unclear representation, we are not sure which part or aspect you are referring to. We will carefully proofread the paper. We guess you don't understand the definition of generalized supervision or generalized label. In fact, in essence, our work defines the way to fit the cluster center in deep clustering as generalized supervision, and the information such as cluster center and sample similarity is defined as generalized label. Then the samples with high confidence are taken as positive samples and the samples with low confidence are taken as unlabeled samples to optimize the clustering performance and text features. That's the core idea of what we do. Finally, Thank you for your kind correction. We will revise the expressions in the paper according to your comments. Thank you very much for your comments.\n\n> Q4. The experiments lack of ablation study.\n> \n\nA4. In the appendix C, we show the influence of the ablation experiment and explore the influence of the setting of $\\pi_p$ on the experiment.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThe proposed method can be added to other methods as a strategy to improve the clustering performance. \n2.\tThe experimental results show the proposed DCGLU can improve the clustering features and clustering results.\n", "reasons_to_reject": "1.\tThe specific problems in the existing methods have not been fully discussed.\n2.\tThe presentation of the article is not very clear.\n3.\tThe experiments lack of ablation study.", "questions_to_authors": "1.\tIn lines 139 to 142, what does the ‘contrary’ mean? Is it for low confidence samples?", "paper_topic_and_main_contributions": "The authors propose the concept of generalized supervision with generalized label, and introduce a general deep clustering optimization method, which can be leveraged to reduce the impact of noise in clustering. The experiments show that the proposed DCGLU can improve the clustering performance over other baselines.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LlSHwajage", "reviewer": "Reviewer_kABq", "summary": "", "questions": "1) Did you evaluate your algorithm in other domains (e.g. computer vision)? If not, why did you decide only working with text data?\n\n2) Did you perform runtime complexity experiments? It would be vital to showcase, that the extension does not suffer from high computational costs.\n\n3) In your method section you mentioned 'instance weighting' (127-129) as a comparable approach, yet you did not evaluate your approach against this technique. Does you method perform better than the aforementioned algorithm?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for you constructive comments and generous supports! For your questions, we have the following replies:\n\n> Q1. Did you evaluate your algorithm in other domains (e.g. computer vision)? If not, why did you decide only working with text data?\n> \n\nA1. We have conducted experiments on image datasets and found that the performance improvement is limited (not significant), but it has shown good results on text datasets. We leave the analysis of the insignificant improvement on images as future work.\n\n> Q2. Did you perform runtime complexity experiments? It would be vital to showcase, that the extension does not suffer from high computational costs.\n> \n\nA2. We did a runtime complexity experiment, and the following table shows the run time. “Proportion(DCGLU Time/Base Time)” indicates the proportion of the calculation time of DCGLU in the entire program running time. As can be seen from the table, our  method does not bring much additional time over the entire run time. This is because DCGLU has a time complexity of O(N), and at the same time, the additional spatial complexity brought by DCGLU is O(N). We will release our code.\n\n| Method-Dataset | DCGLU Time | Base Time | Proportion(DCGLU Time/Base Time) |\n| :---: | :---: | :---: | :---: |\n| DAC-Snips | 10.74s | 829.22s | 1.29% |\n| DAC-DBPedia | 10.30s | 1193.92s | 0.86% |\n| DeepDPM-Snips | 104.60s | 1832.19s | 5.70% |\n| DeepDPM-DBPedia | 102.29s | 2855.89s | 3.58% |\n\n> Q3. In your method section you mentioned 'instance weighting' (127-129) as a comparable approach, yet you did not evaluate your approach against this technique. Does you method perform better than the aforementioned algorithm?\n> \n\nA3. We show the influence of instance weighting on the DAC method in lines 217-229 of this paper. For your convenience, we summarize the results in the table below.\n\n| Dataset |  | SNIPS |  |  | DBPedia |  |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| Method | NMI | ARI | ACC | NMI | ARI | ACC |\n| DAC | 75.51 ± 4.93 | 67.13 ± 7.10 | 77.17 ± 6.61 | 72.73 ± 3.57 | 53.01 ± 5.04 | 63.27 ± 5.74 |\n| DAC + Instance Weighting | 72.2 ± 6.03 | 63.34 ± 9.02 | 74.54 ± 7.94 | 65.54 ± 3.61 | 46.42 ± 4.26 | 57.11 ± 4.53 |\n| DAC + DCGLU | 79.92 ± 4.40 | 70.67 ± 6.89 | 79.64 ± 6.22 | 74.25 ± 2.57 | 56.93 ± 4.36 | 66.90 ± 4.96 |\n\n> Q4. It would be beneficial to make the main paper more self-contained by moving vital parts of the algorithm, such as equation (12), to the main body instead of placing them in the Appendix. This would enhance the overall clarity and accessibility of the paper. There are hardly any references to deep clustering in the text domain.\n> \n\nA4. Thank you for your kind correction. We put formula 12 in the appendix because it is relatively less important. We will revise the expressions and citations in the paper as we will have another page to revise the paper after acceptence . Thank you very much for your comments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The simple extension to deep clustering models has demonstrated superior performance across all evaluated datasets, making it applicable to various scenarios and improving overall results. The paper is well-motivated, written with clarity, and easy to understand.", "reasons_to_reject": "The novelty of the approach is somewhat limited, as it is a combination of two already existing and well-established techniques.\nThe proposed model uses Learning from Positive and Unlabeled Data and evaluates it on a deep clustering framework.\nFurthermore, the labeling of the method as 'Deep Text Clustering' seems unclear, as it is capable of performing well in other domains too. Apart from the BERT model as the backbone, which could be substituted with any pre-trained model (e.g., ResNet), no component is specifically tailored to the NLP domain.", "questions_to_authors": "1) Did you evaluate your algorithm in other domains (e.g. computer vision)? If not, why did you decide only working with text data?\n\n2) Did you perform runtime complexity experiments? It would be vital to showcase, that the extension does not suffer from high computational costs.\n\n3) In your method section you mentioned 'instance weighting' (127-129) as a comparable approach, yet you did not evaluate your approach against this technique. Does you method perform better than the aforementioned algorithm?", "paper_topic_and_main_contributions": "The authors enhance deep clustering algorithms by integrating Positive and Unlabeled Learning. They showcase, that this minor adaption to the loss function improves existing deep clustering models.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "There are hardly any references to deep clustering in the text domain. The following publications would be interesting:\n\n[1] Supporting Clustering with Contrastive Learning (Zhang et.al.)\n[2] A Self-Training Approach for Short Text Clustering (Hadifar et.al)\n\nThe current related work focuses on deep clustering for images, which would be fine if the paper would not be submitted with the title 'deep text clustering'.", "Typos_Grammar_Style_and_Presentation_Improvements": "While the paper is well-written, it would be beneficial to make the main paper more self-contained by moving vital parts of the algorithm, such as equation (12), to the main body instead of placing them in the Appendix. This would enhance the overall clarity and accessibility of the paper."}}, {"review_id": "iCkk0GsOWZ", "reviewer": "Reviewer_ynRi", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for you constructive comments and generous supports! For your questions, we have the following replies:\n\n> Q1. It does not provide a clear definition or formalization of the concept of generalized supervision and generalized labels, which may cause confusion or misunderstanding for the readers.\n> \n\nA1. In essence, our method performs text clustering in unsupervised scenarios, but we find that many clustering methods use high-confidence samples for fitting training in disguised form, and this high-confidence sample can be understood as a label in supervised learning. Therefore, we find that there exists a phenomenon of using generalized labels for generalized supervised learning in clustering algorithms. We will update the statements in the paper and redefine the concepts of generalized supervision and generalized labeling.\n\n> Q2. It does not compare the proposed method with other existing methods that also aim to reduce the noise in clustering, such as instance weighting or noise filtering, which may limit the comprehensiveness and fairness of the evaluation.\n> \n\nA2. We show the influence of instance weighting on the DAC method in lines 217-229 of this paper. For your convenience, we summarize the results in the table below.\n\n| Dataset |  | SNIPS |  |  | DBPedia |  |\n| :---: | :---: | :---: | :---: | :---:| :---: | :---: |\n| Method | NMI | ARI | ACC | NMI | ARI | ACC |\n| DAC | 75.51 ± 4.93 | 67.13 ± 7.10 | 77.17 ± 6.61 | 72.73 ± 3.57 | 53.01 ± 5.04 | 63.27 ± 5.74 |\n| DAC + Instance Weighting | 72.2 ± 6.03 | 63.34 ± 9.02 | 74.54 ± 7.94 | 65.54 ± 3.61 | 46.42 ± 4.26 | 57.11 ± 4.53 |\n| DAC + DCGLU | 79.92 ± 4.40 | 70.67 ± 6.89 | 79.64 ± 6.22 | 74.25 ± 2.57 | 56.93 ± 4.36 | 66.90 ± 4.96 |\n\n> Q3. It does not conduct ablation studies or sensitivity analysis to show the contribution of each component or parameter of the proposed method, which may affect the interpretability and reproducibility of the results.\n> \n\nA3. In the appendix, we show the influence of the ablation experiment and explore the influence of the setting of $\\pi_p$ on the experiment. For the definition parameters of confidence, DAC native method automatically adjusts the definition of high confidence samples, which we directly followed, while DeepDPM method requires us to manually set confidence threshold. In future work, we will explore the induction study of automatic confidence threshold.\n\n> Q4. It does not discuss the limitations or potential drawbacks of the proposed method, such as the scalability or efficiency issues, or the possible failure cases or scenarios.\n> \n\nA4. Thank you for your valuable concern. We would like to answer your question as follows.\n\n- **scalability or efficiency issues:** Our method does not increase the complexity of the primary clustering algorithm in terms of efficiency. We did a runtime complexity experiment, and the following table shows the run time. “Proportion(DCGLU Time/Base Time)” indicates the proportion of the calculation time of DCGLU in the entire program running time. As can be seen from the table, our  method does not bring much additional time over the entire run time. This is because DCGLU has a time complexity of O(N), and at the same time, the additional spatial complexity brought by DCGLU is O(N). We will release our code.\n    \n    \n    | Method-Dataset | DCGLU Time | Base Time | Proportion(DCGLU Time/Base Time) |\n    | :---: | :---: | :---: | :---: |\n    | DAC-Snips | 10.74s | 829.22s | 1.29% |\n    | DAC-DBPedia | 10.30s | 1193.92s | 0.86% |\n    | DeepDPM-Snips | 104.60s | 1832.19s | 5.70% |\n    | DeepDPM-DBPedia | 102.29s | 2855.89s | 3.58% |\n- **possible failure cases or scenarios**: For example, if the high-confidence sample is not trusted, the high-confidence sample can not bring substantial generalized supervised feedback, resulting in the failure of our optimization algorithm, but in fact, this problem exists for all clustering algorithms.\n\n> Q5. It may not provide enough novelty or significance for the NLP community, as deep text clustering is a well-studied problem and the proposed method is based on existing techniques such as PU learning and binary classification.\n> \n\nA5. Although we integrate existing mature techniques, the fact that we have identified a problem prevalent in deep clustering, generalised and abstracted it, and described it formally in a mathematical language is already a clear contribution in itself. The reason for utilising PU learning is because we have found it to be a good solution to the above problem and have experimentally demonstrated the improvement in clustering performance by solving the problem.\n\n> Q6. It may not provide enough clarity or rigor for the NLP community, as some of the concepts and methods are not well-defined or justified, and some of the experiments and analyses are not well-designed or reported.\n> \n\nA6. We will proofread the paper carefully. Redefine and rationalise the concepts and methods we present. And add additional experiments and analyses to the new version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. It tackles an important and challenging problem of deep text clustering, which is a useful technique for many NLP applications such as information retrieval, text summarization, and topic modeling.\n2. It proposes a novel and general method that can be easily integrated with existing deep clustering methods, and does not require any additional parameters or hyperparameters.\n3. It provides a clear and rigorous theoretical analysis of the proposed method, and shows its connection to the empirical risk minimization principle.\n4. It conducts extensive experiments on two text datasets using two strong deep clustering methods, and demonstrates the effectiveness and robustness of the proposed method1.\n5. It compares the proposed method with several baselines and variants, and analyzes the impact of different factors such as confidence level, prior probability, and feature dimensionality.", "reasons_to_reject": "1. It does not provide a clear definition or formalization of the concept of generalized supervision and generalized labels, which may cause confusion or misunderstanding for the readers.\n2. It does not compare the proposed method with other existing methods that also aim to reduce the noise in clustering, such as instance weighting or noise filtering, which may limit the comprehensiveness and fairness of the evaluation.\n3. It does not conduct ablation studies or sensitivity analysis to show the contribution of each component or parameter of the proposed method, which may affect the interpretability and reproducibility of the results.\n4. It does not discuss the limitations or potential drawbacks of the proposed method, such as the scalability or efficiency issues, or the possible failure cases or scenarios.\n5. It may not provide enough novelty or significance for the NLP community, as deep text clustering is a well-studied problem and the proposed method is based on existing techniques such as PU learning and binary classification.\n6. It may not provide enough clarity or rigor for the NLP community, as some of the concepts and methods are not well-defined or justified, and some of the experiments and analyses are not well-designed or reported.\n\nOver all, a finding paper may be suitable for this paper.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper is about deep text clustering, which is the task of grouping text data into different categories without using any labels. The paper addresses the problem of noise in the generalized labels that are used to provide weak supervision for deep clustering models. Generalized labels are derived from different sources such as cluster centers, similarity matrices, or clustering assignments, and they may contain errors due to the limitation of clustering accuracy. The paper proposes a novel method called Deep Clustering optimization from Generalized Labeled and Unlabeled data (DCGLU), which reduces the impact of noise in generalized labels by dividing the samples into high confidence and low confidence ones, and applying a binary classification framework inspired by positive and unlabeled learning1. The paper makes the following contributions:\n1. It introduces the concept of generalized supervision and generalized labels in clustering, which can help to study the impact of noise in clustering and thus improve the performance of deep clustering.\n2. It proposes a general deep clustering optimization method, which can be leveraged to reduce the impact of noise in clustering and can be applied to most deep clustering methods of text.\n3. It conducts experiments on two text datasets using two strong deep clustering methods, and shows that the proposed method can improve the clustering features and clustering results of the baselines.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper delves into the problem of optimizing deep clustering from an empirical risk minimization perspective, with a focus on utilizing the correlation between samples.

The reviewers have expressed varying opinions on this paper. After a thorough examination of the manuscript, I have identified several concerns that need to be addressed. These include limited novelty, poorly presented details (e.g., Deep Text Clustering, generalized supervision and generalized labels), the absence of SOTA baseline comparisons, and a lack of ablation study. Additionally, while the authors acknowledge some prior shortcomings in the Introduction, these issues should be more clearly elucidated and appropriately cited.

To address these concerns and enhance the quality, I recommend that the authors thoroughly revise their work and resubmit it for further evaluation.
Decision: Accept-Findings