SlowGuess/ACL_2020_OCR huggingface dataset:
Ganesh Jawahar, Benoit Sagot, and Djamé Seddah. 2019. What does bert learn about the structure of language? In 57th Annual Meeting of the Association for Computational Linguistics (ACL), Florence, Italy.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. international Conference on Learning Representations.
Diederik P Kingma and Max Welling. 2014. Auto-encoding variational bayes. stat, 1050:1.
Thomas N Kipf and Max Welling. 2016. Variational graph auto-encoders. NIPS Workshop on Bayesian Deep Learning.
Ken Lang. 1995. Newsweeder: Learning to filter netnews. In Proceedings of the Twelfth International Conference on Machine Learning, pages 331-339.
David D Lewis, Yiming Yang, Tony G Rose, and Fan Li. 2004. Rcv1: A new benchmark collection for text categorization research. Journal of machine learning research, 5(Apr):361-397.
Wei Li, Jingjing Xu, Yancheng He, Shengli Yan, Yunfang Wu, and Xu Sun. 2019. Coherent comment generation for chinese articles with a graph-to-sequence model. In ACL.
Chin-Yew Lin and Eduard Hovy. 1997. Identifying topics by position. In Fifth Conference on Applied Natural Language Processing, pages 283-290.
Bang Liu, Di Niu, Haojie Wei, Jinghong Lin, Yancheng He, Kunfeng Lai, and Yu Xu. 2019. Matching article pairs with graphical decomposition and convolutions. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6284-6294.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013a. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111-3119.
Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013b. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 746-751.
George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39-41.
Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. 1999. The pagerank citation ranking: Bringing order to the web. Technical report, Stanford InfoLab.
Christos Papadimitriou and Kenneth Steiglitz. 1982. Combinatorial Optimization: Algorithms and Complexity, volume 32.
Hao Peng, Jianxin Li, Yu He, Yaopeng Liu, Mengjiao Bao, Lihong Wang, Yangqiu Song, and Qiang Yang. 2018. Large-scale hierarchical text classification with recursively regularized deep graph-cnn. In Proceedings of the 2018 World Wide Web Conference, pages 1063-1072. International World Wide Web Conferences Steering Committee.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proc. of NAACL.
Jan Wira Gotama Putra and Takenobu Tokunaga. 2017. Evaluating text coherence based on semantic similarity graph. In Proceedings of TextGraphs-11: the Workshop on Graph-based Methods for Natural Language Processing, pages 76-85.
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3973-3983.
Sunil Kumar Sahu, Fenia Christopoulou, Makoto Miwa, and Sophia Ananiadou. 2019. Inter-sentence relation extraction with document-level graph convolutional neural network. In ACL.
Phi Vu Tran. 2018a. Learning to make predictions on graphs with autoencoders. In 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA), pages 237-245. IEEE.
Phi Vu Tran. 2018b. Multi-task graph autoencoders. NIPS 2018 Workshop on Relational Representation Learning.
Nguyen Xuan Vinh, Julien Epps, and James Bailey. 2010. Information theoretic measures for clusterings comparison: Variants, properties, normalization and correction for chance. Journal of Machine Learning Research, 11(Oct):2837-2854.
Pengtao Xie and Eric P. Xing. 2013. Integrating document clustering and topic modeling. In Proceedings of the Twenty-Ninth Conference on Uncertainty in Artificial Intelligence, UAI'13, pages 694-703, Arlington, Virginia, United States. AUAI Press.
Jiaming Xu, Peng Wang, Guanhua Tian, Bo Xu, Jun Zhao, Fangyuan Wang, and Hongwei Hao. 2015. Short text clustering via convolutional neural networks. *NAACL HLT* 2015, pages 62-69.
Liang Yao, Chengsheng Mao, and Yuan Luo. 2019. Graph convolutional networks for text classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7370-7377.
Jianbo Ye, Yanran Li, Zhaohui Wu, James Z Wang, Wenjie Li, and Jia Li. 2017. Determining gains acquired from word embedding quantitatively using discrete distribution clustering. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1847-1856.
Jie Zhou, Xingyi Cheng, and Jinchao Zhang. 2019. An end-to-end neural network framework for text clustering. arXiv preprint arXiv:1903.09424.
Sentence Position Edge
We connect a keyword pair by considering the average position of their sentence sets in text. Formally, for a keyword node  $a$  with a sentence set of size  $m$ , its position score  $P_{a}$  is computed as:
$$
P _ {a} = \frac {\sum_ {t = 0} ^ {m} \exp (- \lambda * S e n t P o s _ {t})}{m} \tag {1}
$$
SentPos denotes the position of the sentence in which the keyword  $a$  appears (e.g. 0 implies that  $a$  is in the first sentence of the document).  $\lambda$  is a decay parameter pre-defined, we use  $\lambda = 0.2$  throughout the study. The final position weight between a keyword pair  $(P_{ab})$  is calculated by the average of their position scores (i.e.  $P_{ab} = \frac{P_a + P_b}{2}$ ).
Graph Autoencoder (GAE), Multi-Task GAE (MTGAE) and Variational GAE (VGAE)
Given a graph  $G = (A, X)$  where  $A$  is the weighted adjacency matrix, denoting the correlation between the keyword nodes and  $X$  is the node feature matrix, the encoder aims at learning the latent representation  $(Z)$  that can effectively reconstruct  $A$  by the decoder. For a 2-layer encoder, its output is given by  $Z^2 = Z = q(Z|Z, A)$ ,
$$
\begin{array}{l} Z ^ {1} = f _ {r e l u} \left(Z ^ {0}, X A | W ^ {0}\right) \\ Z ^ {2} = f _ {\text {l i n e a r}} \left(Z ^ {1}, X A \mid W ^ {1}\right) \tag {2} \\ \end{array}
$$
where  $f(Z, X A|W)$  is a spectral convolutional operation for feature extraction, and  $Z$  is the set of latent features extracted for the nodes. The decoder reconstructs an adjacency matrix  $\hat{A}$  by computing an inner product between these latent features, its output is given by  $\hat{A} = p(A|Z)$ ,
$$
\hat {A} = \sigma (Z Z ^ {T}) \tag {3}
$$
During training, GAE learns to minimize the reconstruction loss  $(\mathcal{L}_{\mathcal{R}})$ , as measured by the cross entropy between its input  $A$  and its reconstructed  $\hat{A}$ ,
$$
\mathcal {L} _ {\mathcal {R}} = E _ {q (Z | X, A)} \left[ \log p (A | Z) \right] \tag {4}
$$
At inference time, we use the latent representation  $Z$  for document clustering and disregard the reconstructed part  $\hat{A}$ .
To encode more content information from the graph, one can reconstruct both the input adjacency matrix  $(A)$  and feature matrix  $(X)$ . Regarding this, Tran (2018a,b) proposed the Multi-Task GAE (MTGAE). Here, the MT-reconstruction loss is defined as:
$$
\mathcal {L} _ {\mathcal {R}} = \mathcal {L} _ {\left(a _ {i}, \hat {a} _ {i}\right)} + \mathcal {L} _ {\left(x _ {i}, \hat {x} _ {i}\right)} \tag {5}
$$
where  $\mathcal{L}_{(a_i,\hat{a}_i)}$  and  $\mathcal{L}_{(x_i,\hat{x}_i)}$  both are the standard cross-entropy loss with sigmoid function  $\sigma (\cdot)$ , as,
$$
\mathcal {L} _ {\left(a _ {i}, \hat {a} _ {i}\right)} = - a _ {i} \log \left(\sigma \left(\hat {a} _ {i}\right)\right) - \left(1 - a _ {i}\right) \log \left(1 - \sigma \left(\hat {a} _ {i}\right)\right) \tag {6}
$$
$$
\mathcal {L} _ {(x _ {i}, \dot {x} _ {i})} = - x _ {i} l o g (\sigma (\dot {x} _ {i})) - (1 - x _ {i}) l o g (1 - \sigma (\dot {x} _ {i}))
$$
Variational Graph Autoencoder (VGAE) is an extension of the GAE architecture proposed by Kipf and Welling (2016). VGAE extends GAE by introducing an inference encoder, which is defined as:
$$
q (Z | X, A) = \prod_ {i = 1} ^ {n} q \left(z _ {i} | X, A\right) \tag {7}
$$
$$
q (z _ {i} | X, A) = \mathcal {N} (x _ {i} | \mu_ {i}, d i a g (\sigma^ {2}))
$$
$\mu = Z^2$  is a matrix of mean vectors  $z_{i}$ ,  $\sigma = f_{l}$  linear  $(Z^{1},A|W^{1})$  is the covariance matrix. During training, the VGAE optimizes the variational lower bound as:
$$
\mathcal {L} _ {\mathcal {R}} = \mathcal {L} _ {\mathcal {R}} + K L (q (Z | X, A) | | p (Z)) \tag {8}
$$
$KL(q(\cdot)||p(\cdot))$  denotes the Kullback-Leibler divergence and  $p(Z) = \prod_{i}\mathcal{N}(z_i|0,I)$  denotes the Gaussian prior for the latent data distribution. We perform the reparameterization trick (Kingma and Welling, 2014) to train the variational model.
Adjusted Mutual Information (AMI), and Accuracy (ACC)
Here, we describe the details of AMI and ACC. AMI is formally defined as:
$$
A M I (U, C) = \frac {M I (U , C) - E \{M I (U , C) \}}{\operatorname {a v g} \{H (U) , H (C) \} - E \{M I (U , C) \}} \tag {9}
$$
$U$  and  $C$  are the ground truth and predict classes (resp.).  $MI$  and  $H$  stand for Mutual Information and Entropy (resp.).  $E\{MI\}$  stands for the expected mutual information.
ACC is formally defined as:
$$
A C C = \frac {\sum_ {i = 1} ^ {N} \delta \left(y _ {i} = \operatorname {m a p} \left(c _ {i}\right)\right)}{N} \tag {10}
$$
where  $\delta (\cdot)$  is an indicator function,  $c_{i}$  is the predicted label for  $x_{i}$ ,  $map(\cdot)$  transforms the predicted label  $c_{i}$  to its group label by the Hungarian algorithm (Papadimitriou and Steiglitz, 1982), and  $y_{i}$  is the ground truth of  $x_{i}$ .