SlowGuess/NAACL_2021_OCR huggingface dataset:
87.2% accuracy and 88.4% F1 score, which is 7.1% and 6.5% better than the best baseline method (DSHT). The improvements over all the baselines are statistically significant at a p-value of 0.01. The baseline methods including InferSent, BERT have high recalls, but low F1 scores and precision. This is because these approach cannot distinguish unmatched documents but predict most of documents are matched.
4.2.2 Summary-Reference Matching
Table 2 shows the result of Summary-Reference Matching task in CL-SciSumm 2017 dataset. From the results, we conclude that our approach outperforms all the baselines on all metrics. The results are statistically significant at  $p < 0.01$  using the Wilcoxon signed rank test (Smucker et al., 2007). Since the summary-reference task only has one true reference, the other citing papers being distractors, the P@1 result becomes especially important for this task. Our approach achieves  $90\%$  precision, which is  $3.3\%$  better than the precision of the best baseline method (WMD). We also find the global representation methods such as BERT and InferSent performs worse than approaches using local features such as WMD and TF-IDF, which is different from the results in concept-project dataset. But our concept-based approach that utilizes both local and global features has consistently outperforms these baseline methods.
4.2.3 Summary-Citance Matching
Figure 2 shows the precision at K result of summary-citance matching task in CL-SciSumm 2018 dataset when k is set from one to five. Both mean and variance are presented by 10 experimental runs. From the result, we conclude that our method can significantly outperform the other baselines for all the settings of K. Specifically, our model performs around  $5\%$  better than the best baseline method, WMD. Moreover, the variance of our model is also much smaller than the other baselines, which indicates that our model is less impacted by random selected distractors. The result of WRD is not available to compute due to its out-of-memory issue. In addition, we also find the similar results that local feature-based approaches perform better than global representation methods.
4.2.4 Ablation Study
To verify the effectiveness of designed components in our approach, we make an ablation study in the
$^{5}$ https://fasttext.cc/
Acc	Prec	Recall	F1
TF-IDF	0.538	0.540	0.993	0.700
InferSent	0.540	0.541	0.997	0.701
BERT	0.548	0.545	1.000	0.706
WMD	0.685	0.656	0.880	0.752
WRD	0.704	0.678	0.863	0.759
DSHT	0.801	0.807	0.832	0.819
CGNet	0.872	0.865	0.904	0.884
Table 1: Performance result of Concept-Project
MRR	P@1	NDCG
TF-IDF	0.918	0.867	0.938
InferSent	0.457	0.267	0.581
BERT	0.181	0.033	0.357
WMD	0.933	0.867	0.951
WRD	0.701	0.533	0.773
DSHT	0.555	0.367	0.661
CGNet	0.944	0.900	0.959
Table 2: Result of Summary-Reference Matching
following settings: (i)  $w/o$  Sinkhorn: To demonstrate the effectiveness of the sinkhorn-based reconstruction loss, we remove the sinkhorn loss and instead simply use mean-square-error between the embeddings of phrases and concepts. (ii)  $w/o$  Cluster Divergence Loss (CDL): We remove the cluster divergence loss in Equation (4) in our training process. (iii)  $w/o$  Cluster Evidence Loss (CEL): To show the effectiveness of the concept evidence loss, we remove the CEL in the concept learning process in Equation (5). (iv)  $w/o$  Enhanced Concept-weight Constraint (ECC): We replace the concept-weight constraint to one in our concept mover's distance to demonstrate the performance of the module.
Table 4 shows the results of the ablation study, which demonstrates that each component improves the overall performance in concept-project matching task, across our evaluation metrics. This indicates that our modeling choices are suited to tackle the inherent challenges involved in matching the length-varying documents. In particular, the cluster divergence loss has great impact on the performance since the loss can avoid assigning all the cluster centroids to the same value.
We conduct several experiments to investigate the impact of the following two hyper-parameters: concept number and phrase length. (i) Concept Num
Figure 2: Result of Summary-Citance Matching
Figure 3: Parameter Analysis of Concepts Number
ber. Figure 3 shows the results in concept-project dataset using different concept numbers from 1 to 30. We conclude that both the F1 score and Accuracy can continuously be improved when the number of concepts are increased to 10. After the concept number reaches to 10, the performance starts to degrade but still keep in a high level, which indicates that our model is not sensitive to the setting of concept number. (ii) Phrase Length. Figure 4 shows the performance results in concept-project dataset using different settings of phrase length. From the results, we conclude that the token-level phrases have the best performance compared to other settings even including the combination of length 1 and 2. The main reason is that the 2-gram features contain a large portion of noisy phrases that make the extracted concepts less effective for document matching.
The running time of training are shown in Table 5. We can see the training time is increased linearly when the data size is increased. Since our model can be converged in a few epochs (usually
Phrase assignments for each concept
Concept-1	fat, fur, fats, blubber, adipose, whale, beluga, oils, mammal, blubber adipose, carbohydrates, warm blooded, colligative, ” or, fats, animal, calories, adipose, protein, mitochondria
Concept-2	sea, isle, tide, seas, tides, compass, vessel, boat, islands, ocean, waters, pirate, currents, winds, oceans, atlantic, ship, coast, oceanic, waves
Concept-3	bug, bugs, ants, bee, bees, insect, insects, katydid, spiders, grasshoppers, sowbugs, weevil, pillbugs, crickets, snails, peanuts, flies, do more, lions, peanut
Concept-4	odor, smell, scent, smells, rancid, taste, rancidity, emit, fishy, gone, emitting, tastes, auditory, sounds, emits, emitted, stimuli, unpleasant, bloom, sensation
Concept-5	cow, cows, age, milk, rex, ages, formula, rennet, lactase, horses, tablet, formulas, gestation, pasteurizing, calcium, matrix, ratio, breast, milkshake, cream
Table 3: Case Study of Concepts
Acc	Prec	Recall	F1
w/o Sinkhorn	0.866	0.890	0.859	0.874
w/o CDL	0.562	0.555	0.976	0.707
w/o CEL	0.859	0.848	0.900	0.873
w/o ECC	0.805	0.780	0.890	0.832
CGNet	0.872	0.865	0.904	0.884
Figure 4: Parameter Analysis of Ngrams
less than 100 epochs), our model can be trained in a reasonable duration. Moreover, we find the evaluation time of each dataset has less difference compared to the training time since the evaluation time is related to the size of phrases and concepts.
4.4 Interpretation of Concepts
Table 3 give some interpretation of concepts, showing top-20 phrases ranked by the phrase assignment probability in Equation (2) of five concepts generated by our CGNet model. From the results, we conclude that: (i) The generated concept is capable of representing high-level topics. For instance, the Concept-1 relates to fat and energy of sea mammals when phrases such as fat, blubber adipose and carbohydrates appear; the Concept-2 relates to the sea sailing when phrases such as tide, isle, compass
Table 4: Result of Ablation Study
Concept-Project	Summary-Reference	Summary-Citance
Training Time (sec/epoch)	20.76	6.35	10.95
Eval Time (sec/pair)	0.412	0.143	0.281
Table 5: Efficiency Result for Training (second/epoch) and Testing (second/pair).
are assigned to the concept. (ii) phrases in concepts are not only grouped by the similar semantics but the inherent co-occurrence in the text corpus. For example, the calories and mammal shares very few semantic similarity but these two terms can be connected by documents that introduce the energy storage system of sea mammals. (iii) The 2-gram phrases can introduce useful phrases such as "blubber adipose" in Concept-1. However, sometimes it produces some noises such as "do more" in Concept-3.
In this paper, an unsupervised concept representation learning method is proposed to address the length-varying text comparison problem. To achieve this, we propose a deep neural network based model to generate corpus-level concept representation and design a concept-based document matching method based on augmented concept representation that leverages the advances of both local phrase features and global concept features. Extensive experiments on real-world datasets demonstrated that our proposed method dramatically outperforms competing methods, exhibiting a significant improvement in all the metrics in different length-vary text comparison tasks.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2016. Enriching word vectors with subword information. arXiv preprint arXiv:1607.04606.
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017a. Supervised learning of universal sentence representations from natural language inference data. arXiv preprint arXiv:1705.02364.
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017b. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670-680, Copenhagen, Denmark. Association for Computational Linguistics.
Marco Cuturi. 2013. Sinkhorn distances: Lightspeed computation of optimal transport. In Advances in neural information processing systems, pages 2292-2300.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Nicolas Fournier and Arnaud Guillin. 2015. On the rate of convergence in Wasserstein distance of the empirical measure. *Probability Theory and Related Fields*, 162(3-4):707-738.
Vivek Gupta, Ankit Saw, Pegah Nokhiz, Praneeth Netrapalli, Piyush Rai, and Partha Talukdar. 2020. P-sif: Document embeddings using partition averaging. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7863-7870.
Suma Bhat Hongyu Gong, Tarek Sakakini and Jinjun Xiong. 2018. Document similarity for texts of varying lengths via hidden topics. In the 56th Annual Meeting of the Association for Computational Linguistics (ACL). Association for Computational Linguistics.
Ting Hua, Xuchao Zhang, Wei Wang, Chang-Tien Lu, and Naren Ramakrishnan. 2016. Automatical story-line generation with help from twitter. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, pages 2383-2388.
Gao Huang, Chuan Guo, Matt J Kusner, Yu Sun, Fei Sha, and Kilian Q Weinberger. 2016. Supervised word mover's distance. In Advances in Neural Information Processing Systems, pages 4862-4870.
Kokil Jaidka, Michihiro Yasunaga, Muthu Kumar Chandrasekaran, Dragomir Radev, and MinYen Kan. 2019. The cl-scisumm shared task 2018: Results and key insights. arXiv preprint arXiv:1909.00764.
Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in neural information processing systems, pages 3294-3302.
Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. 2015. From word embeddings to document distances. In International conference on machine learning, pages 957-966.
Xiaodong Liu, Kevin Duh, and Jianfeng Gao. 2018. Stochastic answer networks for natural language inference. arXiv preprint arXiv:1804.07888.
Christopher D Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008. Introduction to information retrieval. Cambridge university press.
Donald Metzler, Susan Dumais, and Christopher Meek. 2007. Similarity measures for short segments of text. In European conference on information retrieval, pages 16-27. Springer.
Ankur P Parikh, Oscar Tackstrom, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. arXiv preprint arXiv:1606.01933.
Brandon Pincombe. 2004. Comparison of human and latent semantic analysis (lsa) judgements of pairwise document similarities for a news corpus. Technical report, DEFENCE SCIENCE AND TECHNOLOGY ORGANISATION SALISBURY (AUSTRALIA) INFO ....
Animesh Prasad. 2017. Wing-nus at cl-scisumm 2017: Learning from syntactic and semantic similarity for citation contextualization. In BIRNDL@ SIGIR (2), pages 26-32.
Mark D Smucker, James Allan, and Ben Carterette. 2007. A comparison of statistical significance tests for information retrieval evaluation. In Proceedings of the sixteenth ACM conference on Conference on information and knowledge management, pages 623-632.
Ming Tan, Cicero Dos Santos, Bing Xiang, and Bowen Zhou. 2016. Improved representation learning for question answer matching. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 464-473.
Zihao Wang, Datong Zhou, Yong Zhang, Hao Wu, and Chenglong Bao. 2019. Wasserstein-fisher-rao document distance. arXiv preprint arXiv:1904.10294.
Runqi Yang, Jianhai Zhang, Xing Gao, Feng Ji, and Haiqing Chen. 2019a. Simple and effective text matching with richer alignment features. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4699-4709, Florence, Italy. Association for Computational Linguistics.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019b. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems, pages 5754-5764.
Wenpeng Yin and Hinrich Schütze. 2015. Convolutional neural network for paraphrase identification. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 901-911.
Sho Yokoi, Ryo Takahashi, Reina Akama, Jun Suzuki, and Kentaro Inui. 2020. Word rotator's distance: Decomposing vectors gives better representations. arXiv preprint arXiv:2004.15003.
Xuchao Zhang, Wei Cheng, Bo Zong, Yuncong Chen, Jianwu Xu, Ding Li, and Haifeng Chen. 2020. Temporal context-aware representation learning for question routing. In Proceedings of the 13th International Conference on Web Search and Data Mining, pages 753-761.
Xuchao Zhang, Dheeraj Rajagopal, Michael Gamon, Sujay Kumar Jauhar, and ChangTien Lu. 2019. Modeling the relationship between user comments and edits in document revision. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5003-5012.
Xuchao Zhang, Liang Zhao, Zhiqian Chen, Arnold P Boedihardjo, Jing Dai, and Chang-Tien Lu. 2017. Trendi: Tracking stories in news and microblogs via emerging, evolving and fading topics. In 2017 IEEE International Conference on Big Data (Big Data), pages 1590-1599. IEEE.
Xixi Zhou, Chengxi Li, Jiajun Bu, Chengwei Yao, Keyue Shi, Zhi Yu, and Zhou Yu. 2020. Matching text with deep mutual information estimation. arXiv preprint arXiv:2003.11521.