SlowGuess/NAACL_2021_OCR huggingface dataset:
Xuchao Zhang, Bo Zong, Wei Cheng, Jingchao Ni, Yanchi Liu, Haifeng Chen
NEC Laboratories America, Princeton, NJ, USA
{xuczhang,bzong,weicheng,jni,yanchi,haifeng}  $@$  nec-labs.com
Measuring document similarity plays an important role in natural language processing tasks. Most existing document similarity approaches suffer from the information gap caused by context and vocabulary mismatches when comparing varying-length texts. In this paper, we propose an unsupervised concept representation learning approach to address the above issues. Specifically, we propose a novel Concept Generation Network (CGNet) to learn concept representations from the perspective of the entire text corpus. Moreover, a concept-based document matching method is proposed to leverage advances in the recognition of local phrase features and corpus-level concept features. Extensive experiments on real-world data sets demonstrate that new method can achieve a considerable improvement in comparing length-varying texts. In particular, our model achieved  $6.5\%$  better F1 Score compared to the best of the baseline models for a concept-project benchmark dataset.
Measuring the similarity between documents is a fundamental problem in several natural language tasks such as information retrieval (Manning et al., 2008), paraphrase identification (Yin and Schütze, 2015) and question routing (Zhang et al., 2020). A wide range of document similarity approaches (Kusner et al., 2015; Huang et al., 2016) have been proposed to handle the fundamental problem; however, most of them are based on the assumption that the documents being compared have similar document length. However, varying-length document matching tasks are ubiquitous in many real-world scenarios. For instance, in the news categorization task, the news articles may include both short reports for breaking news or narrative reports with cumbersome details.
The document matching in varying length may
introduce the information gap between two documents in the following two aspects: (i) context mismatch, which is caused by the long-length documents usually provide more detailed context to support the key information while the short-length documents contain limited context information. The issue renders the existing pre-trained natural language representation models (Conneau et al., 2017a; Devlin et al., 2018) pay more attention to the long but less important contexts, which makes their document representations distinct from the short-length documents with little context information. (ii) vocabulary mismatch, which is usually caused by the different terms usage between short and long texts, which leads them do not share majority terms. Existing document distance such as word mover's distance (Kusner et al., 2015) focus on comparing the local features. Still, the vocabulary mismatch issue makes the local features hard to be matched while the majority of vocabulary is not shared.
To address the above challenges, our approach proposes a concept-based document matching method that incorporates both local phrase features and corpus-level concepts in an unsupervised setting, where concepts can be interpreted as a group of representative features that are interpretable for humans. The main contributions of this paper can be summarized as follows: (i) A novel unsupervised concept generation network is proposed to learn corpus-level concepts in the perspective of entire text corpus. Specifically, each concept and its phrase assignment is iteratively optimized by the reconstruction loss between local phrase features and global concept representations. (ii) A new concept-based document comparison method is proposed to measure the similarity between two text documents based on augmented concept representations, which leverages the advances of local phrases and corpus-level concepts. Moreover, an enhanced concept-weight constraint is proposed to
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5611-5620 June 6-11, 2021. ©2021 Association for Computational Linguistics
improve the performance in optimizing the concept-based document similarity. (iii) Extensive experiments on several length-varying text matching datasets demonstrate that the effectiveness of our proposed approach consistently outperforms existing state-of-the-art methods. In particular, our method improved  $7.1\%$  Accuracy and  $6.5\%$  F1-score in concept-project dataset compared to the best baseline method.
The rest of this paper is organized as follows. Section 2 reviews related work, and Section 3 provides a detailed description of our proposed model. The experiments on multiple real-world data sets are presented in Section 4. The paper concludes with a summary of the research in Section 5.
In this section, we briefly describe recent advances in document similarity research. We start our discussion with recent progress in supervised methods, and then we shift our focus to unsupervised settings.
2.1 Supervised Methods
A large group of previous studies (Parikh et al., 2016; Liu et al., 2018; Zhang et al., 2019; Gupta et al., 2020; Zhang et al., 2020) learns document matching model between two text sequences in supervised settings. Tan et al. (2016) exploit attention mechanism to distil important words from sentences. Yang et al. (2019a) propose an intersequence alignment approach considering both previous aligned features and original point-wise features. Zhou et al. (2020) present a neural approach for general-purpose text matching with deep mutual information estimation. However, these semantic alignment approaches require massive human annotations in their training process, which are expensive and infeasible to obtain in many real-world scenarios.
2.2 Unsupervised Methods
Some approaches can be used to match document in unsupervised manners, including traditional statistical approaches (Metzler et al., 2007; Pincombe, 2004; Hua et al., 2016; Zhang et al., 2017). In past few years, neural-network-based methods have been used for document representation, which includes Doc2Vec (Conneau et al., 2017b), SkipThought vectors (Kiros et al., 2015). More recently, the state-of-the-art representation methods focus
on the contextual representations to encode words in their context such as BERT (Devlin et al., 2018) and XLNet (Yang et al., 2019b). A comparably long text may lose its local information after being encoded as a fix-length representation due to the informative contexts. Word Mover's Distance (WMD) approaches (Yokoi et al., 2020; Wang et al., 2019) can partially solve the problem since they focus on local feature matching. However, these methods still suffer from the vocabulary mismatch issue from length-varying texts, which makes the local features hard to be matched since texts share different majority of vocabulary terms.
Few approaches consider the length-varying texts in unsupervised settings. Hongyu Gong and Xiong (2018) proposed an unsupervised document matching approach by comparing documents in a common space of hidden topics (DSHT), which is optimized by Singular Value Decomposition (SVD). Compared to this approach, our method leverages both local features and global corpus-level concepts while DSHT only compares corpus-level topics. Moreover, the proposed CGNet can generate concepts in more scalable data set compared to the matrix decomposition solution in DSHT.
We now describe our approach to calculate the document similarity for length-varying texts. We begin by introducing the overview of our model in Section 3.1. Then we provide details of the concept generation and document matching components in Section 3.2 and 3.3. Last, the implementation details are described in Section 3.4.
Given a corpus of documents  $\mathcal{D} = \{d_1, d_2, \dots, d_n\}$ , we propose a concept-based document matching approach to compute the document distance  $\mathrm{dist}(d_i, d_j)$  between any two documents  $d_i$  and  $d_j$  in the corpus. The overall architecture is shown in Figure 1, which includes two main components: 1) Concept Generation, which is to generate the corpus-level concepts from the entire document corpus. Each concept  $c_i$  consists of a group of document phrases by minimizing the reconstruction loss between local phrase representation and global concept representation. Moreover, both cluster divergence and evidence regularization terms are proposed to regularize the generated concepts. 2) Doc
Figure 1: Overall Architecture
ument Matching. After the corpus-level concepts are learned from previous step, document matching is to calculate the document similarity based on concept-based document comparison method. Specifically, the concept-based similary adopt the Wasserstein distance (Fournier and Guillin, 2015) to compute similarity between two documents' concept representations in terms of enhanced concept-weight constraint.
3.2 Concept Generation
To generate concepts from a document corpus, we propose an unsupervised Concept Generation Network (CGNet). First, we extract a set of phrases  $S_{p}$  from the text corpus  $\mathcal{D}$ . The extracted phrases can be in different formats such as word tokens, noun phrases or n-grams according to the data corpus and language. Then, pre-trained language representation models such as Transformers (Devlin et al., 2018; Yang et al., 2019b) can be adopted to encode the extracted phrases into embeddings as their semantic representations. Specifically, we denote the embedding of the  $i$ -th phrase in document  $d_{j}$  as  $p_{i}^{(j)} \in \mathbb{R}^{\theta}$ , where  $\theta$  is the dimension of the phrase embedding. Suppose  $\sigma(d_{j})$  is the number of phrases in document  $d_{j}$ , we denote the phrase embedding set  $\mathbb{P}^{(j)}$  for document  $d_{j}$  as  $\mathbb{P}^{(j)} = \{p_{i}^{(j)} \mid i \leq \sigma(d_{j}), i \in \mathbb{Z}^{+}\}$ , where  $\mathbb{Z}^{+}$  represents the set of positive integers. Specifically, we use  $\mathbb{P} = \bigcup_{i=1}^{n} \mathbb{P}^{(i)}$  to represent the entire phrase set for all the documents.
We assume each document can not only be represented as a group of phrases but a set of corpus-level concepts, which are treated as good approxi
mations of phrase representations. Especially for short-length texts, the limited phrases makes phrase representation hard to represent both text semantics and phrase importance. Instead, our concept representation can represent short-text semantics and weight document features in corpus perspective rather than individual document.
To learn the corpus-level concepts, we first randomly initialize  $\kappa$  concept centroid embeddings in the same feature space of phrases, where  $\kappa$  is the number of concepts. Specifically, we denote  $c_{i} \in \mathbb{R}^{\theta}$  as the embedding of the  $i$ -th concept centroid, where the concept dimension  $\theta$  shares the same dimension as phrase representation. Noted that the concept centroid embeddings will be trained as model parameters in our CGNet model.
Then we assign each phrase to concepts based on its phrase embedding and concept centroids by student-t distribution as follows:
$$
s _ {i k} ^ {(j)} = \frac {\left(1 + \| \boldsymbol {p} _ {i} ^ {(j)} - \boldsymbol {c} _ {k} \| ^ {2} / \alpha\right) ^ {- \frac {\alpha + 1}{2}}}{\sum_ {k ^ {\prime} = 1} ^ {\kappa} \left(1 + \| \boldsymbol {p} _ {i} ^ {(j)} - \boldsymbol {c} _ {k ^ {\prime}} \| ^ {2} / \alpha\right) ^ {- \frac {\alpha + 1}{2}}}, \tag {1}
$$
where  $s_{ik}^{(j)}$  can be interpreted as the probability of the  $i$ -th phrase in document  $d_j$  assigned to the  $k$ -th concept. Since Student-t distribution has heavier tails, which makes it more prone to producing values that fall far from its mean. This characteristics can help to assign lower probability to phrases that do not belong to any concept. The parameter  $\alpha$  can control the degrees of freedom of Student's t-distribution. Since our unsupervised setting, we let  $\alpha = 1$  for all experiments.
Based on the phrase assignment on each concept, the concept representation for document  $d_{j}$  can be
represented as:
$$
\mathbb {C} ^ {(j)} = \left\{\boldsymbol {c} _ {k} \big | s _ {i k} ^ {(j)} \geq \gamma , \forall i \leq \sigma (d _ {j}), \forall k \leq \kappa \right\}, (2)
$$
where  $\mathbb{C}^{(j)}$  is the set of concept centroid embeddings for document  $d_{j}$  and  $\gamma$  is a threshold to assign concepts for each document. When the probability  $s_{ik}^{(j)}$  is greater than  $\gamma$ , the concept  $c_{k}$  is added into the concept embedding set  $\mathbb{C}^{(j)}$ ; otherwise, the concept  $c_{k}$  is excluded.
To improve the concept assignment, we propose to optimize the concept centroids by minimizing the reconstruction loss between local phrases and corpus-level concepts for each document. The reconstruction loss is defined as follows:
$$
\mathcal {L} _ {r} = \frac {1}{n} \sum_ {i} ^ {n} \operatorname {s i n k h o r n} \left(\mathbb {P} ^ {(i)}, \mathbb {C} ^ {(i)}\right), \tag {3}
$$
where  $\mathbb{P}^{(i)}$  and  $\mathbb{C}^{(i)}$  represents the embedding sets of phrases and concepts for the  $i$ -th document, respectively. Function sinkhorn( $\cdot$ ) represents sinkhorn divergence (Cuturei, 2013), a sensible approximation of the Wasserstein distance (Fournier and Guillin, 2015) at a low computational cost. The experimental results in Section 4.2.4 show the sinkhorn divergence achieves empirical better performance than traditional mean squared error (MSE).
Only minimizing the reconstruction loss can easily get trivial local optima that assigns all the phrases to one concept. Thus, we propose two regularization terms, concept divergence loss and concept evidence loss, to regularize the concept centroid and avoid trivial solutions.
Concept Divergence. To prevent the similar or even duplicate concepts, we propose a divergence regularization term  $\mathcal{L}_d$  that penalizes on concepts that are close to each other. The regularization term  $\mathcal{L}_d$  is defined as follows:
$$
\mathcal {L} _ {d} = \sum_ {i = 1} ^ {\kappa} \sum_ {j = i + 1} ^ {\kappa} \max  \left(0, \mu - \| \boldsymbol {c} _ {i} - \boldsymbol {c} _ {j} \| _ {2} ^ {2}\right), \tag {4}
$$
where  $\mu$  is a threshold that justifies whether two concepts are similar or not. We set  $\mu$  to 1.0 in our experiments. The divergence regularization exerts a large penalty when the  $L_{2}$  norm distance between two concept embeddings are smaller than the threshold  $\mu$ ; otherwise, no penalty is produced.
Concept Evidence. To encourage each concept as close to encoded phrase instances, we propose
a concept evidence regularization term  $\mathcal{L}_e$ , which penalizes the long distance between each concept embedding and its corresponding closest encoded phrases. The evidence regularization term  $\mathcal{L}_e$  is defined as follows:
$$
\mathcal {L} _ {e} = \frac {1}{\kappa} \sum_ {k = 1} ^ {\kappa} \sum_ {j = 1} ^ {\tau} \min  _ {j} \left(\bigcup_ {\boldsymbol {p} _ {i} \in \mathbb {P}} \| \boldsymbol {c} _ {k} - \boldsymbol {p} _ {i} \| _ {2} ^ {2}\right), \tag {5}
$$
where  $\min_j(\cdot)$  represents the  $j$ -th minimum value in the given set and  $\pmb{p}_i \in \mathbb{P}$  is one of the phrase embedding from the entire phrase embedding set  $\mathbb{P}$ . We denote  $\cup$  as the union operator to combine all the  $L_2$  norm distance between concept centroids and phrase embeddings. We choose the sum of top- $\tau$  minimum distances as the concept evidence loss for each concept. The value of  $\tau$  determines the minimum number of phrases we desired in each concept. By default, we set  $\tau$  to five, which indicates a large penalty is produced while the  $k$ -th  $(k \leq \tau)$  closest phrases has a long distance to the concept centroid.
Finally, our loss function is the combination of reconstruction loss  $\mathcal{L}_r$ , concept divergence loss  $\mathcal{L}_d$  and concept evidence loss  $\mathcal{L}_e$  with their corresponding weights  $\lambda_r, \lambda_d$  and  $\lambda_e$ .
3.3 Concept-based Document Matching
Our concept-based document matching method is based on the concept generated in Section 3.2. According to the document concept assignment in Equation (2), some local phrases are excluded from any concept while its concept assignment probability  $s_{ik}^{(j)} < \gamma$  for  $\forall k < \kappa$ . However, these local phrases may contain distinguished semantics that cannot be grouped with enough phrases as a concept, but play an important role in distinguishing the difference between documents. To involve the local phrases into our document matching task, we generate a local-feature augmented representation  $\mathbb{C}_{\dagger}^{(j)}$  as follows:
$$
\mathbb {C} _ {\dagger} ^ {(j)} = \mathbb {C} ^ {(j)} \cup \left\{\boldsymbol {p} _ {i} ^ {(j)} \Bigg | \bigcup_ {\substack {i \leq \sigma (d _ {j}) \\ j \leq \kappa}} \left\{\boldsymbol {c} _ {k} \mid s _ {i k} ^ {(j)} \geq \gamma \right\} = \emptyset \right\}, \tag{6}
$$
where all the local phrases that do not belong to any concept in  $\mathbb{C}^{(j)}$  are added into the augmented concept embedding set  $\mathbb{C}_{\dagger}^{(j)}$ . The parameter  $\gamma$  is the same threshold as in Equation (2).
Based on the idea of the Wasserstein distance (Fournier and Guillin, 2015), we propose the
concept-based document similarity  $\Psi$  between augmented concept representations of documents  $d_{i}$  and  $d_{j}$  as follows:
$$
\begin{array}{l} \Psi (\mathbb {C} _ {\dagger} ^ {(p)}, \mathbb {C} _ {\dagger} ^ {(q)}) = \max  \sum_ {\boldsymbol {c} _ {i} \in \mathbb {C} _ {\dagger} ^ {(p)}} \sum_ {\boldsymbol {c} _ {j} \in \mathbb {C} _ {\dagger} ^ {(q)}} f _ {i, j} \frac {\boldsymbol {c} _ {i} \cdot \boldsymbol {c} _ {j}}{\| \boldsymbol {c} _ {i} \| \| \boldsymbol {c} _ {j} \|} \\ \text {s . t .} \quad \sum_ {i \in \mathbb {Z} ^ {+}} f _ {i, j} \leq w _ {p, i}, \forall i \leq \left| \mathbb {C} _ {\dagger} ^ {(p)} \right| \\ \sum_ {j \in \mathbb {Z} ^ {+}} f _ {i, j} \leq w _ {q, j}, \quad \forall j \leq \left| \mathbb {C} _ {\dagger} ^ {(q)} \right|, \tag {7} \\ \end{array}
$$
where the  $f_{i,j}$  is a flow from concept representation  $\pmb{c}_i$  in  $\mathbb{C}_{\dagger}^{(p)}$  to  $\pmb{c}_j$  in  $\mathbb{C}_{\dagger}^{(q)}$ . Parameters  $w_{p,i}$  and  $w_{q,j}$  represent the weight of concept  $i$  and  $j$  in document  $p$  and  $q$ , respectively. We choose the concept weight as the averaged TF-IDF weight of phrases that are assigned to the concept, which is used as upper bound constraint of the flow parameters. Overall, the concept-based document similarity is to find a flow between concept representations of two documents that maximize the similarity score.
The proposed CGNet model described in this section is implemented using the Pytorch $^1$  framework and trained on a single Nvidia Quadro RTX 6000 GPU with 24GB memory. For phrase extraction, we set the minimum phrase frequency to 10 and maximum document frequency to 0.5. The phrases embeddings are initialized with pre-trained fastText model (Bojanowski et al., 2016) using the default dimensionality of 300. We set the number of training epochs of CGNet to 100 and batch size to 8. For the sinkhorn divergence used in Equation (3), we apply an approximate Wasserstein distance implementation $^2$ . For the settings of concepts, we set number of concept  $\kappa$  to 100 and concept threshold  $\gamma$  to 0.8. It should be noted that while we train our CGNet with the text corpus, the model – once trained – can be applied to new document in the same domain that is not included in the text corpus.
In this section, we evaluate the performance of the model described in Section 3 on document matching task for length-varying texts.
We begin by introducing the evaluation settings, with details on the datasets, metrics and baselines that we use in our experiments.
4.1.1 Datasets and Labels
We conducted experiments on three publicly available datasets in different tasks: (i) Concept-Project (Hongyu Gong and Xiong, 2018). The dataset is to match science projects and concepts when people intend to search related projects that match a given concept. It includes 537 pairs of projects and concepts involving 53 unique concepts from the Next Generation Science Standards<sup>3</sup> (NGSS) and 230 unique projects from Science Buddies<sup>4</sup>. Each pair is labeled by human beings with the decision wether it is a good match or not. (ii) CL-SciSumm 2017 (Prasad, 2017). The dataset consists of 494 ACL Computational Linguistics research papers covering 30 categories in total. Each category contains a reference paper and its corresponding human-annotated summary. We compare the reference summary with its corresponding reference paper and use all the citing papers as negative cases. The matching task is formulated as a ranking problem and use the reference paper as the top-1 ground-truth. (iii) CL-SciSumm 2018 (Jaidka et al., 2019). The dataset consists of 605 research papers with reference papers including summaries and citing papers, which covers 40 categories. Different from dataset CL-SciSumm 2017, we randomly select 5 corresponding citing papers as the true candidate for each reference summary and choose the other 15 citing papers from all the citing papers as distractors.
For Concept-Project dataset, we use Accuracy, Precision, Recall and F1-score as evaluation metrics based on the binary classification predictions. The metrics including Precision, Recall and F1-score are based on positive predictions.
For the CL-SciSumm 2017 dataset, we use popular ranking evaluation metrics from the literature, which includes: (i) Precision@1: The proportion of predicted instances where the true reference paper appears in the ranked top-1 result. (ii) Mean Reciprocal Rank (MRR): the average multiplicative inverse of the rank of the correct answer, represented mathematically as MRR =  $\frac{1}{N}\sum_{i=1}^{N}\frac{1}{\mathrm{rank}_i}$ ,
<sup>3</sup>https://www.nextgenscience.org/
<sup>4</sup>https://www.sciencebuddies.org/
<sup>1</sup>https://pytorch.org/
$^{2}$ https://github.com/dfdazac/wassdistance
where  $N$  is the number of samples and  $\mathrm{rank}_i$  is the rank assigned to the true comment by a model. (iii) Normalized Discounted Cumulative Gain (NDCG): the normalized gain of each reference paper based on its ranking position in the results. We set the relevance score of the true comment to one and those of the distractors (citing papers) to zero.
For the CL-SciSumm 2018 dataset, the results are evaluated by Precision@K: The proportion of predicted instances where the true citing papers appear in the ranked top-K result. For example, P@3 or "Precision at 3" corresponds to the percentage of cases where the true citing appears in the top 3 ranked results. We vary the value of K from 1 to 5 in our experiments.
4.1.3 Competing Methods
The following methods are included in the performance comparison: (i) TF-IDF, which uses the cosine similarity between the TfIdf-weighted vectors of the document as a measure of document similarity. (ii) Infersent, which finds the cosine similarity between document embeddings generated by the state-of-the-art sentence embedding method InferSent (Conneau et al., 2017a). (iii) BERT, which uses inner product between the document representation generated by the pre-trained deep bidirectional transformer (Devlin et al., 2018). (iv) WMD (Kusner et al., 2015), which uses word mover's distance metric based on the embeddings of document words generated by fastText $^5$ . (v) WRD (Yokoi et al., 2020), which is a variant of traditional WMD method. WRD separates word importance and word meaning by decomposing word vectors into their norm and direction. The alignment-based similarity is computed by earth mover's distance. (vi) DSHT (Hongyu Gong and Xiong, 2018), which matches documents by comparing them in a common space of hidden topics.
We now present and discuss the empirical results of our evaluation for the three document matching tasks.
4.2.1 Concept-Project Matching
Table 1 summarizes results of the concept-project document matching task. Our model significantly outperforms all the baselines in accuracy, precision and F1-score. In particular, our model achieves