SlowGuess/NAACL_2022_OCR huggingface dataset:
Mingyang Song, Yi Feng and Liping Jing*
Beijing Key Lab of Traffic Data Analysis and Mining
Beijing Jiaotong University, China
mingyang.song@bjtu.edu.cn
Keyphrase extraction is a fundamental task in natural language processing that aims to extract a set of phrases with important information from a source document. Identifying important keyphrases is the central component of keyphrase extraction, and its main challenge is learning to represent information comprehensively and discriminate importance accurately. In this paper, to address the above issues, we design a new hyperbolic matching model (HyperMatch) to explore keyphrase extraction in hyperbolic space. Concretely, to represent information comprehensively, HyperMatch first takes advantage of the hidden representations in the middle layers of RoBERTa and integrates them as the word embeddings via an adaptive mixing layer to capture the hierarchical syntactic and semantic structures. Then, considering the latent structure information hidden in natural languages, HyperMatch embeds candidate phrases and documents in the same hyperbolic space via a hyperbolic phrase encoder and a hyperbolic document encoder. To discriminate importance accurately, HyperMatch estimates the importance of each candidate phrase by explicitly modeling the phrase-document relevance via the Poincaré distance and optimizes the whole model by minimizing the hyperbolic margin-based triplet loss. Extensive experiments are conducted on six benchmark datasets and demonstrate that HyperMatch outperforms the recent state-of-the-art baselines.
Keyphrase Extraction (KE) aims to extract a set of phrases related to the main points discussed in the source document, a fundamental task in Natural Language Processing (NLP). Because of their succinct and accurate expression, keyphrase extraction is helpful for a variety of applications such as information retrieval (Kim et al., 2013) and text summarization (Liu et al., 2009a).
Figure 1: Sample partial of the document in OpenKP dataset. For ease of presentation, we assume "a large region of land" is a 5-gram candidate phrase as an example in the document.
Typically, most existing keyphrase extraction models mainly include two procedures: candidate keyphrase extraction and keyphrase importance estimation. Specifically, the former extracts candidate phrases from the document via some heuristics (Nguyen and Phan, 2009; Liu et al., 2009b; Grineva et al., 2009; Wan and Xiao, 2008; Liu et al., 2009c), and the latter determines which candidate phrases are keyphrases via unsupervised or supervised models (Mihalcea and Tarau, 2004; Xiong et al., 2019; Sun et al., 2020; Song et al., 2021). The keyphrase importance estimation procedure usually plays a more critical role than the candidate keyphrase extraction procedure in the supervised setting.
In the supervised neural keyphrase extraction models, the keyphrase importance estimation procedure can be subdivided into information representation and importance discrimination. Specifically, the information representation part focuses on modeling the encoding procedure, and the importance discrimination part focuses on measuring the important scores of candidate phrases. To represent information comprehensively, recent keyphrase extraction studies have been proposed to build better representations via Bi-LSTM (Meng et al., 2017), GCNs (Sun et al., 2019; Zhang et al., 2020), and the
Human Language Technologies, pages 5710 - 5720
pre-trained language models (e.g., ELMo (Xiong et al., 2019), BERT, and RoBERTa (Liu et al., 2020; Sun et al., 2020)). To discriminate the importance of candidate phrases precisely, most existing supervised keyphrase extraction models (Sun et al., 2020; Mu et al., 2020; Song et al., 2021) estimate and rank the importance of candidate phrases to extract keyphrases by using different approaches, such as classification and ranking models.
Although the existing keyphrase extraction models mentioned above have achieved significant performance, the keyphrase extraction task still needs improvement. Among them, there are the following two main issues. The first issue lies in the information representation. Typically, candidate phrases often exhibit the inherent hierarchical structures ingrained with complex syntactic and semantic information (Dai et al., 2021; Zhou et al., 2020). In general, the longer phrases contain more complex structures. (as shown in Figure 1, the phrase "a large region of land" has more complex inherent structures than "region" or "a large region". Similarly, the phrase "a large region" is more complex than "region"). Besides the phrases, since linguistic ontologies are intrinsic hierarchies (Dai et al., 2021), the conceptual relations between phrases and their corresponding document can also form the hierarchical structures. Therefore, the hierarchical structures need to be considered when representing both phrases and documents and estimating the phrase-document relevance. However, it is difficult to capture such structural information even with infinite dimensions in the Euclidean space (Linial et al., 1995). The second issue lies in distinguishing the importance of phrases. Keyphrases are typically used to retrieve and index their corresponding document, so they should be highly related to the main points of the source document (Hasan and Ng, 2014). However, most existing supervised keyphrase extraction methods ignore explicitly modeling the relevance between candidate keyphrases and their corresponding document, resulting in biased keyphrase extraction.
Motivated by the above issues, in this paper, we explore the potential of hyperbolic space for the keyphrase extraction task and propose a new hyperbolic relevance matching model (HyperMatch) for supervised neural keyphrase extraction. Firstly, to capture hierarchical syntactic and semantic structure information, HyperMatch integrates the hidden representations in all the intermediate layers of
RoBERTa to collect the adaptive contextualized word embeddings via an adaptive mixing layer based on the self-attention mechanism. And then, considering the hierarchical structure hidden in the natural language content, HyperMatch represents both phrases and documents in the same hyperbolic space via a hyperbolic phrase encoder and a hyperbolic document encoder. Meanwhile, we adopt the Poincaré distance to calculate the phrase-document relevance by considering the latent hierarchical structures between candidate keyphrases and the document. In this setting, the keyphrase extraction task can be regarded as a matching problem and effectively implemented by minimizing a hyperbolic margin-based triplet loss. To the best of our knowledge, we are the first work to explore the supervised keyphrase extraction in hyperbolic space. Experiments on six benchmark datasets demonstrate that HyperMatch outperforms the state-of-the-art keyphrase extraction baselines.
Hyperbolic space is an important concept in hyperbolic geometry, which is considered as a special case in the Riemannian geometry (Hopper and Andrews, 2011). Before presenting our model, this section briefly introduces the basic information of hyperbolic space.
In a traditional sense, hyperbolic spaces are not vector spaces; one cannot use standard operations such as summation, multiplication, etc. To remedy this problem, one can utilize the formalism of Möbius gyrovector spaces allowing the generalization of many standard operations to the hyperbolic spaces (Khrulkov et al., 2020). Similarly to the previous work (Nickel and Kiela, 2017; Ganea et al., 2018; Tifrea et al., 2019), we adopt the Poincaré ball and use an additional hyper-parameter  $c$  which modifies the curvature of Poincaré ball; it is then defined as  $\mathbb{D}_c^n = \{\mathbf{x} \in \mathbb{R}^n : c\| \mathbf{x}\|^2 < 1, c \geq 0\}$ . The corresponding conformal factor now takes the form  $\lambda_{\mathbf{x}}^c \coloneqq \frac{2}{1 - c\|\mathbf{x}\|^2}$ . In practice, the choice of  $c$  allows one to balance the hyperbolic and the euclidean geometries, which is made precise by noting that when  $c \to 0$ , all the formulas discussed below take their usual Euclidean form.
In the following, we restate the definitions of fundamental mathematical operations for the generalized Poincaré ball model (Ganea et al., 2018). Next, we give the details of the closed-form formulas of several Möbius operations.
Figure 2: Framework of the hyperbolic relevance matching model (HyperMatch).
Mobius Addition. For a pair  $\mathbf{x},\mathbf{y}\in \mathbb{D}_c^n$  , the Mobius addition is defined as,
$$
\mathbf {x} \oplus_ {c} \mathbf {y} = \frac {(1 + 2 c \langle \mathbf {x} , \mathbf {y} \rangle + c \| \mathbf {y} \| ^ {2}) \mathbf {x} + (1 - c \| \mathbf {x} \| ^ {2}) \mathbf {y}}{1 + 2 c \langle \mathbf {x} , \mathbf {y} \rangle + c ^ {2} \| \mathbf {x} \| ^ {2} \| \mathbf {y} \| ^ {2}}. \tag {1}
$$
Mobius Matrix-vector Multiplication. For a linear map  $\mathbf{M}:\mathbb{R}^n\to \mathbb{R}^m$  and  $\forall \mathbf{x}\in \mathbb{D}_c^n$  , if  $\mathbf{Mx}\neq 0$  then the Mobius matrix-vector multiplication is defined as,
$$
\mathbf {M} \otimes_ {c} \mathbf {x} = (\frac {1}{\sqrt {c}}) \tanh  (\frac {\| \mathbf {M x} \|}{\| \mathbf {x} \|} \tanh  ^ {- 1} (\| \sqrt {c} \mathbf {x} \|)) \frac {\mathbf {M x}}{\| \mathbf {M x} \|}, \tag {2}
$$
where  $\mathbf{M} \otimes_{c} \mathbf{x} = 0$  if  $\mathbf{M}\mathbf{x} = 0$ .
Poincaré Distance. The induced distance function is defined as,
$$
d _ {c} (\mathbf {x}, \mathbf {y}) = \frac {2}{\sqrt {c}} \operatorname {a r c t a n h} (\sqrt {c} \| - \mathbf {x} \oplus_ {c} \mathbf {y} \Vert). \tag {3}
$$
Note that with  $c = 1$  one recovers the geodesic distance, while with  $c\to 0$  we obtain the Euclidean distance  $\lim_{c\to 0}d_c(\mathbf{x},\mathbf{y}) = 2\| \mathbf{x} - \mathbf{y}\|$
Exponential and Logarithmic Maps. To perform operations in hyperbolic space, one first needs to define a mapping function from  $\mathbb{R}^n$  to  $\mathbb{D}_c^n$  to map the euclidean vectors to the hyperbolic space. Let  $T_{\mathbf{x}}\mathbb{D}_c^n$  denote the tangent space of  $\mathbb{D}_c^n$  at  $\mathbf{x}$ . The exponential map  $\exp_{\mathbf{x}}^{c}(\cdot): T_{\mathbf{x}}\mathbb{D}_{c}^{n}\to \mathbb{D}_{c}^{n}$  for  $\mathbf{v}\neq 0$  is defined as:
$$
\exp_ {\mathbf {x}} ^ {c} (\mathbf {v}) = \mathbf {x} \oplus_ {c} \left(\tanh  \left(\sqrt {c} \frac {\lambda_ {\mathbf {x}} ^ {c} \| \mathbf {v} \|}{2}\right) \frac {\mathbf {v}}{\sqrt {c} \| \mathbf {v} \|}\right). \tag {4}
$$
As the inverse of  $\exp_{\mathbf{x}}^{c}(\cdot)$ , the logarithmic map  $\log_{\mathbf{x}}^{c}(\cdot):\mathbb{D}_{c}^{n}\to T_{x}\mathbb{D}_{c}^{n}$  for  $\mathbf{y}\neq \mathbf{x}$  is defined as:
$$
\log_ {\mathbf {x}} ^ {c} (\mathbf {y}) = \frac {2}{\sqrt {c} \lambda_ {\mathbf {x}} ^ {c}} \tanh  ^ {- 1} \left(\sqrt {c} \| - \mathbf {x} \oplus_ {c} \mathbf {y} \|\right) \frac {- \mathbf {x} \oplus_ {c} \mathbf {y}}{\| - \mathbf {x} \oplus_ {c} \mathbf {y} \|} \tag {5}
$$
Hyperbolic Averaging Pooling. The average pooling, as an important operation common in natural language processing, is averaging of feature vectors. In the euclidean setting, this operation takes the following form:
$$
\mathrm {A P} \left(\mathbf {x} _ {1}, \dots , \mathbf {x} _ {i}, \dots , \mathbf {x} _ {M}\right) = \frac {1}{M} \sum_ {i = 1} ^ {M} \mathbf {x} _ {i}. \tag {6}
$$
Extension of this operation to hyperbolic spaces is called the Einstein Midpoint and takes the most simple form in Klein coordinates:
$$
\operatorname {H y p e r A P} \left(\mathbf {x} _ {1}, \dots , \mathbf {x} _ {i}, \dots , \mathbf {x} _ {M}\right) = \sum_ {i = 1} ^ {M} \gamma_ {i} \mathbf {x} _ {i} / \sum_ {i = 1} ^ {M} \gamma_ {i}, \tag {7}
$$
where  $\gamma_{i} = \frac{1}{\sqrt{1 - c\|\mathbf{x}_{i}\|^{2}}}$  is the Lorentz factor. Recent work (Khrulkov et al., 2020) demonstrates that the Klein model is supported on the same space as the Poincaré ball; however, the same point has different coordinate representations in these models. Let  $\mathbf{x}_{\mathbb{D}}$  and  $\mathbf{x}_{\mathbb{K}}$  denote the coordinates of the same point in the Poincaré and Klein models correspondingly. Then the following transition formulas hold.
$$
\mathbf {x} _ {\mathbb {D}} = \frac {\mathbf {x} _ {\mathbb {K}}}{1 + \sqrt {1 - c \| \mathbf {x} _ {\mathbb {K}} \| ^ {2}}}, \tag {8}
$$
$$
\mathbf {x} _ {\mathbb {K}} = \frac {2 \mathbf {x} _ {\mathbb {D}}}{1 + c \| \mathbf {x} _ {\mathbb {D}} \| ^ {2}}. \tag {9}
$$
Therefore, given points in the Poincaré ball, we can first map them to the Klein model via Eq.(9), compute the average using Eq.(7), and then move it back to the Poincaré model via Eq.(8).
3 HyperMatch
Given a document  $\mathcal{D} = \{w_1, \dots, w_i, \dots, w_M\}$ , the candidate phrases are first extracted from the source document by the n-gram rules, where  $M$  indicates the max length of the input document. Then, to determine which candidate phrases are keyphrases, we design a new hyperbolic relevance matching model (HyperMatch), which mainly consists of two components: information representation and importance discrimination. Figure 2 illustrates the overall framework of HyperMatch.
3.1 Information Representation
Information representation is one of the essential parts of keyphrase importance estimation, which needs to represent information comprehensively. To capture rich syntactic and semantic information, HyperMatch first embeds words by the pre-trained language model RoBERTa with the adaptive mixing layer. Then, phrases and documents are embedded in the same hyperbolic space by a hyperbolic phrase encoder and a hyperbolic document encoder. In the following subsections, the information representation procedure will be described in detail.
3.1.1 Contextualized Word Encoder
Pre-trained language models (Peters et al., 2018; Devlin et al., 2019; Liu et al., 2019) have emerged as a critical technology for achieving impressive gains in natural language tasks. These models extend the idea of word embeddings by learning contextualized text representations from large-scale corpora using a language modeling objective. Thus, recent keyphrase extraction methods (Xiong et al., 2019; Sun et al., 2020; Wang et al., 2020; Mu et al., 2020) represent words / documents by the last intermediate layer of pre-trained language models.
However, various probing tasks (Jawahar et al., 2019; de Vries et al., 2020) are proposed to discover linguistic properties learned in contextualized word embeddings, which demonstrates that different intermediate layers in pre-trained language models contain different linguistic properties or information. Specifically, each layer has specific specializations, so combining features from different layers may be more beneficial than selecting the last one based on the best overall performance.
Motivated by the phenomenon above, we propose a new adaptive mixing layer to combine all intermediate layers of RoBERTa (Liu et al., 2019) to obtain word representations. Firstly, each word
in the source document  $\mathcal{D}$  is represented by all the intermediate layers in RoBERTa, which is encoded to a sequence of vector  $\mathbf{H} = \{\mathbf{h}_1,\dots ,\mathbf{h}_i,\dots ,\mathbf{h}_M\}$  as follows,
$$
\mathbf {H} = \operatorname {R o B E R T a} \left\{\mathbf {w} _ {1}, \dots , \mathbf {w} _ {i}, \dots , \mathbf {w} _ {M} \right\}. \tag {10}
$$
Specially,  $\mathbf{h}_i\in \mathbb{R}^{L*d_r}$  indicates the  $i$ -th contextualized word embedding of  $\mathbf{w}_i$ , where  $L$  and  $d_{r}$  are set to 12 and 768. Then, the self-attention mechanism is adopted to aggregate the multi-layer representations of each word from RoBERTa as follows:
$$
\alpha_ {i} = \operatorname {s o f t m a x} \left(\mathbf {V} _ {a} \mathbf {h} _ {i}\right), \tag {11}
$$
$$
\hat {\mathbf {h}} _ {i} = \mathbf {W} _ {a} \alpha_ {i} \mathbf {h} _ {i}, \tag {12}
$$
where  $\mathbf{V}_a\in \mathbb{R}^{d_r}$  and  $\mathbf{W}_a\in \mathbb{R}^{d_r*d_r}$  denote the learnable weights. Here,  $\alpha_{i}\in \mathbb{R}^{L}$  represents the adaptive mixing weights of the proposed adaptive mixing layer in HyperMatch. In this case, each word in the source document  $\mathcal{D}$  is transferred to a sequence of vector  $\hat{\mathbf{H}} = \{\hat{\mathbf{h}}_1,\dots,\hat{\mathbf{h}}_i,\dots,\hat{\mathbf{h}}_M\}$ . The adaptive mixing layer allows our model to obtain more comprehensive word embeddings, capturing more meaningful features (e.g., surface, syntactic, and semantic).
3.1.2 Hyperbolic Phrase Encoder
Phrases often exhibit inherent hierarchies ingrained with complex syntactic and semantic information (Zhu et al., 2020). Therefore, representing information requires sufficiently encoding semantic and syntactic information, especially for the latent hierarchical structures hidden in the natural languages. Recent studies (Sun et al., 2020; Xiong et al., 2019) typically obtain phrase representations in Euclidean space, which makes it difficult to learn representations with such latent structural information even with infinite dimensions in Euclidean space (Linial et al., 1995). On the contrary, hyperbolic spaces are non-Euclidean geometric spaces that can naturally capture the latent hierarchical structures (Sarkar, 2011; Sa et al., 2018).
Lately, the use of hyperbolic space in NLP (Dhingra et al., 2018; Tifrea et al., 2019; Nickel and Kiela, 2017) is motivated by the ubiquity of hierarchies (e.g., the latent hierarchical structures in phrases, sentences, and documents) in NLP tasks. Therefore, in this paper, we propose to embed phrases in hyperbolic space. Concretely, the phrase representation of the  $i$ -th  $n$ -gram  $c_i^n$  is computed as follows,
$$
\hat {\mathbf {h}} _ {i} ^ {n} = \operatorname {C N N} ^ {n} \left(\hat {\mathbf {h}} _ {i: i + n}\right), \tag {13}
$$
where  $\hat{\mathbf{h}}_i^n\in \mathbb{R}^{d_h}$  represents the  $i$  -th  $n$  -gram representation,  $n\in [1,N]$  indicates the length of ngrams, and  $N$  is the maximum length of n-grams. Each  $n$  -gram has its own set of convolution filters  $\mathrm{CNN}^n$  with window size  $n$  and stride 1.
To capture the latent hierarchies of phrases, we map phrases representation to the Poincaré ball using the exponential map,
$$
\tilde {\mathbf {h}} _ {i} ^ {n} = \exp_ {\mathbf {0}} ^ {c} \left(\hat {\mathbf {h}} _ {i} ^ {n}\right), \tag {14}
$$
where  $\tilde{\mathbf{h}}_i^n$  is the  $i$ -th  $n$ -gram candidate phrase representation in the hyperbolic space. By mapping the representations of candidate phrases into the hyperbolic space, it is possible to implicitly capture the latent hierarchical structure of candidate phrases during the training procedure.
3.1.3 Hyperbolic Document Encoder
When using the source document as the query to match keyphrases, the representation of the document should cover its main points (important information). Meanwhile, documents are usually long text sequences with richer semantic and syntactic information than candidate phrases. Many current BERT-based methods (Mu et al., 2020; Zhong et al., 2020) in NLP obtain documents representation by using the first output token (the [CLS] token) of the pre-trained language models. However, recent studies (Reimers and Gurevych, 2019; Li et al., 2020) demonstrate that in many NLP tasks, documents representation obtained by the average pooling of words representation is better than the [CLS] token.
Motivated by the above issues, we use the average pooling, a simple and effective operation, to encode documents. To further consider the latent hierarchical structures of documents, we map word representations and transfer the average pooling operation to the hyperbolic space. In this case, we first map word representations to the hyperbolic space via the exponential map as follows:
$$
\tilde {\mathbf {H}} = \left\{\tilde {\mathbf {h}} _ {1}, \dots , \tilde {\mathbf {h}} _ {i}, \dots , \tilde {\mathbf {h}} _ {M} \right\} = \exp_ {\mathbf {0}} ^ {c} \left(\hat {\mathbf {H}} \mathbf {W} _ {h}\right), \tag {15}
$$
where  $\mathbf{W}_h\in \mathbb{R}^{d_r*d_h}$  maps the original BERT embedding space to the tangent space of the origin of the Poincaré ball. Then  $\exp_0(\cdot)$  maps the tangent space inside the Poincaré ball. Next, we use the hyperbolic averaging pooling to encode the source document as follows:
$$
\tilde {\mathbf {h}} = \operatorname {H y p e r A P} \left(\left\{\tilde {\mathbf {h}} _ {1}, \dots , \tilde {\mathbf {h}} _ {i}, \dots , \tilde {\mathbf {h}} _ {M} \right\}\right), \tag {16}
$$
where  $\tilde{\mathbf{h}}\in \mathbb{R}^{d_h}$  indicates the hyperbolic document representation (called Einstein Midpoint pooling
vectors in the Poincaré ball (Gulcehre et al., 2019)). The hyperbolic average pooling emphasizes semantically specific words that usually contain more information but occur less frequently than general ones. It should be noted that points near the boundary of the Poincaré ball get larger weights in the Einstein Midpoint formula, which may be more representative content in the source document (Dhingra et al., 2018; Zhu et al., 2020).
3.2 Importance Discrimination
Importance discrimination is one of the primary parts of the keyphrase importance estimation procedure, which estimates the important scores of candidate phrases accurately to extract keyphrases. To reach this goal, we first calculate the scaled phrase-document relevance between candidate keyphrases and their corresponding document via the Poincaré distance as the important score of each candidate keyphrase. Then, the whole model is optimized by the hyperbolic margin-based triplet loss to extract keyphrases accurately.
3.2.1 Scaled Phrase-Document Relevance
Besides the intrinsic hierarchies of linguistic ontologies, the conceptual relations between candidate phrases and their corresponding document can also form hierarchical structures. Once the document representation  $\tilde{\mathbf{h}}$  and phrase representations  $\tilde{\mathbf{h}}_i^n$  are obtained, it is expected that the phrases and their corresponding document embedded close to each other based on their geodesic distance if they are highly relevant. Specifically, the scaled phrase-document relevance of the  $i$ -th  $n$ -gram representation  $c_i^n$  can be computed as follows:
$$
S \left(c _ {i} ^ {n}, \mathcal {D}\right) = - \frac {\lambda \left(d _ {c} \left(\tilde {\mathbf {h}} _ {i} ^ {n} , \tilde {\mathbf {h}}\right)\right) ^ {2}}{\sqrt {d _ {h}}} + (1 - \lambda) f _ {c} \left(\tilde {\mathbf {h}} _ {i} ^ {n}\right), \tag {17}
$$
where  $S(\cdot)$  indicates the scaled phrase-document relevance. Here,  $d_{c}$  indicates the Poincaré distance, which is introduced in Eq.(3). Here,  $f_{c}$  indicates the linear transformation in hyperbolic space. Specifically, for Eq. 17, the first term models the phrase-document relevance explicitly, and the second term models the phrase-document relevance implicitly. Estimating the phrase-document relevance via the Poincaré distance in hyperbolic
<sup>1</sup>Note that cosine similarity (Wang et al., 2017) is not appropriate to be the metric since there does not exist a clear hyperbolic inner-product for the Poincaré ball (Tifrea et al., 2019), so the Poincaré distance is more suitable.
space allows HyperMatch to model the latent hierarchical structures between candidate phrases and their document, accurately estimating the importance of candidate keyphrases. In addition, we find that increasing the dimension  $d_h$  of representations will increase the value of the phrase-document relevance, causing the optimization collapse of our model. To counteract this effect, we scale the phrase-document relevance by  $\frac{1}{\sqrt{d_h}}$ .
3.2.2 Margin-based Triplet Loss
To select phrases with higher importances, we adopt the margin-based triplet loss in our model and optimize for margin separation in hyperbolic space. Therefore, we first treat the candidate keyphrases in the document that are labelled as keyphrases, in the positive set  $\mathbf{P}^{+}$ , and the others to the negative set  $\mathbf{P}^{-}$ , to obtain the matching labels. Then, the loss function is calculated as follows:
$$
\mathcal {L} = \max  \left(0, \frac {\delta}{\sqrt {d _ {h}}} - S \left(p ^ {+}, \mathcal {D}\right) + S \left(p ^ {-}, \mathcal {D}\right)\right), \tag {18}
$$