SlowGuess/NAACL_2021_OCR huggingface dataset:
Xingwu Sun $^{1*}$ , Yanling Cui $^{2,3}$ , Hongyin Tang $^{2,3}$ , Qiuyu Zhu $^{1}$ , Fuzheng Zhang $^{1}$ , Beihong Jin $^{2,3*}$ $^{1}$ Meituan Inc., Beijing, China
$^{2}$ State Key Laboratory of Computer Sciences, Institute of Software, Chinese Academy of Sciences
*The corresponding authors: sunxingwu01@gmail.com, beihong@iscas.ac.cn
In this paper, we focus on the problem of keyword and document matching by considering different relevance levels. In our recommendation system, different people follow different hot keywords with interest. We need to attach documents to each keyword and then distribute the documents to people who follow these keywords. The ideal documents should have the same topic with the keyword, which we call topic-aware relevance. In other words, topic-aware relevance documents are better than partially-relevance ones in this application. However, previous tasks never define topic-aware relevance clearly. To tackle this problem, we define a three-level relevance in keyword-document matching task: topic-aware relevance, partially-relevance and irrelevance. To capture the relevance between the short keyword and the document at above-mentioned three levels, we should not only combine the latent topic of the document with its deep neural representation, but also model complex interactions between the keyword and the document. To this end, we propose a Two-stage Interaction and Topic-Aware text matching model (TITA). In terms of "topic-aware", we introduce neural topic model to analyze the topic of the document and then use it to further encode the document. In terms of "two-stage interaction", we propose two successive stages to model complex interactions between the keyword and the document. Extensive experiments reveal that TITA outperforms other well-designed baselines and shows excellent performance in our recommendation system.
The keyword-document matching is mostly like the query-document matching task. The query-document matching task, aiming to calculate relevance score between a query and a document, has been extensively studied over the past few years. It is widely applicable in many real scenarios: (1) in
the information retrieval systems (Guo et al., 2016), query-document matching is an important feature in the ranking models. (2) as for the task of question answering (Yang et al., 2016), query-document matching method can be used to find document candidates or to help predict the answer span. (3) it is also widely applied to recommendation systems (Jiang et al., 2019).
In many scenarios, we need to distinguish different keyword-document (query-document) relevance levels. For instance, in our recommendation system, we need to attach documents to some hot keywords and then distribute the documents to the people who follow the keywords. In this circumstance, the document and the keyword should better have the same topic, which we call topic-aware relevance. As shown in Table 1, for the hot keyword "cherry blossoms", the document (labeled 2) should be the ideal document which should be attached because it has the same topic with the keyword while the document (labeled 1) should be a secondary choice, because only several words or phrases in this document match the keyword but the topics of the document mismatch the keyword.
To tackle this problem, we define a three-level relevance: topic-aware relevance, partially-relevance and irrelevance. The topic-aware relevance means the keyword and the document have the same topic while the partially-relevance means only part of the document matches with the keyword. Our task is more challenging than previous query-document matching tasks. To capture the relevance between the keyword and the document at above-mentioned three levels, we should not only combine the latent topic of the document with its deep representation, but also model complex interactions between the keyword and the document.
Previous neural query-document matching models (similar as keyword-document matching) can be divided into two categories according to their model architectures (Guo et al., 2016). One is the
001   
002   
003   
004   
005   
006   
007   
008   
009   
010   
011   
012   
013   
014   
015   
016   
017   
018   
019   
020   
021   
022   
023   
024   
025   
026   
027   
028   
029   
030   
031   
032   
033   
034   
035   
036   
037   
038   
039   
040   
041   
042   
043
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5431-5440 June 6-11, 2021. ©2021 Association for Computational Linguistics
044   
045   
046   
047   
048   
049   
050   
051   
052   
053   
054   
055   
056   
057   
058   
059   
060   
061   
062   
063   
064   
065   
066   
067   
068   
069   
070   
071   
072   
073   
074   
075   
076   
077   
078   
079   
080   
081   
082   
083   
084
Keyword: cherry blossoms
Original Keyword: 樱花
Label: 0
Irrelevance Case
Translated Document: There was a flower shop which has opened for a few months. I bought some flowers to decorate my house. The shop had common flowers such as lilies and carnations, but there were not many colors to be chosen...
Original Document: 这家花店开了有几个月了。我买了一些花来装饰我的房子。店里有百合花、康乃馨等普通花卉，但可供选择的颜色不多...
Label: 1
Partially-Relevance Case
Translated Document: The food in this restaurant is very delicious. I tried some dishes, such as foie gras, steak, squid, noodles, desserts, etc. All the dishes are really yummy, especially the filt mignon... By the way, there is a cherry blossoms exhibition near this restaurant.
Original Document: 这家餐馆的菜都很好吃，我试吃了一些菜，如鹅肝、牛排、鱿鱼、面条、甜点等。所有的菜都很好吃，尤其是菲力牛排...顺便说一下，这家餐厅附近有樱花展。
Label: 2
Topic-Aware Relevance Case
Translated Document: Yuyuantan Park is the best to enjoy cherry blossoms. The cherry blossoms in the park are available in a variety of colors and varieties. Their flowering period is short...
Original Document: 玉渊潭公园是赏樱花的最佳去处。公园里的樱花有各种颜色和品种。它们的花期很短...
Table 1: A piece of example describing three levels of keyword-document relevance: topic-aware relevance, partially-relevance and irrelevance, which are labeled 2, 1 and 0 respectively. As for the keyword "cherry blossoms", the topic-aware relevance case and the partially-relevance case both have some words relevant to the keyword. However, the document, labeled 2, has the same topic with the keyword. By contrast, the topic of the partially-relevance document, labeled 1, is "restaurant", which mismatches the keyword. Note that this case is translated from Chinese.
representation-based models, in which representations for a query and a document are built independently. In other words, there are no word-level or phrase-level interactions between the query and the document. For instance, the well-known DSSM (Huang et al., 2013) has been verified effective in query-document matching tasks. However, these representation-based series cannot model complex interactive signals between a query and a document effectively. The other one we call interaction-based models, in which word or phrase-level information fusion occurs. It has been verified more effective to directly learn interactions than individual representations. Examples include ARC II (Hu et al., 2014), MatchPyramid (Pang et al., 2016). Recently, interaction-based methods are widely used in many NLP tasks, like BIDAF (Seo et al., 2016) and R-NET (Wang et al., 2017).
More recently, BERT (Devlin et al., 2018) has made great influence in the field of NLP. It has achieved state-of-the-art results in many NLP applications. The pre-trained language models can be
applied directly to this keyword-document matching task.
However, these above-mentioned types of keyword-document (query-document) matching models can be improved to be applied to our recommendation system in the following aspects: (1) They do not analyse the topic of the document. It is expected that topic model can be used to solve this problem. (2) Previous interaction-based models can still be improved to capture complex matching signals between a query and a document. To this end, we propose the TITA model. By topic-aware, we introduce neural topic model (Miao et al., 2017) to analyze the latent topic representation of the document and then use this latent topic to further encode the document. By two-stage interaction, we propose a two-stage interaction to model complex interactions between a query and a document.
Our research contributions can be summarized as follows.
- We observe two major shortcomings in current keyword-document matching models and
107   
108   
109   
110   
111   
112   
113   
114   
115   
116   
117   
118   
119   
120   
121   
122   
123   
124   
125   
126   
127   
128
085   
086   
087   
088   
089   
090   
091   
092   
093   
094   
095   
096   
097   
098   
099   
100   
101   
102   
103   
104   
105   
106
propose the TITA model to improve them. Our model has two advantages: (1) it encodes the latent topic embedding into the deep neural representation of the document, which can aid the prediction of the topic-aware relevance. (2) it can model more complex interactions between a keyword and a document through a two-stage keyword-document interaction.
- We perform extensive experiments on our keyword-document matching dataset. The results reveal that the proposed TITA model outperforms the well-designed baselines.
- From a real recommendation system, we define a three-level relevance in keyword-document matching task and construct a new dataset.
- Our model is applied in our recommendation system and improves the click-through rate by  $4.35\%$ .
Depending on the model architectures, text matching models can be divided into two categories: representation-based and interaction-based. The former ones first transform every piece of text to a representation with neural networks, such as Deep Semantic Similarity Model(DSSM) (Huang et al., 2013), Convolutional Deep Semantic Similarity Model(CDSSM) (Shen et al., 2014), LSTM-RNN (Palangi et al., 2016), Bi-LSTM, etc. Conversely, the latter models focus on modeling the interaction between a query and a document, such as Arc-II(Hu et al., 2014), MatchPyramid (Pang et al., 2016), BIDAF(Seo et al., 2016) and RNET(Wang et al., 2017).
Representation-based methods generate distributed representations from input texts through neural networks. There are a number of works employing these methods, which differ mainly in the procedure to construct the representations and the way of calculating a matching score. Huang et al. (2013) propose DSSM, which is the first one to apply a neural network. In DSSM, each piece of the query or the document is represented through a multilayer perceptron and then a matching score is calculated by the cosine similarity. Compared to traditional text matching models, DSSM shows significant improvements.
Compared with representation-based methods, the interaction-based methods aim to capture di
rect matching features: the degree and the structure of matching. The interaction-based model, which means query-document interaction occurs before matching, can somewhat solve the above-mentioned problem in the representation-based models. It has been verified more effective to directly learn interactions than individual representations. Hu et al. (2014) propose ARC-II, which first represents the query and the document by the knowledge of each other, and adjusts the sliding windows in the first convolution layer to focus on adjacent word vectors. Inspired by the success of convolutional neural network in image recognition, Pang et al. (2016) propose MatchPyramid to model text matching as the problem of image recognition. Leveraging the attention mechanism, Seo et al. (2016) and Wang et al. (2017) introduce attention mechanism to improve the matching degree of the query and the document.
Recently BERT (Devlin et al., 2018) has caused a stir in the field of NLP. It has achieved state-of-the-art results in many NLP applications. The pre-trained language model series can be applied directly to this keyword-document matching task.
Topic models aim to discover the topics as well as the topic representations of documents in the document collection. It learns latent topics from documents in an unsupervised manner. Topics are captured as latent variables that have a word probability distribution. Topic models have a long tradition in this scenario area as well, such as bibliometrics, translations and recommendations.
Hall et al. (2008) describe the flow of topics between papers. Zhao and Xing (2006) enable word alignment process to leverage topical contents of document-pairs. Jiang et al. (2015) use topic model to enrich users' information for effective inference.
In this section, we describe details of the TITA model. As depicted in Figure 1, our TITA model has three major components: (1) a two-stage keyword-document interaction, see Part A; (2) a neural topic model, see Part B; (3) a joint training mechanism, see Part C. First, we introduce the task definition. Then, we elaborate the two-stage keyword-document interaction and neural topic model in the TITA model respectively. Finally, a joint training mechanism is introduced to incorporate latent topics to the deep representation
129   
130   
131   
132   
133   
134   
135   
136   
137   
138   
139   
140   
141   
142   
143   
144   
145   
146   
147   
148   
149   
150   
151   
152   
153   
154   
155   
156   
157   
158   
159   
160   
161   
162   
163   
164   
165   
166   
167   
168   
169   
170   
171   
172   
173   
174   
175   
176   
177
178   
179   
180   
181   
182   
183   
184   
185   
186   
187   
188   
189   
190   
191   
192   
193   
194   
195   
196   
197   
198   
199   
200   
201   
202   
203   
204   
205   
206   
207   
208   
209   
210   
211   
212   
213   
214   
215   
216   
217   
218   
219   
220   
221   
222   
223   
224   
225   
226   
227
Figure 1: The architecture of the TITA model, which consists of three major components: (1) a two-stage keyword-document interaction, which combines the multi-head attention in BERT and a successive cross representation layer to link the keyword and the document; (2) a neural topic model, which calculate a latent topic of the document to further enrich the document representation; (3) a joint training mechanism to train the model in a joint process. In this part, "T" indicates the input memory while "O" indicates the output memory.
of the document and train the model in a joint process. Notably, we conduct experiments using both Bi-LSTM and BERT as text encoders. Here, we only describe the proposed methods with BERT as the encoder for simplicity.
In our keyword-document matching task, we explicitly model the relevance between a keyword and a document as a relevance level prediction task. The input of the task is a keyword  $Q$  and a document  $D$ . The output  $r^{Q,D} \in \{0,1,2\}$  indicates the keyword-document relevance levels.
3.2 A Two-stage Keyword-document Interaction
The keyword-document matching model is desired to capture the rich interactions between the keyword and the document in the matching process. As show in Table 1, the keyword "cherry blossoms" and the topic-aware relevance document have many correlating signals, e.g., the phrase "cherry blossoms" in the keyword and the phrase "flowering period" in the document.
The two-stage keyword-document interaction in the TITA model is to fuse the information of the document and the keyword. In the first-stage interaction, we employ BERT (Devlin et al., 2018) as the encoder to simultaneously model the sequential information of the keyword and the document along with their interactive relationship
by the multi-head self-attention mechanism. In the second-stage interaction, we perform a cross-attention between the representations of the keyword and the document to further capture their interactive relationship.
First-stage Interaction As shown in Figure 1, in the first-stage interaction, we concatenate the keyword and the document by a separator [SEP] as input and then feed them into BERT. The input consists of the keyword characters  $c^Q = \{c_m^Q\}_{m=1}^M$  and the document characters  $c^D = \{c_n^D\}_{n=1}^N$ , where  $M$ ,  $N$  indicate the length of the keyword characters and the document characters respectively. The states in the last hidden layer of BERT can be regarded as the encoding of the document, i.e.,  $e^D$ .
$$
e ^ {D} = \operatorname {B E R T} ([ c ^ {Q}; [ \mathrm {S E P} ]; c ^ {D} ]) \tag {1}
$$
where  $e^{D} = \{e_{n}^{D}\}_{n = 1}^{N}\in \mathbb{R}^{N\times d}$ . In each hidden layer of BERT, the multi-head self-attention mechanism is performed as the following equations:
$$
\operatorname {A t t e n t i o n} (\tilde {Q}, \tilde {K}, \tilde {V}) = \operatorname {s o f t m a x} \left(\frac {\tilde {Q} \tilde {K} ^ {T}}{\sqrt {d _ {k}}}\right) \tilde {V} \tag {2}
$$
$$
\operatorname {M u l t i H e a d} (\tilde {Q}, \tilde {K}, \tilde {V}) = \operatorname {C o n c a t} \left(\mathrm {h d} _ {1}, \dots , \mathrm {h d} _ {h}\right) W ^ {O} \tag {3}
$$
$$
\mathrm {h d} _ {i} = \text {A t t e n t i o n} \left(\tilde {Q} W _ {i} ^ {\tilde {Q}}, \tilde {K} W _ {i} ^ {\tilde {K}}, \tilde {V} W _ {i} ^ {\tilde {V}}\right) \tag {4}
$$
where  $\tilde{Q},\tilde{K}$  and  $\tilde{V}$  are the output hidden states of the former layer.  $W_{i}^{\tilde{Q}},W_{i}^{\tilde{K}}$  and  $W_{i}^{\tilde{V}}$  are the parameters corresponding to each head.  $W^{O}$  is
228   
229   
230   
231   
232   
233   
234   
235   
236   
237   
238   
239   
240   
241   
242   
243   
244   
245   
246   
247   
248   
249   
250   
251   
252   
253   
254   
255   
256
257   
258   
259   
260   
261   
262   
263   
264   
265   
266   
267   
268   
269   
270   
271   
272   
273   
274   
275   
276   
277   
278   
279   
280   
281
the output projection parameter. For more details, readers can refer to (Devlin et al., 2018).
Second-stage Interaction Note that in the first-stage interaction, the query and the document characters are concatenated as input. The model learns keyword-keyword, keyword-document, document-document interactions simultaneously through self-attention mechanism in transformer blocks of BERT. In our keyword-document matching task, keyword-document interaction is more important than document-document and keyword-keyword interactions. Therefore, we introduce the second-stage interaction layer to conduct keyword-document contextualization independently. Firstly, we obtain the representation of the keyword  $e^Q$  by the BERT encoder.
$$
e ^ {Q} = \operatorname {B E R T} \left(c ^ {Q}\right) \in \mathbb {R} ^ {M \times d} \tag {5}
$$
where  $e^Q = \{e_m^Q\}_{m=1}^M$ . Then, we compute a similarity matrix using the keyword embedding and the document embedding.
$$
S = \left(s _ {m n}\right) \in \mathbb {R} ^ {M \times N} \tag {6}
$$
$$
s _ {m n} = \left\langle e _ {m} ^ {Q}, e _ {n} ^ {D} \right\rangle v ^ {T} \in \mathbb {R} \tag {7}
$$
where  $\langle e_m^Q,e_n^D\rangle$  represents a element-wise multiplication,  $v\in \mathbb{R}^{d}$  is a trainable weight vector. In this similarity matrix, the value  $s_{mn}$  indicates the link between the  $m$ -th character embedding in the keyword and the representation of the  $n$ -th character in the document. Then, we apply this similarity matrix to further encode the keyword by calculating attention over the document:
$$
u ^ {Q} = \left\{u _ {m} ^ {Q} \right\} _ {m = 1} ^ {M} \tag {8}
$$
$$
u _ {m} ^ {Q} = \sum_ {n = 1} ^ {N} a _ {m n} e _ {n} ^ {D} \in \mathbb {R} ^ {d} \tag {9}
$$
$$
a _ {m} = \operatorname {s o f t m a x} \left(s _ {m}\right) \in \mathbb {R} ^ {N} \tag {10}
$$
where  $s_m = \{s_{mn}\}_{n=1}^N$  and  $a_m$  means which characters in the document should be attended regarding the  $m$ -th character of the keyword. We then add the original keyword representation  $e^Q$  with  $u^Q$  to get the keyword embedding:
$$
u ^ {Q} = u ^ {Q} + e ^ {Q} \tag {11}
$$
Similarly, we use this similarity matrix to get the document representation  $u^{D} \in \mathbb{R}^{N \times d}$ .
3.3 Neural Topic Model
As show in Table 1, the topic-aware relevance case and the partially-relevance case both have some words relevant to "cherry blossoms". But the topic of the topic-aware relevance document is more related with the keyword "cherry bollosoms". By contrast, the topic of the partial-relevance document is more likely to be a document about a "restaurant", which is not related to the keyword "cherry blossom". Following this direction, analysing the topic of the document is a way to promote keyword-document matching models. Specifically, we introduce neural topic model to produce the latent topic and then use it to update the upstream representation of the document.
As shown in Figure 1, the input of the neural topic model is a word sequence of the document  $w^{D}$ . The bag-of-words (BOW) representation of the document is  $x^{D} \in \mathbb{R}^{|V_{w}|}$ , where  $|V_{w}|$  is the size of the word vocabulary. Assume that the latent variable  $\theta$  represents the topic distribution in the document  $w^{D}$ . The probabilistic topic models, like LDA(Blei et al., 2003), apply the Dirichlet distribution as the prior of the latent variable  $\theta \sim Dir(\alpha)$  where  $\alpha$  is the parameter of the Dirichlet distribution. By contrast, in the neural topic model, Gaussian Softmax Construction (Miao et al., 2017) is applied using a neural network to parameterise the topic distribution  $\theta \sim G_{GSM}(\mu_0, \sigma_0^2)$ :
$$
x \sim \mathcal {N} \left(\mu_ {0}, \sigma_ {0} ^ {2}\right) \tag {12}
$$
$$
\theta = \operatorname {s o f t m a x} \left(W _ {1} ^ {T} x\right) \tag {13}
$$
where  $W_{1}$  is a trainable parameter.  $\mu_0$  and  $\sigma_0$  are the parameters of the prior Gaussian distribution  $\mathcal{N}$ . Assuming there are  $K$  topics, if  $z_{n}\in \{1,\dots,K\}$  is the topic assignment for the observed word  $w_{n}^{D}$ , then:
$$
z _ {n} \sim \operatorname {M u l t i} (\theta) \tag {14}
$$
$\beta_{z_n} \in \mathbb{R}^{|V_w|}$  is a topic distribution over the words in the vocabulary given  $z_n$ . The topic distribution can be calculated by the similarity between the topic and the words in the vocabulary:
$$
\beta_ {z _ {n}} = \operatorname {s o f t m a x} \left(\tilde {v} ^ {T} t _ {z _ {n}}\right) \tag {15}
$$
where  $t \in \mathbb{R}^{d \times K}$  is the topic vector which is a parameter of the neural topic model,  $\tilde{v} \in \mathbb{R}^{d \times |V_w|}$  is the word vector.  $K$  is the total topic number. Then, the generative probability of each word  $w_n$
282   
283   
284   
285   
286   
287   
288   
289   
290   
291   
292   
293   
294   
295   
296   
297   
298   
299   
300   
301   
302   
303   
304   
305   
306   
307   
308   
309   
310   
311   
312   
313   
314   
315   
316   
317   
318   
319   
320   
321   
322
can be calculated by:
$$
p \left(w _ {n} \mid \beta_ {z _ {n}}\right) = \operatorname {M u l t i} \left(\beta_ {z _ {n}}\right) \tag {16}
$$
The neural topic model is implemented by an Auto-Encoding Variational Bayes (AEVB) algorithm (Kingma and Welling, 2013). The encoder is used to approximate the true posterior of the latent variable  $p(\theta | x)$ . Specifically, the encoder takes the BOW (Bag-of-Words) representation of the document as the input and generates the posterior Gaussian Softmax Construction parameters  $\mu$  and  $\sigma^2$  through neural networks. In practice, the latent variable  $\theta$  is sampled by the reparameterization trick.
$$
\begin{array}{l} \mu = f _ {1} \left(x ^ {D}\right), \log \sigma = f _ {2} \left(x ^ {D}\right) (17) \\ \theta \sim G _ {G S M} (\mu , \sigma^ {2}) (18) \\ \end{array}
$$
where  $f_{*}(\cdot)$  is a multi-layer perceptron. The decoder is responsible for reconstructing the document by maximizing the log likelihood of the input document. The latent variable  $z_{n}$  can be integrated out as follows.
$$
\begin{array}{l} \log p \left(w _ {n} \mid \beta , \theta\right) = \log \sum_ {z _ {n}} \left[ p \left(w _ {n} \mid \beta_ {z _ {n}}\right) p \left(z _ {n} \mid \theta\right) \right] (19) \\ = \log (\theta \cdot \beta) (20) \\ \end{array}
$$