SlowGuess/NAACL_2021_OCR huggingface dataset:
Finally, the variational lower bound of the neural topic model is obtained by combining the reconstruction error term and the KL divergence term. The parameters of neural topic model can be trained by maximizing this function.
$$
\begin{array}{l} \mathcal {L} _ {N T M} = \mathbb {E} _ {p ^ {\prime} (\theta | D)} \left[ \sum_ {n = 1} ^ {N} \log \sum_ {z _ {n}} [ p (w _ {n} | \beta_ {z _ {n}}) p (z _ {n} | \theta) ] \right] \\ - D _ {K L} \left[ p ^ {\prime} (\theta | D) \right| \left| p \left(\theta \mid \mu_ {0}, \sigma_ {0} ^ {2}\right) \right] \tag {21} \\ \end{array}
$$
where  $p^{\prime}(\theta |D)$  means the variational posterior distribution of document  $D$ , approximating the true posterior  $p(\theta |D)$ .
3.4 A Joint Training Mechanism
It's expected that introducing topic model can benefit the model in the prediction of the above-mentioned three levels. In this subsection, we design a joint training mechanism to incorporate the
latent topic representation to further encode the document and train the model in a joint process.
As described above,  $u^{D}$  is the document representation after the two-stage keyword-document interaction.  $\beta \in R^{K \times |V_{w}|}$  is the topic distribution over the vocabulary, where  $\beta_{ij}$  means that the weight between the  $i$ -th topic and the  $j$ -th word. We are inspired from an end2end memory network(Sukhbaatar et al., 2015), which is used to memorize multiple sentences in question answering task. Similarly, in TITA, we intend to embed the topic-word weight into the deep representation of the document.
As depicted in Figure 1 part C, the input of memory network is  $\beta$  and the deep document representation after the two-stage keyword-document interaction  $u^{D}$ .  $\beta$  is memorized in the memory of the network, where  $\beta_{k}$  means the representation of the  $k$ -th topic over the vocabulary of size  $|V_{w}|$ .
The TITA model has two memory hops as shown in the Figure 1. In the following, we describe the model in a single memory hop operation for simplicity. One hop has two major components: the input memory and the output memory. In the input memory representation, a matching score is calculated taking  $\beta$  and  $u^{D}$  as input:
$$
p _ {k} = \operatorname {s o f t m a x} \left(\beta_ {k} V u ^ {D}\right) \tag {22}
$$
where  $V \in R^{|V_w| \times d}$  is a trainable weight vector. In the output memory representation part, we compute the slot output vector using the output memory and the matching score:
$$
o ^ {D} = \sum_ {k = 1} ^ {K} \left(p _ {k} c _ {k}\right) \tag {23}
$$
$$
o ^ {D} = W _ {o} \left(o ^ {D} + u ^ {D}\right) \tag {24}
$$
where  $c \in \mathbb{R}^{K \times d}$  is a trainable output memory. We compute two relevance vectors  $r_1$  and  $r_2$ . One takes  $u^D$  and  $u^Q$  as input, while the other one using  $u^Q$  and  $o^D$ . We merge the two relevance vectors and then apply softmax function to get the final relevance level:
$$
r _ {1} = W _ {R 1} \left[ u ^ {D}; u ^ {Q} \right] + b _ {R 1} \tag {25}
$$
$$
r _ {2} = W _ {R 2} \left[ o ^ {D}; u ^ {Q} \right] + b _ {R 2} \tag {26}
$$
$$
r ^ {Q, D} = \operatorname {s o f t m a x} \left(W _ {R} \left[ r _ {1}; r _ {2} \right] + b _ {R}\right) \tag {27}
$$
where  $[;]$  is vector concatenation operation and  $W_{*}, b_{*}$  are all trainable variables.
369   
370   
371   
372   
373   
374   
375   
376   
377   
378   
379   
380   
381   
382   
383   
384   
385   
386   
387   
388   
389   
390   
391   
392   
393   
394   
395   
396   
397   
398   
399   
400   
401   
402   
403   
404
405   
406   
407   
408   
409   
410   
411   
412   
413   
414   
415   
416   
417   
418   
419   
420   
421   
422   
423   
424   
425   
426   
427   
428   
429   
430   
431   
432   
433   
434   
435   
436   
437   
438   
439   
440   
441   
442   
443   
444   
445   
446   
447   
448   
449
The TITA model integrates three different parts as shown in Figure 1: a two-stage keyword-document interaction, a neural topic model and a joint training mechanism. In the training process, the neural topic model and the joint model are trained alternatively to a convergent status. We first train the neural topic model for  $\lambda$  epochs to get a topic distribution over vocabulary, i.e.,  $\beta$ . Then in the joint training process, the model takes the output of the two-stage keyword-document interaction and the output of the neural topic model to conduct training the model parameters for classification.
In this section, we conduct experiments on our keyword-document matching dataset from our recommendation system and the results demonstrate the superiority of the TITA model compared to the baselines.
We apply accuracy as the evaluation metric. In this paper, we care mostly about rigidly distinguishing the three keyword-document matching levels. We believe that documents of different matching levels have different usages. For instance, in our online recommendation system, the goal of our model is to recall the topic-aware relevance documents and there is no need to rank documents of each keyword.
Our keyword-document matching dataset is in Chinese, derived from our recommendation system. The domains mainly lie in food (e.g., beef and western food), sports (e.g., football and jogging), entertainment (e.g., photography and comedy) and so on. For all the 8901 keywords, we get 10 documents for each keyword by users' behavior in our recommendation system, e.g., click-through. As for how to choose 10 documents for each keyword in the baseline online recommendation system, for a certain keyword, hundreds of documents are recalled for different users, in which the topic-aware relevance documents tend to have high click-through rate while irrelevance ones tend to have low click-through rate. For each keyword, we select 6 documents which have high click-through rate as well as 4 documents with low click-through rate. According to our analysis, this setting tends to generate similar ratios of three-level relevance documents for all the keywords. As a result, each keyword has 10 corresponding documents. Each
keyword-document pair is manually annotated at different relevance levels. As shown in Table 1, relevance level-2 means the document and the keyword have the same topic, while relevance level-0 means the keyword and the document are irrelevant. Relevance level-1 is an intermediate relevance level, which means only a small portion of the document describes some useful information of the keyword. To make the ratios of level-2, level-1, level-0 cases nearly the same, we randomly delete some documents. As a result, we have 8,901 keywords and 66,019 corresponding documents. Finally, the dataset is randomly split into  $50\%$  for training,  $25\%$  for validation and  $25\%$  for testing.
In the experiment, we set the cutoff length of the document sequence as 512 characters and the cutoff length of the keyword as 16 characters in Chinese. The size of the character vocabulary  $V_{c}$  is 21128. The size of the word vocabulary for neural topic model  $V_{w}$  is 5000, which contains top frequent words after deleting stop words. We use pre-trained embeddings by BERT to initialize the character embeddings. We directly use BERT base model released by Google with the hidden size of 768. In the neural topic model, we set the number of topics  $\# K = 50$ . We use all the documents in the training set to train the neural topic model for 50 epochs. The topic embedding size  $d$  is set to 384 and we set the word embedding to the same size. The padding is masked to avoid affecting the gradient. We use the optimization algorithm Adam (Kingma and Ba, 2014) with learning rate 5e-5 and batch size as 32. As for the parameters of Adam,  $\beta_{1}$  and  $\beta_{2}$  are set to 0.9 and 0.999 respectively.
As described in the Introduction Section, the keyword-document matching models can be divided into two categories: representation-based and interaction-based matching model. As shown in Table 2, many strong baselines are included in the performance comparison.
4.4 Main Results and Ablation Analysis
Table 2 shows that the TITA model outperforms all the models evaluated by accuracy in this keyword-document matching task. From this table, we have the other observations: (1) The TITA model is more competent in this task. It outperforms ARC-II by  $7.06\%$  and outperforms BERT by  $5.38\%$ , which
450   
451   
452   
453   
454   
455   
456   
457   
458   
459   
460   
461   
462   
463   
464   
465   
466   
467   
468   
469   
470   
471   
472   
473   
474   
475   
476   
477   
478   
479   
480   
481   
482   
483   
484   
485   
486   
487   
488   
489   
490   
491   
492   
493   
494   
495   
496   
497   
498
499   
500   
501   
502   
503   
504   
505   
506   
507   
508   
509   
510   
511   
512   
513   
514   
515   
516   
517   
518   
519   
520   
521   
522   
523   
524   
525   
526   
527   
528   
529   
530   
531   
532   
533   
534   
535   
536   
537   
538   
539   
540   
541   
542   
543   
544   
545   
546   
547
Figure 2: The architecture of online deployment of the TITA model, which consists of two major components: an offline data processor module and an online data usage module.
Models	Acc(%)
Bi-LSTM	67.16
DSSM (Huang et al., 2013)	68.66
CLSM (Shen et al., 2014)	67.21
DSSM-LSTM (Palangi et al., 2016)	66.71
MatchPyramid (Pang et al., 2016)	66.89
ARC-II (Hu et al., 2014)	68.16
BIDAF (Seo et al., 2016)	67.55
RNET (Wang et al., 2017)	67.69
BERT (Devlin et al., 2018)	69.84
The TITA Model	75.22
Table 2: The main experimental results of baselines and the TITA model evaluated by accuracy.
Models	Acc(%)
Bi-LSTM	67.16
+ Neural Topic Model	69.18
+ First-stage Keyword-document Interac-tion	70.86
+ Second-stage Keyword-document Inter-acton	72.45
Replace Bi-LSTM with BERT	75.22
Table 3: Ablation test results of the TITA model evaluated by accuracy.
strongly proves that topic model and two-stage interaction can benefit this task. (2) Most interaction-based models behave better than representation-based ones. (3) Pre-trained word embeddings can also aid this task.
To further examine the effectiveness of the neural topic model and the two-stage keyword-document interaction, we make a detailed ablation analysis as shown in Table 3.
- Bi-LSTM: The TITA model is based on BiLSTM, which encodes a query and a document independently before matching.
+ Neural Topic Model: Bi-LSTM plus neural topic model outperforms the Bi-LSTM baseline by a large scale (i.e.,  $2.02\%$ ), which indicates that the keyword-document matching task can benefit from the latent topic representation of the document.
- + First-stage Keyword-document Interaction: After adding the first-stage keyword-document interaction, the model behaves better. It proves that concatenating the query and document to conduct interaction is effective.
- + Second-stage Keyword-document Interaction: We add the second-stage interaction to make further improvement. We infer that the cross attention is more capable in capturing interactions between a keyword and a document.
- Replace Bi-LSTM with BERT: We apply BERT to initialize the word representation, whose parameters are to be finetuned. We can observe that the model performs even better than the former one, which reveals that the pre-trained word representations are useful in the keyword-document matching task.
5 Online Deployment and Online Gains
Because the model is heavy and the total numbers of keywords are limited (8901 in total), we generate data in offline, as shown in Figure 2. In offline data processor, we first use BM25 to retrieve and rank billions of document candidates and keep the top-10000 candidates for TITA model to further conduct query-document relation prediction. After that we can get a ranked list of topic matching documents and partially relevance documents for all keywords, which will be stored in a KV database. In the online data usage, we recall documents of all
548   
549   
550   
551   
552   
553   
554   
555   
556   
557   
558   
559
560   
561   
562   
563   
564   
565   
566   
567   
568   
569   
570   
571   
572   
573   
574   
575   
576   
577   
578   
579   
580   
581   
582   
583   
584   
585   
586   
587   
588   
589   
590   
591   
592   
593   
594   
595
the keywords, which the user follows, for further re-ranking in our recommendation system.
As for the online gains, we attached more than one million topic-matching documents for the 8901 keywords. These documents are all distributed in our recommendation system with the number of views about  $1.9e^{6}/day$ . We improve the click-through rate by  $4.35\%$  (from  $6.52\%$  to  $10.87\%$ ), which is a great improvement.
We define a new keyword-document matching task with three relevance levels from a real recommendation system, to address the problem that different scenarios require documents of different relevance levels. Further, we propose a TITA model to distinguish different relevance levels, which can capture latent topics of a document and hold complex keyword-document interactions at the same time. Extensive experiments reveal the superiority of our model compared to other strong baselines. Ablation test shows that the model can improve the keyword-document matching in the same way as we think. Moreover, our model shows excellent performance in our recommendation system, in which it improves the click-through rate by  $4.35\%$ .
David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent dirichlet allocation. volume 3, pages 993-1022.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Jiafeng Guo, Yixing Fan, Qingyao Ai, and W. Bruce Croft. 2016. A deep relevance matching model for ad-hoc retrieval. In Proceedings of the 25th ACM International Conference on Information and Knowledge Management, CIKM 2016, Indianapolis, IN, USA, October 24-28, 2016, pages 55-64.
David Hall, Daniel Jurafsky, and Christopher D Manning. 2008. Studying the history of ideas using topic models. In Proceedings of the conference on empirical methods in natural language processing, pages 363-371. Association for Computational Linguistics.
Baotian Hu, Zhengdong Lu, Hang Li, and Qingcai Chen. 2014. Convolutional neural network architectures for matching natural language sentences. In Advances in Neural Information Processing Systems
27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 2042-2050.
Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry P. Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. In 22nd ACM International Conference on Information and Knowledge Management, CIKM'13, San Francisco, CA, USA, October 27 - November 1, 2013, pages 2333-2338.
Jyun-Yu Jiang, Mingyang Zhang, Cheng Li, Michael Bendersky, Nadav Golbandi, and Marc Najork. 2019. Semantic text matching for long-form documents. In The World Wide Web Conference, WWW 2019, San Francisco, CA, USA, May 13-17, 2019, pages 795-806.
Shuhui Jiang, Xueming Qian, Jialie Shen, Yun Fu, and Tao Mei. 2015. Author topic model-based collaborative filtering for personalized poi recommendations. IEEE transactions on multimedia, 17(6):907-918.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Diederik P Kingma and Max Welling. 2013. Auto-encoding variational bayes.
Yishu Miao, Edward Grefenstette, and Phil Blunsom. 2017. Discovering discrete latent topics with neural variational inference. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pages 2410-2419.
Hamid Palangi, Li Deng, Yelong Shen, Jianfeng Gao, Xiaodong He, Jianshu Chen, Xinying Song, and Rabab K. Ward. 2016. Deep sentence embedding using long short-term memory networks: Analysis and application to information retrieval. volume 24, pages 694-707.
Liang Pang, Yanyan Lan, Jiafeng Guo, Jun Xu, Shengxian Wan, and Xueqi Cheng. 2016. Text matching as image recognition. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA., pages 2793-2799.
Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. arXiv preprint arXiv:1611.01603.
Yelong Shen, Xiaodong He, Jianfeng Gao, Li Deng, and Grégoire Mesnil. 2014. A latent semantic model with convolutional-pooling structure for information retrieval. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, CIKM 2014, Shanghai, China, November 3-7, 2014, pages 101-110.
596   
597   
598   
599   
600   
601   
602   
603   
604   
605   
606   
607   
608   
609   
610   
611   
612   
613   
614   
615   
616   
617   
618   
619   
620   
621   
622   
623   
624   
625   
626   
627   
628   
629   
630   
631   
632   
633   
634   
635   
636   
637   
638   
639   
640   
641   
642   
643
645   
646   
647   
648   
649   
650   
651   
652   
653   
654   
655   
656   
657   
658   
659   
660   
661   
662   
663   
664   
665   
666   
667   
668   
669   
670   
671   
672   
673   
674   
675   
676   
677   
678   
679   
680   
681   
682   
683   
684   
685   
686   
687   
688   
689   
690   
691   
692   
693   
694   
695   
696   
697   
698
Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. 2015. End-to-end memory networks. In Advances in neural information processing systems, pages 2440-2448.
Wenhui Wang, Nan Yang, Furu Wei, Baobao Chang, and Ming Zhou. 2017. Gated self-matching networks for reading comprehension and question answering. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 189-198.
Liu Yang, Qingyao Ai, Jiafeng Guo, and W. Bruce Croft. 2016. anmm: Ranking short answer texts with attention-based neural matching model. In Proceedings of the 25th ACM International Conference on Information and Knowledge Management, CIKM 2016, Indianapolis, IN, USA, October 24-28, 2016, pages 287-296.
Bing Zhao and Eric P Xing. 2006. Bitam: Bilingual topic admixture models for word alignment. In Proceedings of the COLING/ACL on Main conference poster sessions, pages 969-976. Association for Computational Linguistics.
699   
700   
701   
702   
703   
704   
705   
706   
707   
708   
709   
710   
711   
712   
713   
714   
715   
716   
717   
718   
719   
720