SlowGuess/ACL_2020_OCR huggingface dataset:
We also include previous conversation recommendation models without learning user interest dynamics: 3) CRJTD (Zeng et al., 2018): a CF-based method that jointly models topics and discourse with LDA-style Bayesian models. 4) CRIM (Zeng et al., 2019b): a neural CF framework with GCN-based interaction modeling, which presents state-of-the-art conversation recommendation results in previous work.
In addition, we compare with the following recent models for product recommendation. 5) RRN (Wu et al., 2017): exploiting RNN model to capture user interest dynamics only with user interaction history (without modeling turn content). 6) LC-RNN (latent cross-RNN) (Beutel et al., 2018): RNN-based user interest dynamic modeling with turn-level representations, with participant interactions in the conversation structure ignored.
We first report the main comparison results in Section 5.1, and then discuss the effects of sparsity and cold start in Section 5.2. Lastly, in Section 5.3, we probe into our model outputs to provide more insights into user interest dynamics.
5.1 Main Comparison Results
Table 2 shows the comparison results on all three datasets. Our model achieves the highest scores, outperforming all comparison models by a large margin. It suggests that dynamic user interests learned from both content and interactions provide clearly useful signals on which conversations a user is likely to engage in. Below describes more detailed observations.
3We also experiment with nDCG@10, and same trend holds.
3336
Models	Tech	Learn	Fun
MAP	P@1	nDCG	MAP	P@1	nDCG	MAP	P@1	nDCG
Simple Baselines
POPULARITY	0.055	0.012	0.031	0.057	0.012	0.033	0.058	0.011	0.033
TOPICRANK (Chen et al., 2011)	0.087	0.037	0.071	0.071	0.031	0.050	0.065	0.024	0.042
Unchanged Interests
CRJTD (Zeng et al., 2018)	0.193	0.173	0.184	0.158	0.135	0.150	0.113	0.085	0.101
CRIM (SOTA) (Zeng et al., 2019b)	0.222	0.180	0.187	0.204	0.151	0.194	0.162	0.114	0.150
Dynamic Interests
RRN (Wu et al., 2017)	0.190	0.210	0.199	0.221	0.270	0.238	0.190	0.227	0.201
LC-RNN (Beutel et al., 2018)	0.212	0.222	0.234	0.222	0.294	0.240	0.198	0.255	0.211
OURS	0.375	0.391	0.369	0.347	0.368	0.344	0.283	0.294	0.274
Table 2: Results of our main experiments (averaged over users). "nDCG" stands for "nDCG@5". CRIM is from our prior work which obtained previous state-of-the-art. The best result for each column is in **boldface**. Our model significantly outperforms all comparisons ( $p < 0.01$ , paired t-test).
The two baselines yield much worse results than others. This shows the challenging nature of conversation recommendation, and the limitation of simply using popularity or topic similarity. TOPICRANK performs slightly better than POPULARITY, indicating that individuals are more inclined to engage in conversations they like (reflected by topic relevance), rather than popular discussions with many participants.
Our model outperforms CRJTD and CRIM (state-of-the-art model), which both assume fixed user interests, showing the usefulness of exploring user's evolving interests over time. We also find that CRIM produces better results than CRJTD, likely because the former additionally captures user interactions among each other.
For recommendation models that consider user interest dynamics, all models perform better than CRIM and CRJTD, which are both based on the CF architecture. This reveals CF's limitation in dealing with cold start, which is a common phenomenon when recommending a large number of future conversations (see Table 1). Nevertheless, we see that our model performs much better than RRN and LC-RNN, indicating that both content and interaction features contribute to capturing user interests and how they change over time.
5.2 History Sparsity and Cold Start
Similar to previous work in product recommendation (Sarwar et al., 2000), conversation recommendation models are also susceptible to the problems of history sparsity and cold start. We compare with
LC-RNN (the best comparison model in Table 2) and CRIM (state-of-the-art model in conversation recommendation), and show in Figure 5 the MAP scores on Tech dataset with varying degrees of sparsity. Our model is shown to be consistently better in face of sparsity, including varying numbers of messages in user history, as well as varying numbers of available turns in conversation contexts. More detailed discussions are presented below.
Varying Messages in User History. Refer to in Figure 5(a), all models produce non-monotonic performance curves, peaking at certain points (e.g. 25 historical messages for our model). This reveals the issue of user history sparsity, and difficulty in coping with excessive historical information. More importantly, it is observed that our model already outperformed LC-RNN and CRIM when the number of history message is 0. This may be attributed to our better modeling on conversation interaction structure.
Varying Turns in Conversation Context. For conversations, Figure 5(b) shows the MAP scores with varying turn numbers available in contexts. All three models produce upward-trending curves, which is expected since more features can be learned from richer contexts, thus leading to better prediction. Our model and CRIM perform worse than LC-RNN when available turn number is small (less than 4). This is because graph-structured networks need minimum amount of interaction infor
Similar trends are observed on all datasets and hence only the results on Tech are displayed.
(a) User History
(b) Conversation Context
mation for effective modeling of the conversation structures.
Conversation Cold Start. To understand how models perform exactly in conversation cold start, we separate the test set into future conversations (newly created in testing and unseen in training data) and existing ones (with context partially in the training data). We then compute the results averaging over conversations. The resultant MAP scores are reported in Table 3. Our model outperforms the other two models by a large margin in recommending future conversations, thanks to the more accurate user interests that are learned from dynamic patterns of content and interactions. CRIM performs much better for existing conversations, by making use of rich user interaction history based on CF architecture. Our model abandons CF framework but still produce competitive performance, as we compute more accurate user-aware representations.
5.3 More Analyses on Our Model
The aforementioned results have shown the efficacy and advantage of our model. In this section, we provide more insights into different factors behind
Models	Future Convs	Existing Convs
Tech	Learn	Fun	Tech	Learn	Fun
CRIM	0.208	0.165	0.142	0.684	0.731	0.455
LC-RNN	0.214	0.220	0.197	0.129	0.587	0.318
OURS	0.384	0.356	0.305	0.590	0.749	0.458
Table 3: MAP scores to predict future and existing conversations (averaged over conversations). Our model performs the best in conversation cold start.
the model, in order to obtain a better understanding of its performance.
Training with More History. We have shown the usefulness of capturing user interest dynamics with historical messages. A natural question is whether the model needs more history to perform better. Figure 6 shows our MAP scores trained on history data in the last  $x$  months ( $x = 1,2,3,4$ ) and the three datasets exhibit diverse characteristics in user interest dynamics. Only Tech exhibits an increasing trend. This is probably because earlier history enables learning of long-term dynamics and technology change usually happens in a time span that is longer than 1-2 months. On the contrary, topics on Fun and Learn may change more rapidly, making the earlier history more noisy and less helpful for modeling users' current interests.
Figure 5: MAP scores on Tech dataset with varying degrees of sparsity in user chatting history (upper) or conversation context (lower). Our model performs consistently better.
Figure 6: MAP scores of our model with training data in the last  $x$  months.
Ablation Study. We then examine the contributions of different components in our model, and display the MAP scores of various ablations in Table 4. We observe that user factor embedding and user-aware attention contribute most to model outputs because they are critical in modeling user interests. Removing Bi-GRU or GCN also has a significant impact on performance, indicating the usefulness of learning user interactions from turn chronology and replying relations.
To further understand the effects of Bi-GRU and
Models	Tech	Learn	Fun
w/o user factor embedding	0.174	0.159	0.122
w/o user-aware attention	0.188	0.183	0.149
w/o Bi-GRU	0.299	0.253	0.206
w/o GCN	0.276	0.307	0.221
Our full model	0.375	0.347	0.283
Table 4: MAP scores with different parts ablated. The best MAP results are highlighted in bold.
GCN in user interaction modeling, we compare the MAP scores of our full model and its variants without Bi-GRU or GCN in recommending conversations with 1, 2, or more root-to-leaf paths (as shown in Figure 7). GCN and Bi-GRU clearly demonstrate different capabilities. The former is good at encoding more complex structures (i.e. those with more paths), and the latter excels at sequential conversations. By leveraging the advantages of both, our full model performs the best for conversations of varying structures.
Figure 7: Results of our full model and its variants without Bi-GRU or GCN for recommending conversations in different structures. X-axis: number of root-to-leaf paths. Y-axis: MAP scores.
Case Study. Lastly, we use the example in Figure 1 to analyze what the model has learned for recommendation. Recall that user  $U$ 's interests shifted from Internet security, signaled earlier in  $C_1$  and  $C_2$ , to operation system, when later chatting in  $C_3$  and  $C_4$ . We examine the predicted likelihoods of  $U$  engaging in two future conversations: Conversation  $A$  and  $B$ . Figure 8 shows their contexts— $A$  focuses on Internet security and  $B$  on file system, and  $U$  later engaged in  $B$  but not  $A$  due to the interest shift. In Table 5, we list our model's outputs when fed with earlier history only ( $C_1$  and  $C_2$ ), later only ( $C_3$  and  $C_4$ ), and full history, respectively. Not surprisingly, much higher scores are given to  $A$  when only the earlier history is given, as it fits well with  $U$ 's previous preference. Similarly,
we correctly predict  $U$  to engage in  $B$  with much higher confidence in the other two situations as file system ( $B$ 's focus) and operation system ( $U$ 's later interests) are highly related. Given the full history, our model produces more closed scores, showing its efficacy of learning user interest dynamics.
Conversation A
$[T_1]$ : Ahhh! This reminds me of when you could hack fax machines and routers by just whistling in the phone!
$[T_2]$ : Hm, that's pretty unrelated, though..
Conversation B
$[T_1]$ : ...just downloaded FileZilla (from SourceForge) last night, and it automatically installed MacKepper and...
$[T_2]$ : Dude, why? Filezilla has a website, you can download it straight from them...
Figure 8: Context turns in Conversation (Conv.)  $A$  and  $B$ . Blue italic words indicate  $A$ 's topic—Internet security and red italic words in  $B$  reflects its focus on file system.
U's History Given	Conv. A	Conv. B
Earlier history only (C1, C2)	0.733	0.267
Later history only (C3, C4)	0.297	0.703
Full history (C1, C2, C3, C4)	0.421	0.579
Table 5: Predicted likelihoods of  $U$  entering Conversations  $A$  and  $B$ .  $B$  is ranked higher than  $A$  due to shifted user interests.
This paper presents a dynamic conversation recommendation model learned from the change of content and user interactions over time. Experimental results on three new datasets from Reddit show that our model significantly outperforms all comparisons, including previous state of the arts. Further discussion demonstrates the robustness of our model against history sparsity and cold start. We also analyze our model's outputs to get more insights into user interest dynamics.
The research described in this paper is partially supported by HK RGC-GRF grant #14204118. Jing Li is partly funded by the Hong Kong Polytechnic University internal fund (1-BE2W). Lu Wang is supported by National Science Foundation through Grant IIS-1813341. We thank the three anonymous reviewers for the insightful suggestions on various aspects of this work.
Yoav Artzi, Patrick Pantel, and Michael Gamon. 2012. Predicting responses to microblog posts. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 602-606. Association for Computational Linguistics.
Lars Backstrom, Jon M. Kleinberg, Lillian Lee, and Cristian Danescu-Niculescu-Mizil. 2013. Characterizing and curating conversation threads: expansion, focus, volume, re-entry. In Sixth ACM International Conference on Web Search and Data Mining, WSDM 2013, Rome, Italy, February 4-8, 2013, pages 13-22.
Alex Beutel, Paul Covington, Sagar Jain, Can Xu, Jia Li, Vince Gatto, and Ed H. Chi. 2018. Latent cross: Making use of context in recurrent recommender systems. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, WSDM 2018, Marina Del Rey, CA, USA, February 5-9, 2018, pages 46-54.
David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent Dirichlet Allocation. Journal of machine Learning research, 3(Jan):993-1022.
Jilin Chen, Rowan Nairn, and Ed Huai-hsin Chi. 2011. Speak little and well: recommending conversations in online social streams. In Proceedings of the International Conference on Human Factors in Computing Systems, CHI 2011, Vancouver, BC, Canada, May 7-12, 2011, pages 217-226.
Kailong Chen, Tianqi Chen, Guoqing Zheng, Ou Jin, Enpeng Yao, and Yong Yu. 2012. Collaborative personalized tweet recommendation. In Proceedings of the 35th international ACM SIGIR Conference on Research and development in information retrieval, pages 661-670. ACM.
Hao Cheng, Hao Fang, and Mari Ostendorf. 2017a. A factored neural network model for characterizing online discussions in vector space. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2296-2306.
Justin Cheng, Michael Bernstein, Cristian Danescu-Niculescu-Mizil, and Jure Leskovec. 2017b. Anyone can become a troll: Causes of trolling behavior in online discussions. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing, CSCW '17, pages 1217-1230. ACM.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gülcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar; A meeting of SIGDAT,
a Special Interest Group of the ACL, pages 1724- 1734.
Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017. Neural collaborative filtering. In Proceedings of the 26th International Conference on World Wide Web, WWW 2017, Perth, Australia, April 3-7, 2017, pages 173-182.
Yunhao Jiao, Cheng Li, Fei Wu, and Qiaozhu Mei. 2018. Find the conversation killers: A predictive study of thread-ending posts. In Proceedings of the 2018 World Wide Web Conference on World Wide Web, pages 1145-1154. International World Wide Web Conferences Steering Committee.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar; A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1746-1751.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Edward Loper and Steven Bird. 2002. Nltk: The natural language toolkit. In Proceedings of the ACL-02 Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics.
Annie Louis and Shay B. Cohen. 2015. Conversation trees: A grammar model for topic structure in for-ums. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1543-1553. Association for Computational Linguistics.
Diego Marcheggiani and Ivan Titov. 2017. Encoding sentences with graph convolutional networks for semantic role labeling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 1506-1515.
Yasuhide Miura, Ryuji Kano, Motoki Taniguchi, Tomoki Taniguchi, Shotaro Misawa, and Tomoko Ohkuma. 2018. Integrating tree structures and graph structures with neural networks to classify discussion discourse acts. In Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018, Santa Fe, New Mexico, USA, August 20-26, 2018, pages 3806-3818.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543.
Alan Ritter, Colin Cherry, and Bill Dolan. 2010. Unsupervised modeling of Twitter conversations. In Human Language Technologies: Conference of the North American Chapter of the Association of Computational Linguistics, pages 172-180.
Badrul Munir Sarwar, George Karypis, Joseph A. Konstan, and John Riedl. 2000. Analysis of recommendation algorithms for e-commerce. In Proceedings of the 2nd ACM Conference on Electronic Commerce (EC-00), Minneapolis, MN, USA, October 17-20, 2000, pages 158-167.
Chao-Yuan Wu, Amr Ahmed, Alex Beutel, Alexander J. Smola, and How Jing. 2017. Recurrent recommender networks. In Proceedings of the Tenth ACM International Conference on Web Search and Data Mining, WSDM 2017, Cambridge, United Kingdom, February 6-10, 2017, pages 495-503.
Rui Yan, Mirella Lapata, and Xiaoming Li. 2012. Tweet recommendation with graph co-ranking. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers Volume 1, pages 516-525. Association for Computational Linguistics.
Victoria Zayats and Mari Ostendorf. 2018. Conversation modeling on reddit using a graph-structured LSTM. TACL, 6:121-132.
Xingshan Zeng, Jing Li, Lu Wang, Nicholas Beauchamp, Sarah Shugars, and Kam-Fai Wong. 2018. Microblog conversation recommendation via joint modeling of topics and discourse. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, Volume 1 (Long Papers), pages 375-385.
Xingshan Zeng, Jing Li, Lu Wang, and Kam-Fai Wong. 2019a. Joint effects of context and user history for predicting online conversation re-entries. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28-August 2, 2019, Volume 1: Long Papers, pages 2809-2818.
Xingshan Zeng, Jing Li, Lu Wang, and Kam-Fai Wong. 2019b. Neural conversation recommendation with online interaction modeling. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4625-4635, Hong Kong, China.