SlowGuess/ACL_2020_OCR huggingface dataset:
We implement mainstream text classification models including TextCNN (Kim, 2014), BiLSTM-Att (bi-directional LSTM with attention) BiLSTM (Graves and Schmidhuber, 2005; Bahdanau et al., 2015), BiGRU-Att (bi-directional GRU with attention) (Cho et al., 2014), BERT (Devlin et al., 2019) (only fine-tune the last layer for simplicity). For a fair comparison, we concatenate the post and its attached comments together as the input, instead of feeding the post only.
Structure-based Methods
Considering that structure-based features of the post and its comment tree are rare and non-systematic in previous works, we integrate the plausible features in (Coletto et al., 2017) and (Hessel and Lee, 2019). As the latter paper does, we feed them into a series of classifiers and choose a best model for classification. We name the method SFC. For a post-comment graph, the feature set contains the average depth (average length of root-to-leaf paths), the maximum relative degree (the largest node degree divided by the degree of the root), C-RATE features (the logged reply time between the post and comments, or over pairs of comments),
$^{5}$ https://pypi.org/project/  
pytorch-pretrained-bert/  
 $^{6}$ https://github.com/Embedding/  
Chinese-Word-Vectors  
 $^{7}$ https://nlp.stanford.edu/projects/glove/
<sup>3</sup>http://mcg.ict.ac.cn/  
controversy-detection-dataset.html  
<sup>4</sup>https://github.com/google-research/bert
Method	Weibo Dataset	Reddit Dataset
Avg. P	Avg. R	Avg. F1	Acc.	Avg. P	Avg. R	Avg. F1	Acc.
Content-based	TextCNN	72.80	68.49	69.08	72.83	56.58	56.33	55.92	56.33
BiLSTM-Att	69.97	70.31	70.10	71.28	62.74	60.66	58.98	60.66
BiGRU-Att	71.35	72.21	71.50	72.21	59.95	59.86	59.77	59.86
BERT	72.17	72.72	72.37	73.35	60.80	60.80	60.80	60.80
Structure-based	SFC	68.15	66.27	66.72	70.10	59.47	59.47	59.47	59.47
Fusion	(Hessel and Lee, 2019)	72.52	70.82	71.34	73.82	63.03	63.03	63.03	63.03
TPC-GCN	74.65	75.33	74.88	75.72	67.00	66.97	66.95	66.97
Table 2: Performance(%) comparison of the intra-topic experiments.
Method	Weibo Dataset	Reddit Dataset
Avg. P	Avg. R	Avg. F1	Acc.	Avg. P	Avg. R	Avg. F1	Acc.
Content-based	TextCNN	71.55	72.63	69.63	69.76	54.20	54.18	54.12	54.18
BiLSTM-Att	67.09	68.09	67.10	68.00	60.96	59.76	58.63	59.76
BiGRU-Att	68.04	67.08	67.35	70.18	58.49	58.17	57.76	58.17
BERT	68.77	68.16	68.42	72.22	60.41	59.96	59.53	59.96
Structure-based	SFC	63.06	63.69	63.04	64.03	58.87	58.86	58.86	58.86
Fusion	(Hessel and Lee, 2019)	69.25	67.15	67.63	70.84	60.77	60.76	60.74	60.76
TPC-GCN	73.84	72.00	71.53	72.11	63.39	63.24	63.14	63.24
DTPC-GCN	75.57	75.31	75.27	75.35	68.76	67.63	67.14	67.63
Table 3: Performance(%) comparison of the inter-topic experiments.
and C-TREE features (statistics in a comment tree, such as maximum depth/total comment ratio).
Fusion Method
The compared fusion method from (Hessel and Lee, 2019) aims to identify the controversial posts with semantic and structure information. They extract text features of topics, posts, and comments by BERT and structural feature including the C-RATE and C-TREE features mentioned above. In addition, publish time features are also exploited.
4.4 Performance Comparison
To answer EQ1, we compare the performance of proposed (D)TPC-GCN with mentioned baselines on the two datasets. The evaluation metrics include the macro average precision (Avg. P), macro average recall (Avg. R), macro average F1 score (Avg. F1), and accuracy (Acc.). Table 2 and 3 show the performance of all compared methods for intra-topic detection and inter-topic detection respectively.
In the intra-topic experiments, we can see that 1) TPC-GCN outperforms all compared methods on the two datasets. This indicates that our model can effectively detect controversy with a significant generalizability on different datasets. 2) The structure-based model, SFC, reports the low scores on the two datasets, indicating that the statistical structural information is insufficient to timely identify the controversy. 3) The fusion models outperform or are comparable to the other baselines, which proves that information fusion of content and
structure is necessary to improve the performance.
In the inter-topic experiments, we can see that 1) DTPC-GCN outperforms all baselines by  $6.4\%$  of F1 score at least, which validates that DTPC-GCN can detect controversy on unseen or dissimilar topics. 2) DTPC-GCN outperforms TPC-GCN by  $3.74\%$  on Weibo and  $4.00\%$  on Reddit. This indicates that feature disentanglement and dynamic fusion can significantly improve the performance of inter-topic controversy detection.
To answer EQ2 and part of EQ3, we also evaluate several internal models, i.e., the simplified variations of (D)TPC-GCN by removing some components or masking some representations. By the ablation study, we aim to investigate the impact of content and structural information in TPC-GCN and topic-related and topic-unrelated information in DTPC-GCN.
Ablation Study of TPC-GCN
We delete certain type of nodes (and the edges connect to them) to investigate their overall impact and mask the content by randomizing the initial representation to investigate the impact of content. Specifically, we investigate on the following simplified models of TPC-GCN:
PC-GCN / TP-GCN: discard the topic / comment nodes.
(RT)PC-GCN / T(RP)C-GCN / TP(RC)-GCN: randomly initialize the representation of topic / post / comment nodes.
Method	Weibo Dataset	Reddit Dataset
Avg. P	Avg. R	Avg. F1	Acc.	Avg. P	Avg. R	Avg. F1	Acc.
TPC-GCN	74.65	75.33	74.88	75.72	67.00	66.97	66.95	66.97
PC-GCN	73.49	74.16	73.72	74.59	66.48	65.60	65.14	65.60
TP-GCN	58.72	59.16	58.20	58.68	52.97	52.83	52.28	52.83
(RT)PC-GCN	71.78	71.07	71.35	73.14	65.86	65.80	65.77	65.80
T(RP)C-GCN	72.30	72.65	72.45	73.55	65.25	64.73	64.43	64.73
TP(RC)-GCN	59.66	59.80	59.71	61.36	62.98	62.80	62.67	62.80
Table 4: Ablation study of TPC-GCN in the intra-topic experiments  $(\%)$
Method	Weibo Dataset	Reddit Dataset
Avg. P	Avg. R	Avg. F1	Acc.	Avg. P	Avg. R	Avg. F1	Acc.
DTPC-GCN	75.57	75.31	75.27	75.35	68.76	67.63	67.14	67.63
U branch only	74.06	74.06	74.05	74.05	63.95	63.94	63.94	63.94
R branch only	74.16	73.33	73.15	73.41	63.41	63.15	62.97	63.15
Table 5: Ablation study of DTPC-GCN in the inter-topic experiments  $(\%)$
From Table 4, we have the following observations: 1) TPC-GCN outperforms all simplified models, indicating that the necessity of structure and content from all types of nodes. 2) PC-GCN uses no extra information (the information of other posts in the same topic), the performance is still better than the baselines (Table 2 and 4), showing the effectiveness of our methods. 3) The models deleting comment information, i.e., TP-GCN and TP(RC)-GCN, experience a dramatic drop in performance, which shows the comment information is of the most importance. 4) The effect of structural information varies in the different situations. Without the contents, the comment structure can individually work (TP(RC)-GCN > TP-GCN), while for topics, the structure has to collaborate with the contents ((RT)PC-GCN < PC-GCN on the Weibo dataset).
Ablation Study of DTPC-GCN
We focus on the roles of the  $U$  (topic-unrelated) branch and  $R$  (topic-related) branch:
U branch only: Only  $U$  branch is trained to capture topic-unrelated features.
R branch only: Only  $R$  branch is trained to capture topic-related features.
Table 5 shows that both of the two branches can identify controversial posts well, but their performances are worse than the fusion model. Specifically, the  $U$  branch performs slightly better than  $R$ , indicating the topic-unrelated features are more suitable for inter-topic detection. We infer that the two branches can learn good but different representation under the guide of the auxiliary task.
Topic: Cancel the Driving License
Target Post 1
Cancelling the physical driving license can
bring much benefits: No punishment because of forgetting to carry the license; reduce the administrative costs; put an end to the use of fake licenses...
Comments Attached to 1
(Support) Yes! Just use the citizen's ID card for replacement.  
(Support) Good proposal! Support!
(Refute) I don't support it.
(Refute) Don't think the cost can be reduced. The costs of new electronic devices and larger data system are not small.
Branch Weights
U:0.874
$R:0.126$
Topic: Suggest Death Penalty for Woman- & Child-traffickers
Target Post 2
Human traffickers are hateful. People's
Congress Baoyan Zhang thinks that woman- and childtrafficking cases should be sentenced to death and the present sentence of five to 10 years in prison is not heavy enough.
Comments Attached to 2
(Support) Directly sentence to death. Execute immediately! (Support) Those harboring traffickers also need death penalty! (Support) Support! All the child traffickers should be sentenced to death penalty!
(Refute) Drug smugglers are sentenced to death, but so many people still do. If we use death penalty to traffickers, they may task crazier actions. Should think more carefully.
U:0.217
$R:0.783$
Figure 3: Examples of controversial posts that rely more on one of the two branches. The attention weights of the two posts are on the horizontal bars (left: the  $U$  branch, right: the  $R$  branch). Post 1 relies more on  $U$ $(0.874 > 0.126)$  while Post 2 more on  $R$ $(0.217 < 0.783)$ .
We conduct a case study to further answer EQ3 from the perspective of samples. We compare the attention weight of the  $U$  and  $R$  branch in DTPC-GCN and exhibit some examples where the final decisions lean on one of the two branches.
Figure 3 shows two examples in the testing set of the Weibo dataset. The DTPC-GCN rely more on the topic-unrelated features from Branch  $U$  when classifying Post 1  $(0.874 > 0.126)$ , while more on the topic-related features from Branch  $R$  when classifying Post 2  $(0.217 < 0.783)$ . The topic of Post 1, Cancel the Driving License, is weakly relevant to topics in training set, and the comments mostly use topic-unspecific words such as simple support and good proposal. Thus, the topic-unrelated features are more beneficial for judging. In contrast, Post 2 discusses the death penalty for women and children traffickers, relevant to one of the topics in the training set, Improve Sentencing Standards for Sexually Assault on Children. Further, both of the two topics are full of comments on death penalty. Exploiting more of the topic-related features is reasonable for the final decision.
4.7 Error Analysis
By conducting the error analysis on 186 misclassified samples in the Weibo dataset, we find three main types of samples that lead to the misclassification: 1)  $22.6\%$  of the wrong samples are with too much noise in the comments, including unrelated and neutral comments. 2)  $16.1\%$  are with a very deep tree structure. This kind of structure is helpful for controversy detection (Hessel and Lee, 2019), but the ability of GCN to obtain information from this kind of structure is limited. 3)  $10.2\%$  are with obscure and complex statements. These wrong cases indicate that better handling the noisy data, learning more deep structural features, and mining the semantic more deeply have the potential to improve the performance.
In this paper, we propose a novel method TPC-GCN to integrate the information from the graph structure and content of topics, posts, and comments for post-level controversy detection on social media. Unlike the existing works, we exploit the information from related posts in the same topic and the reply structure for more effective detection. To improve the performance of our model for inter-topic detection, we propose an extension of TPC-GCN named DTPC-GCN, to disentangle the topic-related and topic-unrelated features and then dynamically fuse them. Extensive experiments conducted on two datasets demonstrate that our proposed models outperform the compared methods
and prove that our models can integrate both semantic and structural information with significant generalizability.
The authors thank Peng Qi, Mingyan Lu, Guang Yang, and Jiachen Wang for helpful discussion. This work is supported by the National Nature Science Foundation of China (U1703261).
Aseel Addawood, Rezvaneh Rezapour, Omid Abdar, and Jana Diesner. 2017. Telling apart tweets associated with controversial versus non-controversial topics. In Proceedings of the Second Workshop on NLP and Computational Social Science, pages 32-41, Vancouver, Canada. Association for Computational Linguistics.
Rawia Awadallah, Maya Ramanath, and Gerhard Weikum. 2012. Harmony and dissonance: organizing the people's voices on political controversies. In Proceedings of the fifth ACM International Conference on Web Search and Data Mining, pages 523-532. ACM.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the third International Conference on Learning Representations.
Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bah-danau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder-decoder approaches. In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pages 103-111. Association for Computational Linguistics.
Yoonjung Choi, Yuchul Jung, and Sung-Hyon Myaeng. 2010. Identifying controversial issues and their subtopics in news articles. In Pacific-Asia Workshop on Intelligence and Security Informatics, pages 140-153. Springer.
Mauro Coletto, Kiran Garimella, Aristides Gionis, and Claudio Lucchese. 2017. A motif-based approach for identifying controversy. In Eleventh International AAAI Conference on Web and Social Media. AAAI.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Shiri Dori-Hacohen and James Allan. 2013. Detecting controversy on the web. In Proceedings of the 22nd ACM Intedrational Conference on Information and Knowledge Management, pages 1845-1848. ACM.
Shiri Dori-Hacohen and James Allan. 2015. Automated controversy detection on the web. In European Conference on Information Retrieval, pages 423-434. Springer.
Kiran Garimella, Gianmarco De Francisci Morales, Aristides Gionis, and Michael Mathioudakis. 2017. Reducing controversy by connecting opposing views. In Proceedings of the Tenth ACM International Conference on Web Search and Data Mining, pages 81-90. ACM.
Kiran Garimella, Gianmarco De Francisci Morales, Aristides Gionis, and Michael Mathioudakis. 2018. Quantifying controversy on social media. ACM Transactions on Social Computing, 1(1):3.
Alex Graves and Jürgen Schmidhuber. 2005. Frameswise phoneme classification with bidirectional LSTM and other neural network architectures. Neural networks, 18(5-6):602-610.
Jack Hessel and Lillian Lee. 2019. Something's brewing! early prediction of controversy-causing posts from discussion features. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1648-1659, Minneapolis, Minnesota. Association for Computational Linguistics.
Myungha Jang, John Foley, Shiri Dori-Hacohen, and James Allan. 2016. Probabilistic approaches to controversy detection. In Proceedings of the 25th ACM International Conference on Information and Knowledge Management, pages 2069-2072. ACM.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746-1751, Doha, Qatar. Association for Computational Linguistics.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Thomas N Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In Proceedings of the fifth International Conference on Learning Representations.
Aniket Kittur, Bongwon Suh, Bryan A Pendleton, and Ed H Chi. 2007. He says, she says: conflict and coordination in wikipedia. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 453-462. ACM.
Chang Li and Dan Goldwasser. 2019. Encoding social information with graph convolutional networks for Political perspective detection in news media. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2594-2604. Association for Computational Linguistics.
Jasper Linmans, Bob van de Velde, and Evangelos Kanoulas. 2018. Improved and robust controversy detection in general web pages using semantic approaches under large scale conditions. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, pages 1647-1650. ACM.
Diego Marcheggiani, Joost Bastings, and Ivan Titov. 2018. Exploiting semantics in neural machine translation with graph convolutional networks. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 486-492. Association for Computational Linguistics.
Azadeh Nematzadeh, Grace Bang, Xiaomo Liu, and Zhiqiang Ma. 2019. Empirical study on detecting controversy in social media. arXiv preprint arXiv:1909.01093.
Ana-Maria Popescu and Marco Pennacchiotti. 2010. Detecting controversial events from twitter. In Proceedings of the 19th ACM International Conference on Information and Knowledge Management, pages 1873-1876. ACM.
Hoda Sepehri Rad and Denilson Barbosa. 2012. Identifying controversial articles in wikipedia: A comparative study. In Proceedings of the eighth Annual International Symposium on Wikis and Open Collaboration, page 7. ACM.
Nils Rethmeier, Marc Hübner, and Leonhard Hennig. 2018. Learning comment controversy prediction in web discussions using incidentally supervised multitask CNNs. In Proceedings of the 9th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 316-321. Association for Computational Linguistics.
Mikalai Tsytsarau, Themis Palpanas, and Kerstin Denecke. 2010. Scalable discovery of contradictions on the web. In Proceedings of the 19th International Conference on World Wide Web, pages 1195-1196. ACM.
Ba-Quy Vuong, Ee-Peng Lim, Aixin Sun, Minh-Tam Le, Hady Wirawan Lauw, and Kuiyu Chang. 2008. On ranking controversies in wikipedia: models and evaluation. In Proceedings of the first International Conference on Web Search and Data Mining, pages 171-182. ACM.
Lu Wang and Claire Cardie. 2014. A piece of my mind: A sentiment analysis approach for online dispute detection. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics
(Volume 2: Short Papers), pages 693-699. Association for Computational Linguistics.
Yaqing Wang, Fenglong Ma, Zhiwei Jin, Ye Yuan, Guangxu Xun, Kishlay Jha, Lu Su, and Jing Gao. 2018. Eann: Event adversarial neural networks for multi-modal fake news detection. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 849-857. ACM.
Liang Yao, Chengsheng Mao, and Yuan Luo. 2019. Graph convolutional networks for text classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7370-7377. AAAI.
Taha Yasseri, Robert Sumi, András Rung, András Kornai, and János Kertész. 2012. Dynamics of conflicts in wikipedia. PloS one, 7(6):1-12.
Rex Ying, Ruining He, Kaifeng Chen, Pong Eksom-batchai, William L Hamilton, and Jure Leskovec. 2018. Graph convolutional neural networks for web-scale recommender systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 974-983. ACM.
#Topics
Table 6: 49 topics in the Weibo dataset. We modify some words and polish the sentences to improve the understandability when translating them into English.
A Topics in the Weibo dataset