Introduction & Prior work
Controversy detection is an increasingly important task. Controversial content can signal the need for moderation on social platforms, either to prevent conflict between users or limit the spread of misinformation. More generally, controversies provide insight into societies BIBREF0 . Often, the controversial content is outside the direct control of a platform on which it is shared, mentioned or discussed. This raises the requirement of generally applicable methods to gauge controversial content on the web for moderation purposes. Unfortunately, what is controversial changes, and may lie more in the way topics are discussed rather than what is discussed, making it difficult to detect controversies in a robust fashion. We take the task of controversy detection and evaluate robustness of different methodologies with respect to the varying nature of controversies.
Prior work on detecting controversies has taken three kinds of approaches: 1) lexical approaches, which seek to detect controversies through signal terms, either through bag-of-word classifiers, lexicons, or lexicon based language models BIBREF1 . 2) explicit modeling of controversy through platform-specific features, often in Wikipedia or social-media settings. Features such as mutual reverts BIBREF2 , user-provided flags BIBREF3 , interaction networks BIBREF4 or stance-distributions BIBREF5 have been used as platform-specific indicators of controversies. The downside of these approaches is the lack of generalizability due to their platform-specific nature. 3) matching models that combine lexical and explicit modelling approaches by looking at lexical similarities between a given text and a set of texts in a domain that provides explicit features BIBREF1 , BIBREF6 , BIBREF7 .
Controversy detection is a difficult task because 1) controversies are latent, like ideology, meaning they are often not directly mentioned as controversial in text. 2) Controversies occur across a vast range of topics with varying topic-specific vocabularies. 3) Controversies change over time, with some topics and actors becoming controversial whereas others stop to be so. Previous approaches lack the power to deal with such changes. Matching and explicit approaches are problematic when the source corpus (e.g. Wikipedia) lags after real-world changes BIBREF8 . Furthermore, lexical methods trained on common (e.g. fulltext) features are likely to memorize the controversial topics in the training set rather than the `language of controversy'. Alleviating dependence on platform specific features and reducing sensitivity to an exact lexical representation is paramount to robust controversy detection. To this end, we focus only on fulltext features and suggest to leverage the semantic representations of word embeddings to reduce the vocabulary-gap for unseen topics and exact lexical representations.
The majority of NLP-task related neural architectures rely on word embeddings, popularized by Mikolov et al BIBREF9 to represent texts. In essence these embeddings are latent-vector representations that aim to capture the underlying meaning of words. Distances between such latent-vectors are taken to express semantic relatedness, despite having different surface forms. By using embeddings, neural architectures are also able to leverage features learned on other texts (e.g. pretrained word embeddings) and create higher level representations of input (e.g. convolutional feature maps or hidden-states). These properties suggest that neural approaches are better able to generalize to unseen examples that poorly match the training set. We use two often applied network architectures adopting word embeddings, to classify controversy: Recurrent Neural Networks BIBREF10 and Convolutional Neural Networks BIBREF11 to answer the following research question. RQ: Can we increase robustness of controversy detection using neural methods?
Currently, there is no open large-size controversy detection dataset that lends itself to test cross-temporal and cross-topic stability. Thus we generate a Wikipedia crawl-based dataset that includes general web pages and is sufficiently large to train and test high capacity models such as neural networks.
Methods
A proven approach in modelling text with neural networks is to use Recurrent Neural Networks (RNNs) which enjoy weight sharing capabilities to model words irrespective of their sequence location. A specific type, the Hierarchical Attention Network (HAN) proposed by BIBREF10 makes use of attention to build document representations in a hierarchical manner. It uses bi-directional Gated Recurrent Units (GRUs) BIBREF12 to selectively update representations of both words and sentences. This allows the network to both capture the hierarchy from words to sentences to documents and to explicitly weigh all parts of the document relevant during inference.
Recently, Convolutional Neural Networks (CNNs) have enjoyed increasing success in text classification. One such network introduced by BIBREF11 looks at patterns in words within a window, such as "Scientology [...] brainwashes people". The occurrences of these patterns are then summarized to their 'strongest' observation (max-pooling) and used for classification. Since pooling is applied after each convolution, the output size of each convolutional operation itself is irrelevant. Therefore, filters of different sizes can be used, each capturing patterns in different sized word windows.
We explore the potential of RNNs and CNNs for controversy detection using both the HAN BIBREF10 and the CNN BIBREF11 model. Similar to BIBREF10 , each bi-directional GRU cell is set to a dimension of 50, resulting in a word/sentence representation of size 100 after concatenation. The word/sentence attention vectors similarly contain 100 dimensions, all randomly initialized. The word windows defined in the CNN model are set to sizes: 2, 3 and 4 with 128 feature maps each. Each model is trained using mini batches of size 64 and uses both dropout (0.5) and INLINEFORM0 regularization (1e-3) at the dense prediction layer. Both networks use pre-trained embeddings, trained on 100 billion words of a Google News corpus, which are further fine-tuned during training on the controversy dataset. The optimization algorithm used is Adam BIBREF13 (learning rate: 1e-3).
Datasets and evaluation
We use the Clueweb09 derived dataset of BIBREF0 for baseline comparison. For cross-temporal, cross-topic and cross-domain training & evaluation, we generate a new dataset based on Wikipedia crawl data. This dataset is gathered by using Wikipedia's `List of Contoversial articles' overview page of 2018 (time of writing) and 2009 (for comparison with baselines) . Using this as a `seed' set of controversial articles, we iteratively crawl the `See also', `References' and `External links' hyperlinks up to two hops from the seed list. The negative seed pages (i.e. non controversial) are gathered by using the random article endpoint. The snowball-sample approach includes general, non-Wikipedia, pages that are referred to from Wikipedia pages. The dataset thus extends beyond just the encyclopedia genre of texts. Labels are assumed to propagate: a page linked from a controversial issue is assumed to be controversial. The resulting dataset statistics are summarized in Table TABREF7 .
To be useful as a flagging mechanism for moderation, a controversy detection algorithm should satisfy both Precision and Recall criteria. F1 scores will therefore be used to evaluate this balance. The AUC values are used to measure classification performance in the unbalanced controversy datasets. The test-train split depends on the task investigated and is listed in the results section for the respective task. To test for significant results, all models were evaluated using a bootstrap approach: by drawing 1000 samples with replacements INLINEFORM0 documents from the test set equal to the test-set size. The resulting confidence intervals based on percentiles provide a measure of significance.
Baseline models
To compare the results of neural approaches to prior work we implemented the previous state-of-the-art controversy detection method: the language model from BIBREF7 . Together with an SVM baseline they act as controversy detection alternatives using only full text features, thus meeting the task-requirements of platform-independence. Note: the implementation of BIBREF7 additionally requires ranking methods to select a subset of the training data for each language model. A simplified version of this, excluding the ranking method but using the same dataset and lexicon to select documents as BIBREF7 , is implemented and included in the baselines comparison section (LM-DBPedia). We also included the same language model trained on the full text Wikipedia pages (LM-wiki). Similarly, for completeness sake, we also include both the state-of-the-art matching model, the TILE-Clique model from BIBREF1 and the sentiment analysis baseline (using the state-of-the-art Polyglot library for python) from BIBREF6 in the comparison with previous work.
Comparison of results with previous work
Table TABREF13 shows the relative performance of the neural models compared to previous controversy detection methods, evaluated on the Clueweb09 derived dataset of BIBREF6 and trained on the Wikipedia data from the same time frame. The TILE-Clique matching model outperforms all other models on Precision although this difference is not significant compared to the neural approaches. Similarly, the language model trained on the DBPedia dataset outperforms other models on Recall but shows no significant difference compared to the CNN model. Notably, the neural approaches show comparable results to the TILE-Clique model in terms of F1, demonstrating a balanced performance in terms of Precision and Recall. Furthermore, the CNN model shows a significant improvement compared to the other non neural baselines in terms of the AUC value (p < 0.05).
Robustness of the model across time
Controversy is expected to change over time. Some issues become controversial, others cease to be so. To investigate robustness of controversy detection models with respect to changes over time, we evaluate model performance in two variants: trained and tested on 2018, or trained on the 2009 Wikipedia data and tested on the 2018 Wikipedia data. Table 3 shows the results for each of the text-based detection models.
Within year, the hierarchical attention model (HAN) outperforms all other models on Recall, F1 and AUC, losing Precision to the CNN and SVM models. However, our main interest is the robustness when a model is trained on a different year (2009) than the test set (2018). These between year experiments show a superior score for the HAN model compared to the non-neural models on Recall, and show significant improvements on F1 (p < 0.05) and AUC (p < 0.05), losing only to the SVM model on Precision (non significantly). In terms of robustness, we can also take the percentage change between the within year and between year experiment into account (were smaller absolute changes are preferable), shown by the delta values. With regard to temporal sensitivity, the CNN shows the least change across all four metrics. In Figure 1, we show the pooled results for the lexical and neural models to illustrate the overall increase in robustness by neural approaches.
Interestingly, the SVM and HAN model show some unexpected improvement with regard to Precision when applied to unseen timeframes. For both models, this increase in Precision is offset by a greater loss in Recall, which seems to indicate both models `memorize` the controversial topics in a given timeframe instead of the controversial language. Overall, the neural approaches seem to compare favorably in terms of cross-temporal stability.
Robustness of the model across topics
To evaluate robustness towards unseen topics, 10-fold cross validation was used on the top ten largest topics present in the Wikipedia dataset in a leave-one-out fashion. The results are shown in table 4. In line with previous results, the language model scores best on Recall, beating all other models with a significant difference (p < 0.01). However in balancing Recall with Precision, the HAN model scores best, significantly outperforming both lexical models in F1 score (p < 0.05). Overall, when grouping together all neural and lexical results, the neural methods outperform the lexical models in Precision (p < 0.01), F1 (p < 0.05) and AUC (p < 0.01) with no significant difference found on the overall Recall scores. These results indicate that neural methods seem better able to generalize to unseen topics.
Robustness of the model across domains
Most work on controversy has looked into using existing knowledge bases as a source of controversy information BIBREF6 , BIBREF1 . In this paper, we focus on text-based classification methods that do not aim to explicitly link general web pages to their knowledge-base counterparts. Therefore, we are interested in the ability of neural models to generalize beyond their training context. In addition to testing across time and topics, we also investigate robustness to changes in domain. By training only on Wikipedia data, and evaluating only on general web-pages, we look at the ability of the four methods to deal with out-of-domain documents.
The hierarchical attention network shows significantly better results (p < 0.05) compared to all other models on F1. Both neural models also outperform both language models on AUC significantly (p < 0.05). Precision and Recall are more mixed, with the CNN and SVM outperforming the HAN on Precision and the language model -again- performing best in terms of Recall. Together, the neural methods seem to work best on three out of the four metrics.
Human agreement
Lastly, we examine model performance with respect to human annotation using the human annotated dataset of BIBREF6 . We assume that models that perform similarly to human annotators are preferable. In Table TABREF20 , we present three Spearman correlation metrics to express model congruence with human annotations. Mean annotation expresses the correlation of model error rates with the controversy values attributed to a web page by human annotators, with positive values expressing greater error rates on controversial, and negative expressing higher error rates on non-controversial pages. Here, the HAN shows most unbiased (closest to zero) performance.
Certainty is the distance of human annotations to the midpoint of the four-point controversy scale, i.e. a score between 0 and 2.5 that expresses how sure annotators are of document (non)controversy. Here, the HAN shows errors most strongly negatively correlated to the certainty of annotators. Finally, annotators disagree on the controversy of some documents, expressed as the standard deviation of their controversy annotations. Again, the HAN model seems preferable, as it's errors are most strongly correlated to annotator disagreement. Overall, the neural methods have less biased performance in relation to (non)controversial documents, correlate more strongly with the certainty of human annotators and are susceptible to errors in similar conditions as when annotators disagree.
Conclusion
Controversy detection is a hard task, as it forms a latent concept sensitive to vocabulary gaps between topics and vocabulary shifts over time. We analysed the performance of language model, SVM, CNN and HAN models on different tasks.
First, we have demonstrated that neural methods perform as state-of-the-art tools in controversy detection on the ClueWeb09 BIBREF0 based testset, even beating matching models. Second, we investigated temporal stability, and demonstrated neural -and especially CNN- robustness in terms of Recall, F1 and AUC performance and stability with train and test sets that are 9 years apart. Thirdly, we show that CNN and HAN models outperform the SVM and LM baselines on Precision, F1 and AUC when tested on held-out-topics. Fourthly, we show that neural methods are better able to generalize from Wikipedia pages to unseen general web pages in terms of Precision, F1 and AUC. Lastly, neural methods seem better in line with human annotators with regard to certainty and disagreement.

Question: Which model architecture do they opt for?
Answer: Recurrent Neural Networks, Convolutional Neural Networks

RNNs and CNNs

HAN BIBREF10, CNN BIBREF11

CNN, RNN

Question: Which dataset do they use?
Answer: Clueweb09

Clueweb09 derived dataset, new dataset based on Wikipedia crawl data

the Clueweb09 derived dataset , dataset based on Wikipedia crawl data

Clueweb09 derived dataset, Wikipedia crawl data

Question: Which setup shows proves to be the hardest: cross-topic, cross-domain, cross-temporal, or across annotators?
Answer: Unanswerable

Unanswerable

Unanswerable

Question: Which weak signal data do they use?
Answer: Unanswerable

semantic representations of word embeddings

Question: Do they compare their semantic feature approach to lexical approaches?
Answer: Yes

Yes

Yes

Yes