Satirical News Detection with Semantic Feature Extraction and Game-theoretic Rough Sets

Abstract: Satirical news detection is an important yet challenging task to prevent spread of misinformation. Many feature based and end-to-end neural nets based satirical news detection systems have been proposed and delivered promising results. Existing approaches explore comprehensive word features from satirical news articles, but lack semantic metrics using word vectors for tweet form satirical news. Moreover, the vagueness of satire and news parody determines that a news tweet can hardly be classified with a binary decision, that is, satirical or legitimate. To address these issues, we collect satirical and legitimate news tweets, and propose a semantic feature based approach. Features are extracted by exploring inconsistencies in phrases, entities, and between main and relative clauses. We apply game-theoretic rough set model to detect satirical news, in which probabilistic thresholds are derived by game equilibrium and repetition learning mechanism. Experimental results on the collected dataset show the robustness and improvement of the proposed approach compared with Pawlak rough set model and SVM.

Satirical news, which uses parody characterized in a conventional news style, has now become an entertainment on social media. While news satire is claimed to be pure comedic and of amusement, it makes statements on real events often with the aim of attaining social criticism and influencing change BIBREF0. Satirical news can also be misleading to readers, even though it is not designed for falsifications. Given such sophistication, satirical news detection is a necessary yet challenging natural language processing (NLP) task. Many feature based fake or satirical news detection systems BIBREF1, BIBREF2, BIBREF3 extract features from word relations given by statistics or lexical database, and other linguistic features. In addition, with the great success of deep learning in NLP in recent years, many end-to-end neural nets based detection systems BIBREF4, BIBREF5, BIBREF6 have been proposed and delivered promising results on satirical news article detection.
However, with the evolution of fast-paced social media, satirical news has been condensed into a satirical-news-in-one-sentence form. For example, one single tweet of “If earth continues to warm at current rate moon will be mostly underwater by 2400" by The Onion is largely consumed and spread by social media users than the corresponding full article posted on The Onion website. Existing detection systems trained on full document data might not be applicable to such form of satirical news. Therefore, we collect news tweets from satirical news sources such as The Onion, The New Yorker (Borowitz Report) and legitimate news sources such as Wall Street Journal and CNN Breaking News. We explore the syntactic tree of the sentence and extract inconsistencies between attributes and head noun in noun phrases. We also detect the existence of named entities and relations between named entities and noun phrases as well as contradictions between the main clause and corresponding prepositional phrase. For a satirical news, such inconsistencies often exist since satirical news usually combines irrelevant components so as to attain surprise and humor. The discrepancies are measured by cosine similarity between word components where words are represented by Glove BIBREF7. Sentence structures are derived by Flair, a state-of-the-art NLP framework, which better captures part-of-speech and named entity structures BIBREF8.
Due to the obscurity of satire genre and lacks of information given tweet form satirical news, there exists ambiguity in satirical news, which causes great difficulty to make a traditional binary decision. That is, it is difficult to classify one news as satirical or legitimate with available information. Three-way decisions, proposed by YY Yao, added an option - deferral decision in the traditional yes-and-no binary decisions and can be used to classify satirical news BIBREF9, BIBREF10. That is, one news may be classified as satirical, legitimate, and deferral. We apply rough sets model, particularly the game-theoretic rough sets to classify news into three groups, i.e., satirical, legitimate, and deferral. Game-theoretic rough set (GTRS) model, proposed by JT Yao and Herbert, is a recent promising model for decision making in the rough set context BIBREF11. GTRS determine three decision regions from a tradeoff perspective when multiple criteria are involved to evaluate the classification models BIBREF12. Games are formulated to obtain a tradeoff between involved criteria. The balanced thresholds of three decision regions can be induced from the game equilibria. GTRS have been applied in recommendation systems BIBREF13, medical decision making BIBREF14, uncertainty analysis BIBREF15, and spam filtering BIBREF16.
We apply GTRS model on our preprocessed dataset and divide all news into satirical, legitimate, or deferral regions. The probabilistic thresholds that determine three decision regions are obtained by formulating competitive games between accuracy and coverage and then finding Nash equilibrium of games. We perform extensive experiments on the collected dataset, fine-tuning the model by different discretization methods and variation of equivalent classes. The experimental result shows that the performance of the proposed model is superior compared with Pawlak rough sets model and SVM.
Satirical news detection is an important yet challenging NLP task. Many feature based models have been proposed. Burfoot et al. extracted features of headline, profanity, and slang using word relations given by statistical metrics and lexical database BIBREF1. Rubin et al. proposed a SVM based model with five features (absurdity, humor, grammar, negative affect, and punctuation) for fake news document detection BIBREF2. Yang et al. presented linguistic features such as psycholinguistic feature based on dictionary and writing stylistic feature from part-of-speech tags distribution frequency BIBREF17. Shu et al. gave a survey in which a set of feature extraction methods is introduced for fake news on social media BIBREF3. Conroy et al. also uses social network behavior to detect fake news BIBREF18. For satirical sentence classification, Davidov et al. extract patterns using word frequency and punctuation features for tweet sentences and amazon comments BIBREF19. The detection of a certain type of sarcasm which contracts positive sentiment with a negative situation by analyzing the sentence pattern with a bootstrapped learning was also discussed BIBREF20. Although word level statistical features are widely used, with advanced word representations and state-of-the-art part-of-speech tagging and named entity recognition model, we observe that semantic features are more important than word level statistical features to model performance. Thus, we decompose the syntactic tree and use word vectors to more precisely capture the semantic inconsistencies in different structural parts of a satirical news tweet.
Recently, with the success of deep learning in NLP, many researchers attempted to detect fake news with end-to-end neural nets based approaches. Ruchansky et al. proposed a hybrid deep neural model which processes both text and user information BIBREF5, while Wang et al. proposed a neural network model that takes both text and image data BIBREF6 for detection. Sarkar et al. presented a neural network with attention to both capture sentence level and document level satire BIBREF4. Some research analyzed sarcasm from non-news text. Ghosh and Veale BIBREF21 used both the linguistic context and the psychological context information with a bi-directional LSTM to detect sarcasm in users' tweets. They also published a feedback-based dataset by collecting the responses from the tweets authors for future analysis. While all these works detect fake news given full text or image content, or target on non-news tweets, we attempt bridge the gap and detect satirical news by analyzing news tweets which concisely summarize the content of news.
In this section, we will describe the composition and preprocessing of our dataset and introduce our model in detail. We create our dataset by collecting legitimate and satirical news tweets from different news source accounts. Our model aims to detect whether the content of a news tweet is satirical or legitimate. We first extract the semantic features based on inconsistencies in different structural parts of the tweet sentences, and then use these features to train game-theoretic rough set decision model.
We collected approximately 9,000 news tweets from satirical news sources such as The Onion and Borowitz Report and about 11,000 news tweets from legitimate new sources such as Wall Street Journal and CNN Breaking News over the past three years. Each tweet is a concise summary of a news article. The duplicated and extreme short tweets are removed.A news tweet is labeled as satirical if it is written by satirical news sources and legitimate if it is from legitimate news sources. Table TABREF2 gives an example of tweet instances that comprise our dataset.
Satirical news is not based on or does not aim to state the fact. Rather, it uses parody or humor to make statement, criticisms, or just amusements. In order to achieve such effect, contradictions are greatly utilized. Therefore, inconsistencies significantly exist in different parts of a satirical news tweet. In addition, there is a lack of entity or inconsistency between entities in news satire. We extracted these features at semantic level from different sub-structures of the news tweet. Different structural parts of the sentence are derived by part-of-speech tagging and named entity recognition by Flair. The inconsistencies in different structures are measured by cosine similarity of word phrases where words are represented by Glove word vectors. We explored three different aspects of inconsistency and designed metrics for their measurements. A word level feature using tf-idf BIBREF22 is added for robustness.
One way for a news satire to obtain surprise or humor effect is to combine irrelevant or less jointly used attributes and the head noun which they modified. For example, noun phrase such as “rampant accountability", “posthumous apology", “Vatican basement", “self-imposed mental construct" and other rare combinations are widely used in satirical news, while individual words themselves are common. To measure such inconsistency, we first select all leaf noun phrases (NP) extracted from the semantic trees to avoid repeated calculation. Then for each noun phrase, each adjacent word pair is selected and represented by 100-dim Glove word vector denoted as $(v_{t},w_{t})$. We define the averaged cosine similarity of noun phrase word pairs as:
where $T$ is a total number of word pairs. We use $S_{N\!P}$ as a feature to capture the overall inconsistency in noun phrase uses. $S_{N\!P}$ ranges from -1 to 1, where a smaller value indicates more significant inconsistency.
Another commonly used rhetoric approach for news satire is to make contradiction between the main clause and its prepositional phrase or relative clause. For instance, in the tweet “Trump boys counter Chinese currency manipulation $by$ adding extra zeros To $20 Bills.", contradiction or surprise is gained by contrasting irrelevant statements provided by different parts of the sentence. Let $q$ and $p$ denote two clauses separated by main/relative relation or preposition, and $(w_{1},w_{1},... w_{q})$ and $(v_{1},v_{1},... v_{p})$ be the vectorized words in $q$ and $p$. Then we define inconsistency between $q$ and $p$ as:
Similarly, the feature $S_{Q\!P}$ is measured by cosine similarity of linear summations of word vectors, where smaller value indicates more significant inconsistency.
Even though many satirical news tweets are made based on real persons or events, most of them lack specific entities. Rather, because the news is fabricated, news writers use the words such as “man",“woman",“local man", “area woman",“local family" as subject. However, the inconsistency between named entities and noun phrases often exists in a news satire if a named entity is included. For example, the named entity “Andrew Yang" and the noun phrases “time vortex" show great inconsistency than “President Trump", "Senate Republicans", and “White House" do in the legitimate news “President Trump invites Senate Republicans to the White House to talk about the funding bill." We define such inconsistency as a categorical feature that:
$S_{N\! E\! R\! N}$ is the cosine similarity of named entities and noun phrases of a certain sentence and $\bar{S}_{N\! E\! R\! N}$ is the mean value of $S_{N\! E\! R\! N}$ in corpus.
We calculated the difference of tf-idf scores between legitimate news corpus and satirical news corpus for each single word. Then, the set $S_{voc}$ that includes most representative legitimate news words is created by selecting top 100 words given the tf-idf difference. For a news tweet and any word $w$ in the tweet, we define the binary feature $B_{voc}$ as:
We construct a Game-theoretic Rough Sets model for classification given the extracted features. Suppose $E\subseteq U \times U$ is an equivalence relation on a finite nonempty universe of objects $U$, where $E$ is reflexive, symmetric, and transitive. The equivalence class containing an object $x$ is given by $[x]=\lbrace y\in U|xEy\rbrace $. The objects in one equivalence class all have the same attribute values. In the satirical news context, given an undefined concept $satire$, probabilistic rough sets divide all news into three pairwise disjoint groups i.e., the satirical group $POS(satire)$, legitimate group $NEG(satire)$, and deferral group $BND(satire)$, by using the conditional probability $Pr(satire|[x]) = \frac{|satire\cap [x]|}{|[x]|}$ as the evaluation function, and $(\alpha ,\beta )$ as the acceptance and rejection thresholds BIBREF23, BIBREF9, BIBREF10, that is,
Given an equivalence class $[x]$, if the conditional probability $Pr(satire|[x])$ is greater than or equal to the specified acceptance threshold $\alpha $, i.e., $Pr(satire|[x])\ge \alpha $, we accept the news in $[x]$ as $satirical$. If $Pr(satire|[x])$ is less than or equal to the specified rejection threshold $\beta $, i.e., $Pr(satire|[x])\le \beta $ we reject the news in $[x]$ as $satirical$, or we accept the news in $[x]$ as $legitimate$. If $Pr(satire|[x])$ is between $\alpha $ and $\beta $, i.e., $\beta <Pr(satire|[x])<\alpha $, we defer to make decisions on the news in $[x]$. Pawlak rough sets can be viewed as a special case of probabilistic rough sets with $(\alpha ,\beta )=(1,0)$.
Given a pair of probabilistic thresholds $(\alpha , \beta )$, we can obtain a news classifier according to Equation (DISPLAY_FORM13). The three regions are a partition of the universe $U$,
Then, the accuracy and coverage rate to evaluate the performance of the derived classifier are defined as follows BIBREF12,
The criterion coverage indicates the proportions of news that can be confidently classified. Next, we will obtain $(\alpha , \beta )$ by game formulation and repetition learning.
We construct a game $G=\lbrace O,S,u\rbrace $ given the set of game players $O$, the set of strategy profile $S$, and the payoff functions $u$, where the accuracy and coverage are two players, respectively, i.e., $O=\lbrace acc, cov\rbrace $.
The set of strategy profiles $S=S_{acc}\times S_{cov}$, where $S_{acc}$ and $S_{cov} $ are sets of possible strategies or actions performed by players $acc$ and $cov$. The initial thresholds are set as $(1,0)$. All these strategies are the changes made on the initial thresholds,
$c_{acc}$ and $c_{cov}$ denote the change steps used by two players, and their values are determined by the concrete experiment date set.
Payoff functions. The payoffs of players are $u=(u_{acc},u_{cov})$, and $u_{acc}$ and $u_{cov}$ denote the payoff functions of players $acc$ and $cov$, respectively. Given a strategy profile $p=(s, t)$ with player $acc$ performing $s$ and player $cov$ performing $t$, the payoffs of $acc$ and $cov$ are $u_{acc}(s, t)$ and $u_{cov}(s, t)$. We use $u_{acc}(\alpha ,\beta )$ and $u_{cov}(\alpha ,\beta )$ to show this relationship. The payoff functions $u_{acc}(\alpha ,\beta )$ and $u_{cov}(\alpha ,\beta )$ are defined as,
where $Acc_{(\alpha , \beta )}(Satire)$ and $Cov_{(\alpha , \beta )}(Satire)$ are the accuracy and coverage defined in Equations (DISPLAY_FORM15) and (DISPLAY_FORM16).
Payoff table. We use payoff tables to represent the formulated game. Table TABREF20 shows a payoff table example in which both players have 3 strategies defined in Equation refeq:stategies.
The arrow $\downarrow $ denotes decreasing a value and $\uparrow $ denotes increasing a value. On each cell, the threshold values are determined by two players.
We repeat the game with the new thresholds until a balanced solution is reached. We first analyzes the pure strategy equilibrium of the game and then check if the stopping criteria are satisfied.
Game equilibrium. The game solution of pure strategy Nash equilibrium is used to determine possible game outcomes in GTRS. The strategy profile $(s_{i},t_{j})$ is a pure strategy Nash equilibrium, if
This means that none of players would like to change his strategy or they would loss benefit if deriving from this strategy profile, provided this player has the knowledge of other player's strategy.
Repetition of games. Assuming that we formulate a game, in which the initial thresholds are $(\alpha , \beta )$, and the equilibrium analysis shows that the thresholds corresponding to the equilibrium are $(\alpha ^{*}, \beta ^{*})$. If the thresholds $(\alpha ^{*}, \beta ^{*})$ do not satisfy the stopping criterion, we will update the initial thresholds in the subsequent games. The initial thresholds of the new game will be set as $(\alpha ^{*}, \beta ^{*})$. If the thresholds $(\alpha ^{*}, \beta ^{*})$ satisfy the stopping criterion, we may stop the repetition of games.
Stopping criterion. We define the stopping criteria so that the iterations of games can stop at a proper time. In this research, we set the stopping criterion as within the range of thresholds, the increase of one player's payoff is less than the decrease of the other player's payoff.
There are 8757 news records in our preprocessed data set. We use Jenks natural breaks BIBREF24 to discretize continuous variables $S_{N\!P}$ and $S_{Q\!P}$ both into five categories denoted by nominal values from 0 to 4, where larger values still fall into bins with larger nominal value. Let $D_{N\!P}$ and $D_{Q\!P}$ denote the discretized variables $S_{N\!P}$ and $S_{Q\!P}$, respectively. We derived the information table that only contains discrete features from our original dataset. A fraction of the information table is shown in Table TABREF23.
The news whose condition attributes have the same values are classified in an equivalence class $X_i$. We derived 149 equivalence classes and calculated the corresponding probability $Pr(X_i)$ and condition probability $Pr(Satire|X_i)$ for each $X_i$. The probability $Pr(X_{i})$ denotes the ratio of the number of news contained in the equivalence class $X_i$ to the total number of news in the dataset, while the conditional probability $Pr(Satire|X_{i})$ is the proportion of news in $X_i$ that are satirical. We combine the equivalence classes with the same conditional probability and reduce the number of equivalence classes to 108. Table TABREF24 shows a part of the probabilistic data information about the concept satire.
We formulated a competitive game between the criteria accuracy and coverage to obtain the balanced probabilistic thresholds with the initial thresholds $(\alpha , \beta )=(1,0)$ and learning rate 0.03. As shown in the payoff table Table TABREF26,
the cell at the right bottom corner is the game equilibrium whose strategy profile is ($\beta $ increases 0.06, $\alpha $ decreases 0.06). The payoffs of the players are (0.9784,0.3343). We set the stopping criterion as the increase of one player's payoff is less than the decrease of the other player's payoff when the thresholds are within the range. When the thresholds change from (1,0) to (0.94, 0.06), the accuracy is decreased from 1 to 0.9784 but the coverage is increased from 0.0795 to 0.3343. We repeat the game by setting $(0.94, 0.06)$ as the next initial thresholds.
The competitive games are repeated seven times. The result is shown in Table TABREF27.
After the eighth iteration, the repetition of game is stopped because the further changes on thresholds may cause the thresholds lay outside of the range $0 < \beta < \alpha <1$, and the final result is the equilibrium of the seventh game $(\alpha , \beta )=(0.52, 0.48)$.
We compare Pawlak rough sets, SVM, and our GTRS approach on the proposed dataset. Table TABREF29 shows the results on the experimental data.
The SVM classifier achieved an accuracy of $78\%$ with a $100\%$ coverage. The Pawlak rough set model using $(\alpha , \beta )=(1,0)$ achieves a $100\%$ accuracy and a coverage ratio of $7.95\%$, which means it can only classify $7.95\%$ of the data. The classifier constructed by GTRS with $(\alpha , \beta )=(0.52, 0.48)$ reached an accuracy $82.71\%$ and a coverage $97.49\%$. which indicates that $97.49\%$ of data are able to be classified with accuracy of $82.71\%$. The remaining $2.51\%$ of data can not be classified without providing more information. To make our method comparable to other baselines such as SVM, we assume random guessing is made on the deferral region and present the modified accuracy. The modified accuracy for our approach is then $0.8271\times 0.9749 + 0.5 \times 0.0251 =81.89\%$. Our methods shows significant improvement as compared to Pawlak model and SVM.
In this paper, we propose a satirical news detection approach based on extracted semantic features and game-theoretic rough sets. In our mode, the semantic features extraction captures the inconsistency in the different structural parts of the sentences and the GTRS classifier can process the incomplete information based on repetitive learning and the acceptance and rejection thresholds. The experimental results on our created satirical and legitimate news tweets dataset show that our model significantly outperforms Pawlak rough set model and SVM. In particular, we demonstrate our model's ability to interpret satirical news detection from a semantic and information trade-off perspective. Other interesting extensions of our paper may be to use rough set models to extract the linguistic features at document level.

Figure and table captions:
Table 1. Examples of instances comprising the news tweet dataset
Table 2. An example of a payoff table
Table 3. The Information Table
Table 4. Summary of the partial experimental data
Table 5. The payoff table
Table 6. The repetition of game
Table 7. Experimental results

Questions about this paper:
How much improvement do they get?

Answers:
Their GTRS approach got an improvement of 3.89% compared to SVM and 27.91% compared to Pawlak.