Introduction
Urban legends are a genre of modern folklore consisting of stories told as true – and plausible enough to be believed – about some rare and exceptional events that supposedly happened to a real person or in a real place.
Whether urban legends are produced by individual authors or emerge spontaneously, they typically spread “virally" across communities and tend to change over time with repetition and embellishment, like memes BIBREF0 . For example the sewer alligator, that originally “appeared” in New York City BIBREF1 , also appeared in different cities to suit regional variations. Though it is considered synonymous of “false belief," the term urban legend refers to a subtler and more complex phenomenon. The crucial factor is that the story is told as true in the absence of verification. Folklorists are generally more interested in the social context and meaning of urban legends than their truth value. From an NLP point of view, instead, it is interesting to computationally explore those linguistic characteristics that make them appealing and bring people to circulate them. With the advent of the Internet, urban legends gained new lifeblood, as they began to be circulated by e-mail.
In BIBREF2 , the authors discuss the idea of “stickiness" popularized by the book “The Tipping Point” BIBREF3 , seeking to explain what makes an idea or concept memorable or interesting. They also focus on urban legends and claim that, by following the acronym “SUCCES" (each letter referring to a characteristic that makes an idea “sticky"), it is possible to describe their prototypical structure:
Such features are allegedly placed at the core of persuasive and viral language; urban legends constitute an ideal framework with which to computationally verify these assertions. Table 1 displays a few examples of urban legends claims.
In particular we will investigate some of the prototypical characteristics that can be found in urban legends as compared to similar literary genres. In our view, urban legends are viral since they are stressed by a tension between credible and incredible: credible like a news and incredible like a fairy tale. We will focus on the idea that urban legends should mimic the details of news (who, where, when) to be credible, and they should be emotional and readable like the story of a fairy tale to be catchy and memorable. We will verify these psychological hypotheses – appeared in the literature – using NLP tools, to drive a quantitative analysis of these qualitative theories. For example, the idea that urban legends derive much of their credibility from details concerning the location where the situation took place, is presented in BIBREF4 . Anecdotically, the television series “1000 Ways to Die" – that recreates unusual supposed deaths and debunked urban legends in a way similar to the Darwin Awards – introducing each story with the location and date of each supposed incident, to render it more credible.
In the tension between credible and incredible, details should be neither too specific, like in the news, nor too few, as in fairy tales: effective urban legends should be credible but not verifiable. Similarly, emotions should be enough to make it sticky/catchy but not too much to render it not-credible. Finally urban legends should be easy to read, similar to fairy tales, to render them more memorable. As an example consider the following excerpt, taken from the “Kidney Theft” urban legend, as reported by snopes.com:
There is no very strong emotional wording in this example, it is the situation itself that is scary; on the contrary the email contains locations, the signature of a presumed Jerry Mayfield, and – noticeably – credibility is also explicitly addressed in the text with the adjectives “real”, “documented” and “confirmable”.
In the following sections we first review relevant work that addresses the problem of deceptive language and behavior both in online and offline scenarios, followed by an overview of work that addresses the virality of online content. Then we describe the data collected for our experiments and the features extracted to model the aforementioned prototypical characteristics of urban legends. We use these features in both descriptive statistics and generalization tasks and we report the best performing features. Finally we discuss future research on further prototypical characteristics of urban legends.
Related Work
The topic of deceptive and/or false messages is a burning topic within the NLP community. A seminal work on the linguistic recognition of lies can be found in BIBREF5 . Still, defense from subtle persuasive language in broadcast messages, including social networks, is needed in many applied scenarios. Viral messages have become a very important factor for persuasion and are currently almost entirely out of control. So, protection from fraudulent communication is needed, especially in competitive commercial situations. Two main approaches are currently under investigation in the literature:
1) Recognizing the linguistic characteristics of deceptive content in the social web: for example preventing deceptive consumer reviews BIBREF6 on sites like Trip Advisor is fundamental both for consumers seeking genuine reviews, and for the reputation of the site itself. Deceptive consumer reviews are fictitious opinions that have been deliberately written to sound authentic. Another example concerns online advertising BIBREF7 : detecting fraudulent ads is in the interest of users, of service providers (e.g. Google AdWords system), and other advertisers. An interesting phenomenon at the crossroad of viral phenomena and deceptive customer reviews, where ironic reviews (such as the case of the mountain three wolf moon) create phenomena of social contagion, is discussed in BIBREF8 .
2) Recognizing on-line behavioral patterns of deceptive users: For example recognizing groups of propagandists or fake accounts that are used to push the virality of content BIBREF9 . Four main patterns are recognized: (i) sending high volumes of tweets over short periods of time, (ii) retweeting while publishing little original content, (iii) quickly retweeting, and (iv) colluding with other, seemingly unrelated, users to send duplicate or near-duplicate messages on the same topic simultaneously. Another example is BIBREF10 where the authors hypothesize that there is a set of representative distributions of review rating scores. Deceptive business entities that hire people to write fake reviews can then be recognized since they will necessarily distort distribution of review scores, leaving “distributional footprints" behind.
We want to consider a third point, which is linked to the previous two but different at the same time: deceptive content that spreads quickly but without an explicit strategy of making them spread, which is the case with urban legends. Finally, the spreading dynamics of an urban legend on one hand closely resembles those of memes that undergo many variations while spreading BIBREF11 ; on the other hand their characteristics resemble those of viral content. Several researchers have studied information flow, community building and similar processes using Social Networking sites as a reference BIBREF12 , BIBREF13 , BIBREF14 . However, the great majority concentrate on network-related features without taking into account the actual content spreading within the network BIBREF15 . A hybrid approach focusing on both product characteristics and network related features is presented in BIBREF16 : in particular, the authors study the effect of passive-broadcast and active-personalized notifications embedded in an application to foster word of mouth.
Recently, the correlation between content characteristics and virality has begun to be investigated, especially with regard to textual content; in BIBREF17 , for example, features derived from sentiment analysis of comments are used to predict stories' popularity. The work in BIBREF18 uses New York Times articles to examine the relationship between emotions evoked by the content and virality, using semi-automated sentiment analysis to quantify the affectivity and emotionality of each article. Results suggest a strong relationship between affect and virality, where virality corresponds to the number of times the article was email forwarded.
The relevant work in BIBREF19 measures a different form of content spreading by analyzing which features of a movie quote make it “memorable" online. Another approach to content virality, somehow complementary to the previous one, is presented in BIBREF11 , and takes the perspective of understanding which modification dynamics make a meme spread from one person to another (while movie quotes spread remaining exactly the same). More recently, some works tried to investigate how different textual contents give rise to different reactions in the audience: the work presented in BIBREF20 correlates several viral phenomena with the wording of a post, while BIBREF21 shows that specific content features variations (like the readability level of an abstract) differentiate among virality level of downloads, bookmarking, and citations.
Datasets
To explore the characteristics of urban legends and understand the effectiveness of our ideas we collected a specific dataset. It is composed of roughly 8000 textual examples: 2518 Urban Legends (UL), 1860 Fairy Tales (FT) and 3575 Google News articles (GN). The description of how the datasets have been created follows.
Feature Extraction
After collecting the datasets we extracted four different groups of features, relevant to the prototypical characteristics we want to analyze.
Named Entities, (NE). To annotate named entities we used the TextPro toolkit BIBREF24 , and in particular its Named Entities recognition module. The output of the tool is in the IOB2 format and includes the tags Person (PER), Organization (ORG), Location (LOC) and Miscellaneous (MISC).
Temporal Expressions, (TIMEX). To annotate temporal expressions we used the toolkit TTK BIBREF25 . The output of the tool is in TimeML annotation language format BIBREF26 . In particular time expressions are flagged with TIMEX3 tags (tern.mitre.org). The tags considered are DATE, DURATION and TIME.
To compute the importance of the aforementioned features, and to explore the characteristics of urban legend texts, we used the method proposed in BIBREF5 . We calculate a score associated with a given set of entities (features), as a measure of saliency for the given word class inside the text, called coverage.
More formally, given a set of feature instances present in a text, C $=$ { $W_1$ , $W_2$ , ..., $W_N$ }, we define the feature coverage in that text (or corpus) $A$ as the percentage of words from $A$ belonging to the feature set $C$ :
$$Coverage_{A}(C) = \frac{\sum _{W_{i} \in C} Frequency_A(W_i)}{Words_{A}}$$   (Eq. 14)
where $Frequency_A(W_i)$ represents the total number of feature occurrences $W_i$ inside the text A, and $Words_{A}$ represents the total size (in words) of the text. Note that we computed features' coverage regardless of their actual length: “New York City" or “Paris” both count as one LOC even if the former is composed of three tokens while the latter only of one. Note also that this approach normalizes according to text length, avoiding biases due to different corpus characteristics.
Sentiment (SENT). Since the three corpora have different characteristics, rather than computing word polarity using specialized bag-of-words approaches, we resort to words' prior polarity - i.e. if a word out of context evokes something positive or something negative. This technique, even if less precise, guarantee that the same score is given to the same word in different contexts, and that none of the corpora is either overestimated or underestimated. To this end, we follow the methodology proposed in BIBREF27 , using SentiWordNet 3.0 BIBREF28 , that assigns prior polarities to words starting from their posterior polarities. In particular we choose the best performing approach. This formula uses a weighted mean, i.e. each sense weight is chosen according to a harmonic series. The rationale behind this choice is based on the assumption that more frequent senses should bear more “affective weight” than very rare senses when computing the prior polarity of a word. In particular, for each word we returned its positive (POS) and negative (NEG) prior polarity score:
$$\qquad POS =\frac{\sum _ {i=1}^{n}( \frac{1}{i} \times posScore_ {i})}{\sum _ {i=1}^{n}( \frac{1}{i})}$$   (Eq. 15)
where $posScore_ {i}$ represents the modulus of the positive polarity of the ith sense of that word. The NEG score is computed following the same procedure.
Emotions (EMO). To sense emotions from text we used the methodology described in BIBREF29 . The idea underlying the method is the distinction between direct and indirect affective words. For direct affective words, we refer to the WordNet Affect BIBREF30 lexicon, an extension of the WordNet database which employs six basic emotion labels (anger, disgust, fear, joy, sadness, surprise) to annotate WordNet synsets. LSA is then used to learn, in an unsupervised setting, a vector space from the British National Corpus. In the LSA space, each emotion label can be represented in various way. In particular, we employ the `LSA Emotion Synset' setting, in which the synsets of direct emotion words are considered. The affective load is computed in terms of its lexical similarity with respect to one of the six emotion labels. The overall affective load of a text is then calculated as the average of its similarity with each emotion label.
Emotions and Sentiment features are grouped under the label Affect (AFF).
Readability (READ). We further analyzed the texts in the three datasets according to readability indices, to understand whether there is a difference in the language difficulty among them. Basically, the task of readability assessment consists of quantifying how difficult a text is for a reader. This kind of assessment has been widely used for several purposes, such as evaluating the reading level of children and impaired persons and improving Web content accessibility, see for example what reported in BIBREF31 . We use three indices to compute the difficulty of a text: the Gunning Fog BIBREF32 , Flesch BIBREF33 and Kincaid BIBREF34 indices. These metrics combine factors such as word and sentence length that are easy to compute and approximate the linguistic elements that have an impact on readability. In the following formulae, $Sent_A$ represents the number of sentences in text $A$ , $Cpx_A$ the number of complex words (those with three or more syllables), and $Syll_A$ the total number of syllables.
The Fog index is a rough measure of how many years of schooling it would take someone to understand the content; higher scores indicate material that is harder to read. Texts requiring near-universal understanding have an index less than 8. Academic papers usually have a score between 15 and 20. The score, for a given text $A$ , is calculated according to the formula:
$$Fog_{A} = 0.4 \Big ( \frac{Words_A}{Sent_A} + 100 \frac{Cpx_A}{Words_A} \Big )$$   (Eq. 16)
The Flesch Index rates texts on a 100-point scale. Higher scores indicate material that is easier to read while lower numbers mark passages that are more difficult to read. Scores can be interpreted as: 90-100 for content easily understood by an average 11-year-old student, while 0-30 for content best understood by university graduates. The score is calculated with the following formula:
$$Flesch_{A} = 206.835 - 1.015 \frac{Words_A}{Sent_A} -84.6 \frac{Syll_A}{Words_A}$$   (Eq. 17)
The Kincaid Index or “Flesch–Kincaid Grade Level Formula" translates the 0-100 score of the Flesch Index to a U.S. grade level. It can be interpreted as the number of years of education required to understand this text, similar to the Gunning Fog index. The grade level is calculated with the following formula:
$$Kincaid_{A} = 0.39 \frac{Words_A}{Sent_A} + 11.8 \frac{Syll_A}{Words_A} - 15.59$$   (Eq. 18)
Descriptive Statistics
As can be seen from Tables 2 (Named Entities) and 3 (Temporal Expressions), urban legends place half-way between fairy tales and news, as we expected. While fairy tales represent out-of-time, out-of-place and always-true stories (“a long time ago in a faraway land"), news represent circumstantial description of events. This is reflected by the overall use of named entities (respectively almost three and four times more in UL and GN) and of temporal expressions (respectively almost two and three times more). Interestingly the use of person names is the only case where FT reduce the lead of UL and GN, and can be explained by the fact that characters in FT are usually addressed with proper names (e.g. “Hansel and Gretel”).
In Table 4 , statistics for sentiment and emotion coverage are reported. As can be seen, in the SENT group of features the differences are less marked and, quite surprisingly, ULs have the lowest scores. As we would expect, FTs have the highest score. Sentiment does not meet our initial expectation and seems in contrast with previous works – see for example what reported in BIBREF35 on UL and evoked emotions; still the results on sentiment as compared to emotions can be explained by the distinction between affective impact and affective language. In fact, affective impact can either derive from the wording of the text itself (usage of strong affect words), or from the depicted situation (i.e. emotions are evoked by describing a vivid situation with a plain language). In our experiment we tested the `wording' using SENT features while the `evoked emotions' with the EMO features. So, UL seem to use a plain and objective language, similar to GN, to gain credibility, but tend to evoke strong emotions (similar to FT) to be catchy. Let us consider the “Kidney Theft” excerpt described in Section 1, as stated, there is no very strong emotional wording in this UL, it is the depicted situation that is scary per se.
In Table 5 , statistics for readability are reported. As can be seen, ULs are readable in a way similar to fairy tales. Still, depending on the readability indices, that grasp different aspects of text difficulty, ULs are either slightly easier than FTs or half-way between FTs and ULs similar to the cases of Tables 2 and 3 .
This behavior can be explained by the fact that ULs have a simpler syntax than FTs but a more complex lexicon. In fact, inspecting the individual elements of the formulae, as reported in the second part of Table 5 , we see that while the percentage of complex words (either $\frac{Cpx_A}{Words_A}$ or $\frac{Syll_A}{Words_A}$ ) puts UL halfway between FT and GN, the average length of sentences ( $\frac{Words_A}{Sent_A}$ ) is surprisingly higher for FT than GN and in turn UL. So, depending on the weight given either to complex words or to sentence length, the results in Table 5 can be interpreted.
All differences in the means reported in the tables are statistically significant (Student's t-test, $p<0.001$ ) apart from TIME, between UL and FT, and DURATION, between UL and GN, (signalled with * in Table 3 ).
Turning to the analysis of variance, we see that FT is – on average – a more cohesive genre, with lower standard deviations, while GN and UL have higher and closer standard deviations. In fact, all differences in the standard deviations reported in the tables are statistically significant (f-test, $p<0.001$ ) apart between UL and GN in Fog, Kincaid and in ALL sentiment (signalled with * in the respective Tables).
Classification Experiments
The goal of our experiments is to understand to what extent it is possible to assign a text to one of the aforementioned classes using just the prototypical characteristics (features) discussed above, and whether there is a subset of features that stands out among the others in this classification task. For every feature combination we conducted a binary classification experiment with ten-fold cross validation on the dataset. We always randomly downsampled the majority class in order to make the dataset balanced, i.e. 50% of positive examples and 50% of negative examples; this accounts for a random baseline of 0.5. We also normalized all features according to z-score. Experiments were carried out using SVM BIBREF36 , in particular libSVM BIBREF37 under its default settings. Results are reported in Table 6 ; all significance tests discussed below are computed using an approximate randomization test BIBREF38 .
Urban Legends vs News. In the UL vs. GN classification task, while all the features together performed well (F1 = 0.833), improving over all other subgroups of features ( $p<0.001$ ), no single group of features performed so well, apart from READ (F1 = 0.763, $p<0.001$ ). Particularly, the temporal features (TIMEX) performed worse than AFF and NE ( $p<0.001$ ). Still, all features improved over the baseline ( $p<0.001$ ).
Urban Legends vs Fairy Tales. In the UL vs. FT classification task, all the features together performed better than the previous experiment (F1 = 0.897), again improving over all the other subgroups of features alone ( $p<0.001$ ). Interestingly, the best discriminative subgroup of features (still READ, F1 = 0.868) in this case reduces the lead with respect to all the features together (ALL) and improves over the others subgroups ( $p<0.001$ ) apart from the AFF group – from which has no significant difference – that in this case performs better than in the previous experiment. On the contrary, the TIMEX group had similar performances as the previous experiment, while NE improved its performance. Finally, all groups of features had a statistically significant improvement over the baseline ( $p<0.001$ ).
In Table 7 we report the performances of the various classification tasks in term of precision, recall and F1 over the single classes. Interestingly, for almost all feature combinations the classifiers had slightly higher precision than recall for UL, while the contrary holds for FT and GN.
News vs Fairy Tales. Finally, we wanted to check whether UL being “half-way" between GN and FT can be observed in our classification experiments as well. If this hypothesis is correct, by classifying GN vs. FT we would expect to find higher performance than previous experiments. Results show that this is in fact the case. All features together performed better than all previous experiment and incredibly well (F1= 0.978), again improving over all the other subgroups of features alone ( $p<0.001$ ) apart from READ that performs equally well (F1=0.973, no statistically significant difference). Notably, all other groups of features improves over the UL vs. GN and the UL vs. FT tasks. Finally, all groups of features had a statistically significant improvement over the random baseline ( $p<0.001$ ).
Three Class Classification. Finally we also tested feature predictivity on a three class classification task (UL vs GN vs FT). Since in this case we did not performed downsampling, we use the ZeroR classifier as a baseline. For the sake of interpretability of results, along with precision, recall and F1 we also provide the Matthews Correlation Coefficient (MCC) which is useful for unbalanced datasets, as presented in BIBREF39 for the multiclass case. MCC returns a value between -1 and +1, where +1 represents a perfect prediction, 0 no better than random and -1 indicates total disagreement. Results are consistent with previous experiments. In Table 8 , all feature configurations show an improvement over the baseline ( $p<0.001$ ) but the temporal features (TIMEX) have far lower discriminative power as compared to others groups of features (MCC=0.339).
Discussion
While between UL and GN the discrimination is given by a skillful mixture of all the prototypical features together, where none has a clear predominance over the others, between UL and FT, readability (READ) and affect (AFF) play a major role. From the summary in Table 8 we see that while ALL features together have the highest averaged F1, READ is the best performing subset of features in all experiments, followed by AFF, NER and TIMEX that perform reasonably well. n general, these experiments proved the goodness of our features in discriminating UL against FT and GN in a machine learning framework, confirming the results emerged from the quantitative analysis part. In particular, as expected, these features gave the best results in the GN vs FT experiments, showing that these two genres represent the extremes of a continuum where ULs are placed.
Ecological Experiments in the News Domain
As a final validation of our feature importance we also set up experiments where we controlled for the medium where the message is delivered, specifically the online news domain. Since Newspapers exist for all kinds of stories and with all sorts of reputations for reliability we focused on two specific websites. One is the Weekly World News (WWN), a news website with very low reliability where many of the stories have the qualities of urban legends (the WWN was famous for stories about Bigfoot and UFOs, etc.). The other website is The New York Times (NYT), known for its high reliability and fact-checking procedures.
We scraped the WWN for a total of 225 stories, and then randomly selected an equal amount of stories from the NYT. For both datasets we extracted the same set of features discussed in the previous sections. For every feature combination we conducted a binary classification experiment with ten-fold cross validation on the dataset. Since the dataset is balanced, this accounts for a random baseline of 0.5. We also normalized all features according to z-score. Results are reported in Table 9 .
Also in this case our features are able to discriminate between reliable and non-reliable stories (namely those coming from NYT and WWN). In particular, all the features together performed very well (F1 = 0.864), improving over all other subgroups of features (p $<$ 0.001), and NE, TIMEX, READ performed equally well improving over AFF that was the least effective (p $<$ 0.001). Still, AFF improves over the random baseline (p $<$ 0.001).
With this last experiment we were able to show that stories from different newspapers of differing reliability might be classified correctly using the features learned for discriminating regular news from urban legends. So, also in more applicative and ecological scenarios, where stories come from the same medium (online news) these features are useful in discriminating stories on the basis of their UL-ness or GN-ness.
Conclusions
In this paper we have presented a study on urban legends, a genre of modern folklore consisting of stories about some rare and exceptional events plausible enough to be believed. We argued that urban legends represent a form of “sticky” deceptive text, marked by a tension between the credible and incredible. To be credible they should resemble a news article while being incredible like a fairy tale. In particular we focused on the idea that ULs should mimic the details of news (who, where, when) to be credible, while being emotional and readable like a fairy tale to be catchy and memorable. Using NLP tools we presented a quantitative analysis of these simple yet effective features and provided some machine learning experiments showing that it is possible to recognize an urban legend using just these prototypical characteristics. In the future we want to explore other prototypical aspects of urban legends like, for example, linguistic style BIBREF40 , BIBREF41 . With regard to sentiment, besides the simple word polarities we used, we will explore the emotions expressed in UL, FT and GN, using an approach similar to the one described in BIBREF29 . Exploiting knowledge-based and corpus-based methods, that approach deals with automatic recognition of affect, annotating texts with six basic emotions. We believe that fine-grained emotion annotation of urban legends could shed more light in the understanding the mechanisms behind persuasive language.

Question: How accurate is their predictive model?
Answer: Unanswerable