Question: How does the author obtain the relevant tweets?
Evidence: Social media is now becoming an important real-time information source, especially during natural disasters and emergencies. It is now very common for traditional news media to frequently probe users and resort to social media platforms to obtain real-time developments of events. According to a recent survey by Pew Research Center, in 2017, more than two-thirds of Americans read some of their news on social media. Even for American people who are 50 or older, INLINEFORM0 of them report getting news from social media, which is INLINEFORM1 points higher than the number in 2016. Among all major social media sites, Twitter is most frequently used as a news source, with INLINEFORM2 of its users obtaining their news from Twitter. All these statistical facts suggest that understanding user-generated noisy social media text from Twitter is a significant task.
In recent years, while several tools for core natural language understanding tasks involving syntactic and semantic analysis have been developed for noisy social media text BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , there is little work on question answering or reading comprehension over social media, with the primary bottleneck being the lack of available datasets. We observe that recently proposed QA datasets usually focus on formal domains, e.g. CNN/DailyMail BIBREF4 and NewsQA BIBREF5 on news articles; SQuAD BIBREF6 and WikiMovies BIBREF7 that use Wikipedia.
In this paper, we propose the first large-scale dataset for QA over social media data. Rather than naively obtaining tweets from Twitter using the Twitter API which can yield irrelevant tweets with no valuable information, we restrict ourselves only to tweets which have been used by journalists in news articles thus implicitly implying that such tweets contain useful and relevant information. To obtain such relevant tweets, we crawled thousands of news articles that include tweet quotations and then employed crowd-sourcing to elicit questions and answers based on these event-aligned tweets. Table TABREF3 gives an example from our TweetQA dataset. It shows that QA over tweets raises challenges not only because of the informal nature of oral-style texts (e.g. inferring the answer from multiple short sentences, like the phrase “so young” that forms an independent sentence in the example), but also from tweet-specific expressions (such as inferring that it is “Jay Sean” feeling sad about Paul's death because he posted the tweet).
Furthermore, we show the distinctive nature of TweetQA by comparing the collected data with traditional QA datasets collected primarily from formal domains. In particular, we demonstrate empirically that three strong neural models which achieve good performance on formal data do not generalize well to social media data, bringing out challenges to developing QA systems that work well on social media domains.
In summary, our contributions are:
To obtain such relevant tweets, we crawled thousands of news articles that include tweet quotations and then employed crowd-sourcing to elicit questions and answers based on these event-aligned tweets.
Answer: The authors crawled thousands of news articles that include tweet quotations and then employed crowd-sourcing to elicit questions and answers based on these event-aligned tweets.