prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. what datasets did they use?
Context 1: Introduction
Accurate grapheme-to-phoneme conversion (g2p) is important for any application that depends on the sometimes inconsistent relationship between spoken and written language  Most prominently, this includes text-to-speech and automatic speech recognition  Most work on g2p has focused on a few languages for which extensive pronunciation data is available BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3   Most languages lack these resources  However, a low resource language's writing system is likely to be similar to the writing systems of languages that do have sufficient pronunciation data  Therefore g2p may be possible for low resource languages if this high resource data can be properly utilized 
We attempt to leverage high resource data by treating g2p as a multisource neural machine translation (NMT) problem  The source sequences for our system are words in the standard orthography in any language  The target sequences are the corresponding representation in the International Phonetic Alphabet (IPA)  Our results show that the parameters learned by the shared encoder–decoder are able to exploit the orthographic and phonemic similarities between the various languages in our data 
Low Resource g2p
Our approach is similar in goal to deri2016grapheme's model for adapting high resource g2p models for low resource languages  They trained weighted finite state transducer (wFST) models on a variety of high resource languages, then transferred those models to low resource languages, using a language distance metric to choose which high resource models to use and a phoneme distance metric to map the high resource language's phonemes to the low resource language's phoneme inventory  These distance metrics are computed based on data from Phoible BIBREF4 and URIEL BIBREF5  
Other low resource g2p systems have used a strategy of combining multiple models  schlippe2014combining trained several data-driven g2p systems on varying quantities of monolingual data and combined their outputs with a phoneme-level voting scheme  This led to improvements over the best-performing single system for small quantities of data in some languages  jyothilow trained recurrent neural networks for small data sets and found that a version of their system that combined the neural network output with the output of the wFST-based Phonetisaurus system BIBREF1 did better than either system alone 
A different approach came from kim2012universal, who used supervised learning with an undirected graphical model to induce the grapheme–phoneme mappings for languages written in the Latin alphabet  Given a short text in a language, the model predicts the language's orthographic rules  To create phonemic context features from the short text, the model naïvely maps graphemes to IPA symbols written with the same character, and uses the features of these symbols to learn an approximation of the phonotactic constraints of the language  In their experiments, these phonotactic features proved to be more valuable than geographical and genetic features drawn from WALS BIBREF6  
Multilingual Neural NLP
In recent years, neural networks have emerged as a common way to use data from several languages in a single system  Google's zero-shot neural machine translation system BIBREF7 shares an encoder and decoder across all language pairs  In order to facilitate this multi-way translation, they prepend an artificial token to the beginning of each source sentence at both training and translation time  The token identifies what language the sentence should be translated to  This approach has three benefits: it is far more efficient than building a separate model for each language pair; it allows for translation between languages that share no parallel data; and it improves results on low-resource languages by allowing them to implicitly share parameters with high-resource languages  Our g2p system is inspired by this approach, although it differs in that there is only one target “language”, IPA, and the artificial tokens identify the language of the source instead of the language of the target 
Other work has also made use of multilingually-trained neural networks  Phoneme-level polyglot language models BIBREF8 train a single model on multiple languages and additionally condition on externally constructed typological data about the language  ostling2017continuous used a similar approach, in which a character-level neural language model is trained on a massively multilingual corpus  A language embedding vector is concatenated to the input at each time step  The language embeddings their system learned correlate closely to the genetic relationships between languages  However, neither of these models was applied to g2p 
Grapheme-to-Phoneme
g2p is the problem of converting the orthographic representation of a word into a phonemic representation  A phoneme is an abstract unit of sound which may have different realizations in different contexts  For example, the English phoneme has two phonetic realizations (or allophones):
English speakers without linguistic training often struggle to perceive any difference between these sounds  Writing systems usually do not distinguish between allophones: and are both written as INLINEFORM0 p INLINEFORM1 in English  The sounds are written differently in languages where they contrast, such as Hindi and Eastern Armenian 
Most writing systems in use today are glottographic, meaning that their symbols encode solely phonological information  But despite being glottographic, in few writing systems do graphemes correspond one-to-one with phonemes  There are cases in which multiple graphemes represent a single phoneme, as in the word the in English:
There are cases in which a single grapheme represents multiple phonemes, such as syllabaries, in which each symbol represents a syllable 
In many languages, there are silent letters, as in the word hora in Spanish:
There are more complicated correspondences, such as the silent e in English that affects the pronunciation of the previous vowel, as seen in the pair of words cape and cap 
It is possible for an orthographic system to have any or all of the above phenomena while remaining unambiguous  However, some orthographic systems contain ambiguities  English is well-known for its spelling ambiguities  Abjads, used for Arabic and Hebrew, do not give full representation to vowels 
Consequently, g2p is harder than simply replacing each grapheme symbol with a corresponding phoneme symbol  It is the problem of replacing a grapheme sequence INLINEFORM0
with a phoneme sequence INLINEFORM0
where the sequences are not necessarily of the same length  Data-driven g2p is therefore the problem of finding the phoneme sequence that maximizes the likelihood of the grapheme sequence: INLINEFORM0
Data-driven approaches are especially useful for problems in which the rules that govern them are complex and difficult to engineer by hand  g2p for languages with ambiguous orthographies is such a problem  Multilingual g2p, in which the various languages have similar but different and possibly contradictory spelling rules, can be seen as an extreme case of that  Therefore, a data-driven sequence-to-sequence model is a natural choice 
Encoder–Decoder Models
In order to find the best phoneme sequence, we use a neural encoder–decoder model with attention BIBREF9   The model consists of two main parts: the encoder compresses each source grapheme sequence INLINEFORM0 into a fixed-length vector  The decoder, conditioned on this fixed-length vector, generates the output phoneme sequence INLINEFORM1  
The encoder and decoder are both implemented as recurrent neural networks, which have the advantage of being able to process sequences of arbitrary length and use long histories efficiently  They are trained jointly to minimize cross-entropy on the training data  We had our best results when using a bidirectional encoder, which consists of two separate encoders which process the input in forward and reverse directions  We used long short-term memory units BIBREF10 for both the encoder and decoder  For the attention mechanism, we used the general global attention architecture described by luong2015effective 
We implemented all models with OpenNMT BIBREF11   Our hyperparameters, which we determined by experimentation, are listed in Table TABREF8  
Training Multilingual Models
Presenting pronunciation data in several languages to the network might create problems because different languages have different pronunciation patterns  For example, the string `real' is pronounced differently in English, German, Spanish, and Portuguese  We solve this problem by prepending each grapheme sequence with an artificial token consisting of the language's ISO 639-3 code enclosed in angle brackets  The English word `real', for example, would be presented to the system as
INLINEFORM0 eng INLINEFORM1 r e a l
The artificial token is treated simply as an element of the grapheme sequence  This is similar to the approach taken by johnson2016google in their zero-shot NMT system  However, their source-side artificial tokens identify the target language, whereas ours identify the source language  An alternative approach, used by ostling2017continuous, would be to concatenate a language embedding to the input at each time step  They do not evaluate their approach on grapheme-to-phoneme conversion 
Data
In order to train a neural g2p system, one needs a large quantity of pronunciation data  A standard dataset for g2p is the Carnegie Mellon Pronouncing Dictionary BIBREF12   However, that is a monolingual English resource, so it is unsuitable for our multilingual task  Instead, we use the multilingual pronunciation corpus collected by deri2016grapheme for all experiments  This corpus consists of spelling–pronunciation pairs extracted from Wiktionary  It is already partitioned into training and test sets  Corpus statistics are presented in Table TABREF10  
In addition to the raw IPA transcriptions extracted from Wiktionary, the corpus provides an automatically cleaned version of transcriptions  Cleaning is a necessary step because web-scraped data is often noisy and may be transcribed at an inconsistent level of detail  The data cleaning used here attempts to make the transcriptions consistent with the phonemic inventories used in Phoible BIBREF4   When a transcription contains a phoneme that is not in its language's inventory in Phoible, that phoneme is replaced by the phoneme with the most similar articulatory features that is in the language's inventory  Sometimes this cleaning algorithm works well: in the German examples in Table TABREF11 , the raw German symbols and are both converted to   This is useful because the in Ansbach and the in Kaninchen are instances of the same phoneme, so their phonemic representations should use the same symbol  However, the cleaning algorithm can also have negative effects on the data quality  For example, the phoneme is not present in the Phoible inventory for German, but it is used in several German transcriptions in the corpus  The cleaning algorithm converts to in all German transcriptions, whereas would be a more reasonable guess  The cleaning algorithm also removes most suprasegmentals, even though these are often an important part of a language's phonology  Developing a more sophisticated procedure for cleaning pronunciation data is a direction for future work, but in this paper we use the corpus's provided cleaned transcriptions in order to ease comparison to previous results 
Experiments
We present experiments with two versions of our sequence-to-sequence model  LangID prepends each training, validation, and test sample with an artificial token identifying the language of the sample  NoLangID omits this token  LangID and NoLangID have identical structure otherwise  To translate the test corpus, we used a beam width of 100  Although this is an unusually wide beam and had negligible performance effects, it was necessary to compute our error metrics 
Evaluation
We use the following three evaluation metrics:
Phoneme Error Rate (PER) is the Levenshtein distance between the predicted phoneme sequences and the gold standard phoneme sequences, divided by the length of the gold standard phoneme sequences 
Word Error Rate (WER) is the percentage of words in which the predicted phoneme sequence does not exactly match the gold standard phoneme sequence 
Word Error Rate 100 (WER 100) is the percentage of words in the test set for which the correct guess is not in the first 100 guesses of the system 
In system evaluations, WER, WER 100, and PER numbers presented for multiple languages are averaged, weighting each language equally BIBREF13  
It would be interesting to compute error metrics that incorporate phoneme similarity, such as those proposed by hixon2011phonemic  PER weights all phoneme errors the same, even though some errors are more harmful than others: and are usually contrastive, whereas and almost never are  Such statistics would be especially interesting for evaluating a multilingual system, because different languages often map the same grapheme to phonemes that are only subtly different from each other  However, these statistics have not been widely reported for other g2p systems, so we omit them here 
Baseline
Results on LangID and NoLangID are compared to the system presented by deri2016grapheme, which is identified in our results as wFST  Their results can be divided into two parts:
High resource results, computed with wFSTs trained on a combination of Wiktionary pronunciation data and g2p rules extracted from Wikipedia IPA Help pages  They report high resource results for 85 languages 
Adapted results, where they apply various mapping strategies in order to adapt high resource models to other languages  The final adapted results they reported include most of the 85 languages with high resource results, as well as the various languages they were able to adapt them for, for a total of 229 languages  This test set omits 23 of the high resource languages that are written in unique scripts or for which language distance metrics could not be computed 
Training
We train the LangID and NoLangID versions of our model each on three subsets of the Wiktionary data:
LangID-High and NoLangID-High: Trained on data from the 85 languages for which BIBREF13 used non-adapted wFST models 
LangID-Adapted and NoLangID-Adapted: Trained on data from any of the 229 languages for which they built adapted models  Because many of these languages had no training data at all, the model is actually only trained on data in 157 languages  As is noted above, the Adapted set omits 23 languages which are in the High test set 
LangID-All and NoLangID-All: Trained on data in all 311 languages in the Wiktionary training corpus 
In order to ease comparison to Deri and Knight's system, we limited our use of the training corpus to 10,000 words per language  We set aside 10 percent of the data in each language for validation, so the maximum number of training words for any language is 9000 for our systems 
Adapted Results
On the 229 languages for which deri2016grapheme presented their final results, the LangID version of our system outperforms the baseline by a wide margin  The best performance came with the version of our model that was trained on data in all available languages, not just the languages it was tested on  Using a language ID token improves results considerably, but even NoLangID beats the baseline in WER and WER 100  Full results are presented in Table TABREF24  
High Resource Results
Having shown that our model exceeds the performance of the wFST-adaptation approach, we next compare it to the baseline models for just high resource languages  The wFST models here are purely monolingual – they do not use data adaptation because there is sufficient training data for each of them  Full results are presented in Table TABREF26   We omit models trained on the Adapted languages because they were not trained on high resource languages with unique writing systems, such as Georgian and Greek, and consequently performed very poorly on them 
In contrast to the larger-scale Adapted results, in the High Resource experiments none of the sequence-to-sequence approaches equal the performance of the wFST model in WER and PER, although LangID-High does come close  The LangID models do beat wFST in WER 100  A possible explanation is that a monolingual wFST model will never generate phonemes that are not part of the language's inventory  A multilingual model, on the other hand, could potentially generate phonemes from the inventories of any language it has been trained on 
Even if LangID-High does not present a more accurate result, it does present a more compact one: LangID-High is 15 4 MB, while the combined wFST high resource models are 197 5 MB 
Results on Unseen Languages
Finally, we report our models' results on unseen languages in Table TABREF28   The unseen languages are any that are present in the test corpus but absent from the training data  Deri and Knight did not report results specifically on these languages  Although the NoLangID models sometimes do better on WER 100, even here the LangID models have a slight advantage in WER and PER  This is somewhat surprising because the LangID models have not learned embeddings for the language ID tokens of unseen languages  Perhaps negative associations are also being learned, driving the model towards predicting more common pronunciations for unseen languages 
Language ID Tokens
Adding a language ID token always improves results in cases where an embedding has been learned for that token  The power of these embeddings is demonstrated by what happens when one feeds the same input word to the model with different language tokens, as is seen in Table TABREF30   Impressively, this even works when the source sequence is in the wrong script for the language, as is seen in the entry for Arabic 
Language Embeddings
Because these language ID tokens are so useful, it would be good if they could be effectively estimated for unseen languages  ostling2017continuous found that the language vectors their models learned correlated well to genetic relationships, so it would be interesting to see if the embeddings our source encoder learned for the language ID tokens showed anything similar  In a few cases they do (the languages closest to German in the vector space are Luxembourgish, Bavarian, and Yiddish, all close relatives)  However, for the most part the structure of these vectors is not interpretable  Therefore, it would be difficult to estimate the embedding for an unseen language, or to “borrow” the language ID token of a similar language  A more promising way forward is to find a model that uses an externally constructed typological representation of the language 
Phoneme Embeddings
In contrast to the language embeddings, the phoneme embeddings appear to show many regularities (see Table TABREF33 )  This is a sign that our multilingual model learns similar embeddings for phonemes that are written with the same grapheme in different languages  These phonemes tend to be phonetically similar to each other 
Perhaps the structure of the phoneme embedding space is what leads to our models' very good performance on WER 100  Even when the model's first predicted pronunciation is not correct, it tends to assign more probability mass to guesses that are more similar to the correct one  Applying some sort of filtering or reranking of the system output might therefore lead to better performance 
Future Work
Because the language ID token is so beneficial to performance, it would be very interesting to find ways to extend a similar benefit to unseen languages  One possible way to do so is with tokens that identify something other than the language, such as typological features about the language's phonemic inventory  This could enable better sharing of resources among languages  Such typological knowledge is readily available in databases like Phoible and WALS for a wide variety of languages  It would be interesting to explore if any of these features is a good predictor of a language's orthographic rules 
It would also be interesting to apply the artificial token approach to other problems besides multilingual g2p  One closely related application is monolingual English g2p  Some of the ambiguity of English spelling is due to the wide variety of loanwords in the language, many of which have unassimilated spellings  Knowing the origins of these loanwords could provide a useful hint for figuring out their pronunciations  The etymology of a word could be tagged in an analogous way to how language ID is tagged in multilingual g2p 
Context 2: 10pt
1 10pt
[ Characterizing Political Fake News in Twitter by its Meta-DataJulio Amador Díaz LópezAxel Oehmichen Miguel Molina-Solana( j amador, axelfrancois oehmichen11, mmolinas@imperial ac uk ) Imperial College London This article presents a preliminary approach towards characterizing political fake news on Twitter through the analysis of their meta-data  In particular, we focus on more than 1 5M tweets collected on the day of the election of Donald Trump as 45th president of the United States of America  We use the meta-data embedded within those tweets in order to look for differences between tweets containing fake news and tweets not containing them  Specifically, we perform our analysis only on tweets that went viral, by studying proxies for users' exposure to the tweets, by characterizing accounts spreading fake news, and by looking at their polarization  We found significant differences on the distribution of followers, the number of URLs on tweets, and the verification of the users 
]
Introduction
While fake news, understood as deliberately misleading pieces of information, have existed since long ago (e g  it is not unusual to receive news falsely claiming the death of a celebrity), the term reached the mainstream, particularly so in politics, during the 2016 presidential election in the United States BIBREF0   Since then, governments and corporations alike (e g  Google BIBREF1 and Facebook BIBREF2 ) have begun efforts to tackle fake news as they can affect political decisions BIBREF3   Yet, the ability to define, identify and stop fake news from spreading is limited 
Since the Obama campaign in 2008, social media has been pervasive in the political arena in the United States  Studies report that up to 62% of American adults receive their news from social media BIBREF4   The wide use of platforms such as Twitter and Facebook has facilitated the diffusion of fake news by simplifying the process of receiving content with no significant third party filtering, fact-checking or editorial judgement  Such characteristics make these platforms suitable means for sharing news that, disguised as legit ones, try to confuse readers 
Such use and their prominent rise has been confirmed by Craig Silverman, a Canadian journalist who is a prominent figure on fake news BIBREF5 : “In the final three months of the US presidential campaign, the top-performing fake election news stories on Facebook generated more engagement than the top stories from major news outlet” 
Our current research hence departs from the assumption that social media is a conduit for fake news and asks the question of whether fake news (as spam was some years ago) can be identified, modelled and eventually blocked  In order to do so, we use a sample of more that 1 5M tweets collected on November 8th 2016 —election day in the United States— with the goal of identifying features that tweets containing fake news are likely to have  As such, our paper aims to provide a preliminary characterization of fake news in Twitter by looking into meta-data embedded in tweets  Considering meta-data as a relevant factor of analysis is in line with findings reported by Morris et al  BIBREF6   We argue that understanding differences between tweets containing fake news and regular tweets will allow researchers to design mechanisms to block fake news in Twitter 
Specifically, our goals are: 1) compare the characteristics of tweets labelled as containing fake news to tweets labelled as not containing them, 2) characterize, through their meta-data, viral tweets containing fake news and the accounts from which they originated, and 3) determine the extent to which tweets containing fake news expressed polarized political views 
For our study, we used the number of retweets to single-out those that went viral within our sample  Tweets within that subset (viral tweets hereafter) are varied and relate to different topics  We consider that a tweet contains fake news if its text falls within any of the following categories described by Rubin et al  BIBREF7 (see next section for the details of such categories): serious fabrication, large-scale hoaxes, jokes taken at face value, slanted reporting of real facts and stories where the truth is contentious  The dataset BIBREF8 , manually labelled by an expert, has been publicly released and is available to researchers and interested parties 
From our results, the following main observations can be made:
Our findings resonate with similar work done on fake news such as the one from Allcot and Gentzkow BIBREF9   Therefore, even if our study is a preliminary attempt at characterizing fake news on Twitter using only their meta-data, our results provide external validity to previous research  Moreover, our work not only stresses the importance of using meta-data, but also underscores which parameters may be useful to identify fake news on Twitter 
The rest of the paper is organized as follows  The next section briefly discusses where this work is located within the literature on fake news and contextualizes the type of fake news we are studying  Then, we present our hypotheses, the data, and the methodology we follow  Finally, we present our findings, conclusions of this study, and future lines of work 
Defining Fake news
Our research is connected to different strands of academic knowledge related to the phenomenon of fake news  In relation to Computer Science, a recent survey by Conroy and colleagues BIBREF10 identifies two popular approaches to single-out fake news  On the one hand, the authors pointed to linguistic approaches consisting in using text, its linguistic characteristics and machine learning techniques to automatically flag fake news  On the other, these researchers underscored the use of network approaches, which make use of network characteristics and meta-data, to identify fake news 
With respect to social sciences, efforts from psychology, political science and sociology, have been dedicated to understand why people consume and/or believe misinformation BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14   Most of these studies consistently reported that psychological biases such as priming effects and confirmation bias play an important role in people ability to discern misinformation 
In relation to the production and distribution of fake news, a recent paper in the field of Economics BIBREF9 found that most fake news sites use names that resemble those of legitimate organizations, and that sites supplying fake news tend to be short-lived  These authors also noticed that fake news items are more likely shared than legitimate articles coming from trusted sources, and they tend to exhibit a larger level of polarization 
The conceptual issue of how to define fake news is a serious and unresolved issue  As the focus of our work is not attempting to offer light on this, we will rely on work by other authors to describe what we consider as fake news  In particular, we use the categorization provided by Rubin et al  BIBREF7   The five categories they described, together with illustrative examples from our dataset, are as follows:
Research Hypotheses
Previous works on the area (presented in the section above) suggest that there may be important determinants for the adoption and diffusion of fake news  Our hypotheses builds on them and identifies three important dimensions that may help distinguishing fake news from legit information:
Taking those three dimensions into account, we propose the following hypotheses about the features that we believe can help to identify tweets containing fake news from those not containing them  They will be later tested over our collected dataset 
Exposure 
Characterization 
Polarization 
Data and Methodology
For this study, we collected publicly available tweets using Twitter's public API  Given the nature of the data, it is important to emphasize that such tweets are subject to Twitter's terms and conditions which indicate that users consent to the collection, transfer, manipulation, storage, and disclosure of data  Therefore, we do not expect ethical, legal, or social implications from the usage of the tweets  Our data was collected using search terms related to the presidential election held in the United States on November 8th 2016  Particularly, we queried Twitter's streaming API, more precisely the filter endpoint of the streaming API, using the following hashtags and user handles: #MyVote2016, #ElectionDay, #electionnight, @realDonaldTrump and @HillaryClinton  The data collection ran for just one day (Nov 8th 2016) 
One straightforward way of sharing information on Twitter is by using the retweet functionality, which enables a user to share a exact copy of a tweet with his followers  Among the reasons for retweeting, Body et al  BIBREF15 reported the will to: 1) spread tweets to a new audience, 2) to show one’s role as a listener, and 3) to agree with someone or validate the thoughts of others  As indicated, our initial interest is to characterize tweets containing fake news that went viral (as they are the most harmful ones, as they reach a wider audience), and understand how it differs from other viral tweets (that do not contain fake news)  For our study, we consider that a tweet went viral if it was retweeted more than 1000 times 
Once we have the dataset of viral tweets, we eliminated duplicates (some of the tweets were collected several times because they had several handles) and an expert manually inspected the text field within the tweets to label them as containing fake news, or not containing them (according to the characterization presented before)  This annotated dataset BIBREF8 is publicly available and can be freely reused 
Finally, we use the following fields within tweets (from the ones returned by Twitter's API) to compare their distributions and look for differences between viral tweets containing fake news and viral tweets not containing fake news:
In the following section, we provide graphical descriptions of the distribution of each of the identified attributes for the two sets of tweets (those labelled as containing fake news and those labelled as not containing them)  Where appropriate, we normalized and/or took logarithms of the data for better representation
Context 3: Introduction
Deep Learning approaches have achieved impressive results on various NLP tasks BIBREF0 , BIBREF1 , BIBREF2 and have become the de facto approach for any NLP task  However, these deep learning techniques have found to be less effective for low-resource languages when the available training data is very less BIBREF3   Recently, several approaches like Multi-task learning BIBREF4 , multilingual learning BIBREF5 , semi-supervised learning BIBREF2 , BIBREF6 and transfer learning BIBREF7 , BIBREF3 have been explored by the deep learning community to overcome data sparsity in low-resource languages  Transfer learning trains a model for a parent task and fine-tunes the learned parent model weights (features) for a related child task BIBREF7 , BIBREF8   This effectively reduces the requirement on training data for the child task as the model would have learned relevant features from the parent task data thereby, improving the performance on the child task 
Transfer learning has also been explored in the multilingual Neural Machine Translation BIBREF3 , BIBREF9 , BIBREF10   The goal is to improve the NMT performance on the source to target language pair (child task) using an assisting source language (assisting to target translation is the parent task)  Here, the parent model is trained on the assisting and target language parallel corpus and the trained weights are used to initialize the child model  The child model can now be fine-tuned on the source-target language pairs, if parallel corpus is available  The divergence between the source and the assisting language can adversely impact the benefits obtained from transfer learning  Multiple studies have shown that transfer learning works best when the languages are related BIBREF3 , BIBREF10 , BIBREF9   Several studies have tried to address lexical divergence between the source and the target languages BIBREF10 , BIBREF11 , BIBREF12   However, the effect of word order divergence and its mitigation has not been explored  In a practical setting, it is not uncommon to have source and assisting languages with different word order  For instance, it is possible to find parallel corpora between English and some Indian languages, but very little parallel corpora between Indian languages  Hence, it is natural to use English as an assisting language for inter-Indian language translation 
To see how word order divergence can be detrimental, let us consider the case of the standard RNN (Bi-LSTM) encoder-attention-decoder architecture BIBREF13   The encoder generates contextual representations (annotation vectors) for each source word, which are used by the attention network to match the source words to the current decoder state  The contextual representation is word-order dependent  Hence, if the assisting and the source languages do not have similar word order the generated contextual representations will not be consistent  The attention network (and hence the decoder) sees different contextual representations for similar words in parallel sentences across different languages  This makes it difficult to transfer knowledge learned from the assisting language to the source language 
We illustrate this by visualizing the contextual representations generated by the encoder of an English to Hindi NMT system for two versions of the English input: (a) original word order (SVO) (b) word order of the source language (SOV, for Bengali)  Figure FIGREF1 shows that the encoder representations obtained are very different  The attention network and the decoder now have to work with very different representations  Note that the plot below does not take into account further lexical and other divergences between source and assisting languages, since we demonstrated word order divergence with the same language on the source side 
To address this word order divergence, we propose to pre-order the assisting language sentences to match the word order of the source language  We consider an extremely resource constrained scenario, where we do not have any parallel corpus for the child task  We are limited to a bilingual dictionary for transfer information from the assisting to the source language  From our experiments, we show that there is a significant increase in the translation accuracy for the unseen source-target language pair 
Addressing Lexical Divergence
BIBREF3 explored transfer learning for NMT on low-resource languages  They studied the influence of language divergence between languages chosen for training the parent and child model, and showed that choosing similar languages for training the parent and child model leads to better improvements from transfer learning  A limitation of BIBREF3 approach is that they ignore the lexical similarity between languages and also the source language embeddings are randomly initialized  BIBREF10 , BIBREF11 , BIBREF12 take advantage of lexical similarity between languages in their work  BIBREF10 proposed to use Byte-Pair Encoding (BPE) to represent the sentences in both the parent and the child language to overcome the above limitation  They show using BPE benefits transfer learning especially when the involved languages are closely-related agglutinative languages  Similarly, BIBREF11 utilize lexical similarity between the source and assisting languages by training a character-level NMT system  BIBREF12 address lexical divergence by using bilingual embeddings and mixture of universal token embeddings

ground_truth: the Carnegie Mellon Pronouncing Dictionary BIBREF12, the multilingual pronunciation corpus collected by deri2016grapheme , ranscriptions extracted from Wiktionary

multilingual pronunciation corpus collected by deri2016grapheme

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What is their definition of tweets going viral?
Context 2:  To gain a better understanding of the significance of those differences, we use the Kolmogorov-Smirnov test with the null hypothesis that both distributions are equal 
Results
The sample collected consisted on 1 785 855 tweets published by 848 196 different users  Within our sample, we identified 1327 tweets that went viral (retweeted more than 1000 times by the 8th of November 2016) produced by 643 users  Such small subset of viral tweets were retweeted on 290 841 occasions in the observed time-window 
The 1327 `viral' tweets were manually annotated as containing fake news or not  The annotation was carried out by a single person in order to obtain a consistent annotation throughout the dataset  Out of those 1327 tweets, we identified 136 as potentially containing fake news (according to the categories previously described), and the rest were classified as `non containing fake news'  Note that the categorization is far from being perfect given the ambiguity of fake news themselves and human judgement involved in the process of categorization  Because of this, we do not claim that this dataset can be considered a ground truth 
The following results detail characteristics of these tweets along the previously mentioned dimensions  Table TABREF23 reports the actual differences (together with their associated p-values) of the distributions of viral tweets containing fake news and viral tweets not containing them for every variable considered 
Exposure
Figure FIGREF24 shows that, in contrast to other kinds of viral tweets, those containing fake news were created more recently  As such, Twitter users were exposed to fake news related to the election for a shorter period of time 
However, in terms of retweets, Figure FIGREF25 shows no apparent difference between containing fake news or not containing them  That is confirmed by the Kolmogorov-Smirnoff test, which does not discard the hypothesis that the associated distributions are equal 
In relation to the number of favourites, users that generated at least a viral tweet containing fake news appear to have, on average, less favourites than users that do not generate them  Figure FIGREF26 shows the distribution of favourites  Despite the apparent visual differences, the difference are not statistically significant 
Finally, the number of hashtags used in viral fake news appears to be larger than those in other viral tweets  Figure FIGREF27 shows the density distribution of the number of hashtags used  However, once again, we were not able to find any statistical difference between the average number of hashtags in a viral tweet and the average number of hashtags in viral fake news 
Characterization
We found that 82 users within our sample were spreading fake news (i e  they produced at least one tweet which was labelled as fake news)  Out of those, 34 had verified accounts, and the rest were unverified  From the 48 unverified accounts, 6 have been suspended by Twitter at the date of writing, 3 tried to imitate legitimate accounts of others, and 4 accounts have been already deleted  Figure FIGREF28 shows the proportion of verified accounts to unverified accounts for viral tweets (containing fake news vs  not containing fake news)  From the chart, it is clear that there is a higher chance of fake news coming from unverified accounts 
Turning to friends, accounts distributing fake news appear to have, on average, the same number of friends than those distributing tweets with no fake news  However, the density distribution of friends from the accounts (Figure FIGREF29 ) shows that there is indeed a statistically significant difference in their distributions 
If we take into consideration the number of followers, accounts generating viral tweets with fake news do have a very different distribution on this dimension, compared to those accounts generating viral tweets with no fake news (see Figure FIGREF30 )  In fact, such differences are statistically significant 
A useful representation for friends and followers is the ratio between friends/followers  Figures FIGREF31 and FIGREF32 show this representation  Notice that accounts spreading viral tweets with fake news have, on average, a larger ratio of friends/followers  The distribution of those accounts not generating fake news is more evenly distributed 
With respect to the number of mentions, Figure FIGREF33 shows that viral tweets labelled as containing fake news appear to use mentions to other users less frequently than viral tweets not containing fake news  In other words, tweets containing fake news mostly contain 1 mention, whereas other tweets tend to have two)  Such differences are statistically significant 
The analysis (Figure FIGREF34 ) of the presence of media in the tweets in our dataset shows that tweets labelled as not containing fake news appear to present more media elements than those labelled as fake news  However, the difference is not statistically significant 
On the other hand, Figure FIGREF35 shows that viral tweets containing fake news appear to include more URLs to other sites than viral tweets that do not contain fake news  In fact, the difference between the two distributions is statistically significant (assuming INLINEFORM0 ) 
Polarization
Finally, manual inspection of the text field of those viral tweets labelled as containing fake news shows that 117 of such tweets expressed support for Donald Trump, while only 8 supported Hillary Clinton  The remaining tweets contained fake news related to other topics, not expressing support for any of the candidates 
Discussion
As a summary, and constrained by our existing dataset, we made the following observations regarding differences between viral tweets labelled as containing fake news and viral tweets labelled as not containing them:
These findings (related to our initial hypothesis in Table TABREF44 ) clearly suggest that there are specific pieces of meta-data about tweets that may allow the identification of fake news  One such parameter is the time of exposure  Viral tweets containing fake news are shorter-lived than those containing other type of content  This notion seems to resonate with our findings showing that a number of accounts spreading fake news have already been deleted or suspended by Twitter by the time of writing  If one considers that researchers using different data have found similar results BIBREF9 , it appears that the lifetime of accounts, together with the age of the questioned viral content could be useful to identify fake news  In the light of this finding, accounts newly created should probably put under higher scrutiny than older ones  This in fact, would be a nice a-priori bias for a Bayesian classifier 
Accounts spreading fake news appear to have a larger proportion of friends/followers (i e  they have, on average, the same number of friends but a smaller number of followers) than those spreading viral content only  Together with the fact that, on average, tweets containing fake news have more URLs than those spreading viral content, it is possible to hypothesize that, both, the ratio of friends/followers of the account producing a viral tweet and number of URLs contained in such a tweet could be useful to single-out fake news in Twitter  Not only that, but our finding related to the number of URLs is in line with intuitions behind the incentives to create fake news commonly found in the literature BIBREF9 (in particular that of obtaining revenue through click-through advertising) 
Finally, it is interesting to notice that the content of viral fake news was highly polarized  This finding is also in line with those of Alcott et al  BIBREF9   This feature suggests that textual sentiment analysis of the content of tweets (as most researchers do), together with the above mentioned parameters from meta-data, may prove useful for identifying fake news 
Conclusions
With the election of Donald Trump as President of the United States, the concept of fake news has become a broadly-known phenomenon that is getting tremendous attention from governments and media companies  We have presented a preliminary study on the meta-data of a publicly available dataset of tweets that became viral during the day of the 2016 US presidential election  Our aim is to advance the understanding of which features might be characteristic of viral tweets containing fake news in comparison with viral tweets without fake news 
We believe that the only way to automatically identify those deceitful tweets (i e  containing fake news) is by actually understanding and modelling them  Only then, the automation of the processes of tagging and blocking these tweets can be successfully performed  In the same way that spam was fought, we anticipate fake news will suffer a similar evolution, with social platforms implementing tools to deal with them  With most works so far focusing on the actual content of the tweets, ours is a novel attempt from a different, but also complementary, angle 
Within the used dataset, we found there are differences around exposure, characteristics of accounts spreading fake news and the tone of the content  Those findings suggest that it is indeed possible to model and automatically detect fake news  We plan to replicate and validate our experiments in an extended sample of tweets (until 4 months after the US election), and tests the predictive power of the features we found relevant within our sample 
Author Disclosure Statement
No competing financial interest exist 
Context 3:  One of the languages' vocabulary, usually English vocabulary is considered as universal tokens and every word in the other languages is represented as a mixture of universal tokens  They show results on extremely low-resource languages 
Addressing Word Order Divergence
To the best of our knowledge, no work has addressed word order divergence in transfer learning for multilingual NMT  However, some work exists for other NLP tasks that could potentially address word order  For Named Entity Recognition (NER), BIBREF14 use a self-attention layer after the Bi-LSTM layer to address word-order divergence for Named Entity Recognition (NER) task  The approach does not show any significant improvements over multiple languages  A possible reason is that the divergence has to be addressed before/during construction of the contextual embeddings in the Bi-LSTM layer, and the subsequent self-attention layer does not address word-order divergence  BIBREF15 use adversarial training for cross-lingual question-question similarity ranking in community question answering  The adversarial training tries to force the encoder representations of similar sentences from different input languages to have similar representations 
Use of Pre-ordering
Pre-ordering the source language sentences to match the target language word order has been useful in addressing word-order divergence for Phrase-Based SMT BIBREF16 , BIBREF17 , BIBREF18 , BIBREF19   Recently, BIBREF20 proposed a way to measure and reduce the divergence between the source and target languages based on morphological and syntactic properties, also termed as anisomorphism  They demonstrated that by reducing the anisomorphism between the source and target languages, consistent improvements in NMT performance were obtained  The NMT system used additional features like word forms, POS tags and dependency relations in addition to parallel corpora  On the other hand, BIBREF21 observed a drop in performance due to pre-ordering for NMT  Unlike BIBREF20 , the NMT system was trained on pre-ordered sentences and no additional features were provided to the system  Note that all these works address source-target divergence, not divergence between source languages in multilingual NMT 
Proposed Solution
Consider the task of translating from an extremely low-resource language (source) to a target language  The parallel corpus between the two languages if available may be too small to train a NMT model  Similar to existing works BIBREF3 , BIBREF10 , BIBREF12 , we use transfer learning to overcome data sparsity and train a NMT model between the source and the target languages  Specifically, the NMT model (parent model) is trained on the assisting language and target language pairs  We choose English as the assisting language in all our experiments  In our resource-scarce scenario, we have no parallel corpus for the child task  Hence, at test time, the source language sentence is translated using the parent model after performing a word-by-word translation into the assisting language 
Since the source language and the assisting language (English) have different word order, we hypothesize that it leads to inconsistencies in the contextual representations generated by the encoder for the two languages  In this paper, we propose to pre-order English sentences (assisting language sentences) to match the word-order of the source language and train the parent model on this pre-ordered corpus  In our experiments, we look at scenarios where the assisting language has SVO word order and the source language has SOV word order 
For instance, consider the English sentence Anurag will meet Thakur  One of the pre-ordering rule swaps the position of the noun phrase followed by a transitive verb with the transitive verb  The original and the resulting re-ordered parse tree will be as shown in the Table TABREF5   Applying this reordering rule to the above sentence Anurag will meet Thakur will yield the reordered sentence Anurag Thakur will meet  Additionally, the Table TABREF5 shows the parse trees for the above sentence with and without pre-ordering 
Pre-ordering should also be beneficial for other word order divergence scenarios (e g , SOV to SVO), but we leave verification of these additional scenarios for future work 
Experimental Setup
In this section, we describe the languages experimented with, datasets used, the network hyper-parameters used in our experiments 
Languages
We experimented with English INLINEFORM0 Hindi translation as the parent task  English is the assisting source language  Bengali, Gujarati, Marathi, Malayalam and Tamil are the primary source languages, and translation from these to Hindi constitute the child tasks

ground_truth: Viral tweets are the ones that are retweeted more than 1000 times

those that contain a high number of retweets

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How do they match words before reordering them?
Context 3:  Hindi, Bengali, Gujarati and Marathi are Indo-Aryan languages, while Malayalam and Tamil are Dravidian languages  All these languages have a canonical SOV word order 
Datasets
For training English-Hindi NMT systems, we use the IITB English-Hindi parallel corpus BIBREF22 ( INLINEFORM0 sentences from the training set) and the ILCI English-Hindi parallel corpus ( INLINEFORM1 sentences)  The ILCI (Indian Language Corpora Initiative) multilingual parallel corpus BIBREF23 spans multiple Indian languages from the health and tourism domains  We use the 520-sentence dev-set of the IITB parallel corpus for validation  For each child task, we use INLINEFORM2 sentences from ILCI corpus as the test set 
Network
We use OpenNMT-Torch BIBREF24 to train the NMT system  We use the standard sequence-to-sequence architecture with attention BIBREF13   We use an encoder which contains two layers of bidirectional LSTMs with 500 neurons each  The decoder contains two LSTM layers with 500 neurons each  Input feeding approach BIBREF1 is used where the previous attention hidden state is fed as input to the decoder LSTM  We use a mini-batch of size 50 and use a dropout layer  We begin with an initial learning rate of INLINEFORM0 and decay the learning rate by a factor of INLINEFORM1 when the perplexity on validation set increases  The training is stopped when the learning rate falls below INLINEFORM2 or number of epochs=22  The English input is initialized with pre-trained embeddings trained using fastText BIBREF25  
English vocabulary consists of INLINEFORM0 tokens appearing at least 2 times in the English training corpus  For constructing the Hindi vocabulary we considered only those tokens appearing at least 5 times in the training split resulting in a vocabulary size of INLINEFORM1 tokens  For representing English and other source languages into a common space, we translate each word in the source language into English using a bilingual dictionary (Google Translate word translation in our case)  In an end-to-end solution, it would have been ideal to use bilingual embeddings or obtain word-by-word translations via bilingual embeddings BIBREF14   But, the quality of publicly available bilingual embeddings for English-Indian languages is very low for obtaining good-quality, bilingual representations BIBREF26 , BIBREF27   We also found that these embeddings were not useful for transfer learning 
We use the CFILT-preorder system for reordering English sentences to match the Indian language word order  It contains two re-ordering systems: (1) generic rules that apply to all Indian languages BIBREF17 , and (2) hindi-tuned rules which improve the generic rules by incorporating improvements found through an error analysis of English-Hindi reordering BIBREF28   These Hindi-tuned rules have been found to improve reordering for many English to Indian language pairs BIBREF29  
Results
In this section, we describe the results from our experiments on NMT task  We report the results on X-Hindi pair, where X is one of Bengali, Gujarati, Marathi, Tamil, and Malayalam  The results are presented in the Table TABREF6   We report BLEU scores and LeBLEU scores BIBREF30   We observe that both the pre-ordering configurations significantly improve the BLEU scores over the baseline scores  We observe larger gains when generic pre-ordering rules are used compared to the Hindi-tuned pre-ordering rules 
These results support our hypothesis that word-order divergence can limit the benefits of multilingual translation  Reducing the word order divergence can improve translation in extremely low-resource scenarios 
An analysis of the outputs revealed that pre-ordering significantly reducing the number of UNK tokens (placeholder for unknown words) in the test output (Table TABREF14 )  We hypothesize that due to word order divergence between English and Indian languages, the encoder representation generated is not consistent leading to decoder generating unknown words  However, the pre-ordered models generate better contextual representations leading to less number of unknown tokens and better translation which is also reflected in the BLEU scores 
Conclusion
In this paper, we show that handling word-order divergence between source and assisting languages is crucial for the success of multilingual NMT in an extremely low-resource setting  We show that pre-ordering the assisting language to match the word order of the source language significantly improves translation quality in an extremely low-resource setting  While the current work focused on Indian languages, we would like to validate the hypothesis on a more diverse set of languages 

ground_truth: Unanswerable

CFILT-preorder system