prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. what datasets did they use?
Context 1: Introduction
Accurate grapheme-to-phoneme conversion (g2p) is important for any application that depends on the sometimes inconsistent relationship between spoken and written language  Most prominently, this includes text-to-speech and automatic speech recognition  Most work on g2p has focused on a few languages for which extensive pronunciation data is available BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3   Most languages lack these resources  However, a low resource language's writing system is likely to be similar to the writing systems of languages that do have sufficient pronunciation data  Therefore g2p may be possible for low resource languages if this high resource data can be properly utilized 
We attempt to leverage high resource data by treating g2p as a multisource neural machine translation (NMT) problem  The source sequences for our system are words in the standard orthography in any language  The target sequences are the corresponding representation in the International Phonetic Alphabet (IPA)  Our results show that the parameters learned by the shared encoder–decoder are able to exploit the orthographic and phonemic similarities between the various languages in our data 
Low Resource g2p
Our approach is similar in goal to deri2016grapheme's model for adapting high resource g2p models for low resource languages  They trained weighted finite state transducer (wFST) models on a variety of high resource languages, then transferred those models to low resource languages, using a language distance metric to choose which high resource models to use and a phoneme distance metric to map the high resource language's phonemes to the low resource language's phoneme inventory  These distance metrics are computed based on data from Phoible BIBREF4 and URIEL BIBREF5  
Other low resource g2p systems have used a strategy of combining multiple models  schlippe2014combining trained several data-driven g2p systems on varying quantities of monolingual data and combined their outputs with a phoneme-level voting scheme  This led to improvements over the best-performing single system for small quantities of data in some languages  jyothilow trained recurrent neural networks for small data sets and found that a version of their system that combined the neural network output with the output of the wFST-based Phonetisaurus system BIBREF1 did better than either system alone 
A different approach came from kim2012universal, who used supervised learning with an undirected graphical model to induce the grapheme–phoneme mappings for languages written in the Latin alphabet  Given a short text in a language, the model predicts the language's orthographic rules  To create phonemic context features from the short text, the model naïvely maps graphemes to IPA symbols written with the same character, and uses the features of these symbols to learn an approximation of the phonotactic constraints of the language  In their experiments, these phonotactic features proved to be more valuable than geographical and genetic features drawn from WALS BIBREF6  
Multilingual Neural NLP
In recent years, neural networks have emerged as a common way to use data from several languages in a single system  Google's zero-shot neural machine translation system BIBREF7 shares an encoder and decoder across all language pairs  In order to facilitate this multi-way translation, they prepend an artificial token to the beginning of each source sentence at both training and translation time  The token identifies what language the sentence should be translated to  This approach has three benefits: it is far more efficient than building a separate model for each language pair; it allows for translation between languages that share no parallel data; and it improves results on low-resource languages by allowing them to implicitly share parameters with high-resource languages  Our g2p system is inspired by this approach, although it differs in that there is only one target “language”, IPA, and the artificial tokens identify the language of the source instead of the language of the target 
Other work has also made use of multilingually-trained neural networks  Phoneme-level polyglot language models BIBREF8 train a single model on multiple languages and additionally condition on externally constructed typological data about the language  ostling2017continuous used a similar approach, in which a character-level neural language model is trained on a massively multilingual corpus  A language embedding vector is concatenated to the input at each time step  The language embeddings their system learned correlate closely to the genetic relationships between languages  However, neither of these models was applied to g2p 
Grapheme-to-Phoneme
g2p is the problem of converting the orthographic representation of a word into a phonemic representation  A phoneme is an abstract unit of sound which may have different realizations in different contexts  For example, the English phoneme has two phonetic realizations (or allophones):
English speakers without linguistic training often struggle to perceive any difference between these sounds  Writing systems usually do not distinguish between allophones: and are both written as INLINEFORM0 p INLINEFORM1 in English  The sounds are written differently in languages where they contrast, such as Hindi and Eastern Armenian 
Most writing systems in use today are glottographic, meaning that their symbols encode solely phonological information  But despite being glottographic, in few writing systems do graphemes correspond one-to-one with phonemes  There are cases in which multiple graphemes represent a single phoneme, as in the word the in English:
There are cases in which a single grapheme represents multiple phonemes, such as syllabaries, in which each symbol represents a syllable 
In many languages, there are silent letters, as in the word hora in Spanish:
There are more complicated correspondences, such as the silent e in English that affects the pronunciation of the previous vowel, as seen in the pair of words cape and cap 
It is possible for an orthographic system to have any or all of the above phenomena while remaining unambiguous  However, some orthographic systems contain ambiguities  English is well-known for its spelling ambiguities  Abjads, used for Arabic and Hebrew, do not give full representation to vowels 
Consequently, g2p is harder than simply replacing each grapheme symbol with a corresponding phoneme symbol  It is the problem of replacing a grapheme sequence INLINEFORM0
with a phoneme sequence INLINEFORM0
where the sequences are not necessarily of the same length  Data-driven g2p is therefore the problem of finding the phoneme sequence that maximizes the likelihood of the grapheme sequence: INLINEFORM0
Data-driven approaches are especially useful for problems in which the rules that govern them are complex and difficult to engineer by hand  g2p for languages with ambiguous orthographies is such a problem  Multilingual g2p, in which the various languages have similar but different and possibly contradictory spelling rules, can be seen as an extreme case of that  Therefore, a data-driven sequence-to-sequence model is a natural choice 
Encoder–Decoder Models
In order to find the best phoneme sequence, we use a neural encoder–decoder model with attention BIBREF9   The model consists of two main parts: the encoder compresses each source grapheme sequence INLINEFORM0 into a fixed-length vector  The decoder, conditioned on this fixed-length vector, generates the output phoneme sequence INLINEFORM1  
The encoder and decoder are both implemented as recurrent neural networks, which have the advantage of being able to process sequences of arbitrary length and use long histories efficiently  They are trained jointly to minimize cross-entropy on the training data  We had our best results when using a bidirectional encoder, which consists of two separate encoders which process the input in forward and reverse directions  We used long short-term memory units BIBREF10 for both the encoder and decoder  For the attention mechanism, we used the general global attention architecture described by luong2015effective 
We implemented all models with OpenNMT BIBREF11   Our hyperparameters, which we determined by experimentation, are listed in Table TABREF8  
Training Multilingual Models
Presenting pronunciation data in several languages to the network might create problems because different languages have different pronunciation patterns  For example, the string `real' is pronounced differently in English, German, Spanish, and Portuguese  We solve this problem by prepending each grapheme sequence with an artificial token consisting of the language's ISO 639-3 code enclosed in angle brackets  The English word `real', for example, would be presented to the system as
INLINEFORM0 eng INLINEFORM1 r e a l
The artificial token is treated simply as an element of the grapheme sequence  This is similar to the approach taken by johnson2016google in their zero-shot NMT system  However, their source-side artificial tokens identify the target language, whereas ours identify the source language  An alternative approach, used by ostling2017continuous, would be to concatenate a language embedding to the input at each time step  They do not evaluate their approach on grapheme-to-phoneme conversion 
Data
In order to train a neural g2p system, one needs a large quantity of pronunciation data  A standard dataset for g2p is the Carnegie Mellon Pronouncing Dictionary BIBREF12   However, that is a monolingual English resource, so it is unsuitable for our multilingual task  Instead, we use the multilingual pronunciation corpus collected by deri2016grapheme for all experiments  This corpus consists of spelling–pronunciation pairs extracted from Wiktionary  It is already partitioned into training and test sets  Corpus statistics are presented in Table TABREF10  
In addition to the raw IPA transcriptions extracted from Wiktionary, the corpus provides an automatically cleaned version of transcriptions  Cleaning is a necessary step because web-scraped data is often noisy and may be transcribed at an inconsistent level of detail  The data cleaning used here attempts to make the transcriptions consistent with the phonemic inventories used in Phoible BIBREF4   When a transcription contains a phoneme that is not in its language's inventory in Phoible, that phoneme is replaced by the phoneme with the most similar articulatory features that is in the language's inventory  Sometimes this cleaning algorithm works well: in the German examples in Table TABREF11 , the raw German symbols and are both converted to   This is useful because the in Ansbach and the in Kaninchen are instances of the same phoneme, so their phonemic representations should use the same symbol  However, the cleaning algorithm can also have negative effects on the data quality  For example, the phoneme is not present in the Phoible inventory for German, but it is used in several German transcriptions in the corpus  The cleaning algorithm converts to in all German transcriptions, whereas would be a more reasonable guess  The cleaning algorithm also removes most suprasegmentals, even though these are often an important part of a language's phonology  Developing a more sophisticated procedure for cleaning pronunciation data is a direction for future work, but in this paper we use the corpus's provided cleaned transcriptions in order to ease comparison to previous results 
Experiments
We present experiments with two versions of our sequence-to-sequence model  LangID prepends each training, validation, and test sample with an artificial token identifying the language of the sample  NoLangID omits this token  LangID and NoLangID have identical structure otherwise  To translate the test corpus, we used a beam width of 100  Although this is an unusually wide beam and had negligible performance effects, it was necessary to compute our error metrics 
Evaluation
We use the following three evaluation metrics:
Phoneme Error Rate (PER) is the Levenshtein distance between the predicted phoneme sequences and the gold standard phoneme sequences, divided by the length of the gold standard phoneme sequences 
Word Error Rate (WER) is the percentage of words in which the predicted phoneme sequence does not exactly match the gold standard phoneme sequence 
Word Error Rate 100 (WER 100) is the percentage of words in the test set for which the correct guess is not in the first 100 guesses of the system 
In system evaluations, WER, WER 100, and PER numbers presented for multiple languages are averaged, weighting each language equally BIBREF13  
It would be interesting to compute error metrics that incorporate phoneme similarity, such as those proposed by hixon2011phonemic  PER weights all phoneme errors the same, even though some errors are more harmful than others: and are usually contrastive, whereas and almost never are  Such statistics would be especially interesting for evaluating a multilingual system, because different languages often map the same grapheme to phonemes that are only subtly different from each other  However, these statistics have not been widely reported for other g2p systems, so we omit them here 
Baseline
Results on LangID and NoLangID are compared to the system presented by deri2016grapheme, which is identified in our results as wFST  Their results can be divided into two parts:
High resource results, computed with wFSTs trained on a combination of Wiktionary pronunciation data and g2p rules extracted from Wikipedia IPA Help pages  They report high resource results for 85 languages 
Adapted results, where they apply various mapping strategies in order to adapt high resource models to other languages  The final adapted results they reported include most of the 85 languages with high resource results, as well as the various languages they were able to adapt them for, for a total of 229 languages  This test set omits 23 of the high resource languages that are written in unique scripts or for which language distance metrics could not be computed 
Training
We train the LangID and NoLangID versions of our model each on three subsets of the Wiktionary data:
LangID-High and NoLangID-High: Trained on data from the 85 languages for which BIBREF13 used non-adapted wFST models 
LangID-Adapted and NoLangID-Adapted: Trained on data from any of the 229 languages for which they built adapted models  Because many of these languages had no training data at all, the model is actually only trained on data in 157 languages  As is noted above, the Adapted set omits 23 languages which are in the High test set 
LangID-All and NoLangID-All: Trained on data in all 311 languages in the Wiktionary training corpus 
In order to ease comparison to Deri and Knight's system, we limited our use of the training corpus to 10,000 words per language  We set aside 10 percent of the data in each language for validation, so the maximum number of training words for any language is 9000 for our systems 
Adapted Results
On the 229 languages for which deri2016grapheme presented their final results, the LangID version of our system outperforms the baseline by a wide margin  The best performance came with the version of our model that was trained on data in all available languages, not just the languages it was tested on  Using a language ID token improves results considerably, but even NoLangID beats the baseline in WER and WER 100  Full results are presented in Table TABREF24  
High Resource Results
Having shown that our model exceeds the performance of the wFST-adaptation approach, we next compare it to the baseline models for just high resource languages  The wFST models here are purely monolingual – they do not use data adaptation because there is sufficient training data for each of them  Full results are presented in Table TABREF26   We omit models trained on the Adapted languages because they were not trained on high resource languages with unique writing systems, such as Georgian and Greek, and consequently performed very poorly on them 
In contrast to the larger-scale Adapted results, in the High Resource experiments none of the sequence-to-sequence approaches equal the performance of the wFST model in WER and PER, although LangID-High does come close  The LangID models do beat wFST in WER 100  A possible explanation is that a monolingual wFST model will never generate phonemes that are not part of the language's inventory  A multilingual model, on the other hand, could potentially generate phonemes from the inventories of any language it has been trained on 
Even if LangID-High does not present a more accurate result, it does present a more compact one: LangID-High is 15 4 MB, while the combined wFST high resource models are 197 5 MB 
Results on Unseen Languages
Finally, we report our models' results on unseen languages in Table TABREF28   The unseen languages are any that are present in the test corpus but absent from the training data  Deri and Knight did not report results specifically on these languages  Although the NoLangID models sometimes do better on WER 100, even here the LangID models have a slight advantage in WER and PER  This is somewhat surprising because the LangID models have not learned embeddings for the language ID tokens of unseen languages  Perhaps negative associations are also being learned, driving the model towards predicting more common pronunciations for unseen languages 
Language ID Tokens
Adding a language ID token always improves results in cases where an embedding has been learned for that token  The power of these embeddings is demonstrated by what happens when one feeds the same input word to the model with different language tokens, as is seen in Table TABREF30   Impressively, this even works when the source sequence is in the wrong script for the language, as is seen in the entry for Arabic 
Language Embeddings
Because these language ID tokens are so useful, it would be good if they could be effectively estimated for unseen languages  ostling2017continuous found that the language vectors their models learned correlated well to genetic relationships, so it would be interesting to see if the embeddings our source encoder learned for the language ID tokens showed anything similar  In a few cases they do (the languages closest to German in the vector space are Luxembourgish, Bavarian, and Yiddish, all close relatives)  However, for the most part the structure of these vectors is not interpretable  Therefore, it would be difficult to estimate the embedding for an unseen language, or to “borrow” the language ID token of a similar language  A more promising way forward is to find a model that uses an externally constructed typological representation of the language 
Phoneme Embeddings
In contrast to the language embeddings, the phoneme embeddings appear to show many regularities (see Table TABREF33 )  This is a sign that our multilingual model learns similar embeddings for phonemes that are written with the same grapheme in different languages  These phonemes tend to be phonetically similar to each other 
Perhaps the structure of the phoneme embedding space is what leads to our models' very good performance on WER 100  Even when the model's first predicted pronunciation is not correct, it tends to assign more probability mass to guesses that are more similar to the correct one  Applying some sort of filtering or reranking of the system output might therefore lead to better performance 
Future Work
Because the language ID token is so beneficial to performance, it would be very interesting to find ways to extend a similar benefit to unseen languages  One possible way to do so is with tokens that identify something other than the language, such as typological features about the language's phonemic inventory  This could enable better sharing of resources among languages  Such typological knowledge is readily available in databases like Phoible and WALS for a wide variety of languages  It would be interesting to explore if any of these features is a good predictor of a language's orthographic rules 
It would also be interesting to apply the artificial token approach to other problems besides multilingual g2p  One closely related application is monolingual English g2p  Some of the ambiguity of English spelling is due to the wide variety of loanwords in the language, many of which have unassimilated spellings  Knowing the origins of these loanwords could provide a useful hint for figuring out their pronunciations  The etymology of a word could be tagged in an analogous way to how language ID is tagged in multilingual g2p 
Context 2: Introduction
Ancient Chinese is the writing language in ancient China  It is a treasure of Chinese culture which brings together the wisdom and ideas of the Chinese nation and chronicles the ancient cultural heritage of China  Learning ancient Chinese not only helps people to understand and inherit the wisdom of the ancients, but also promotes people to absorb and develop Chinese culture 
However, it is difficult for modern people to read ancient Chinese  Firstly, compared with modern Chinese, ancient Chinese is more concise and shorter  The grammatical order of modern Chinese is also quite different from that of ancient Chinese  Secondly, most modern Chinese words are double syllables, while the most of the ancient Chinese words are monosyllabic  Thirdly, there is more than one polysemous phenomenon in ancient Chinese  In addition, manual translation has a high cost  Therefore, it is meaningful and useful to study the automatic translation from ancient Chinese to modern Chinese  Through ancient-modern Chinese translation, the wisdom, talent and accumulated experience of the predecessors can be passed on to more people 
Neural machine translation (NMT) BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 has achieved remarkable performance on many bilingual translation tasks  It is an end-to-end learning approach for machine translation, with the potential to show great advantages over the statistic machine translation (SMT) systems  However, NMT approach has not been widely applied to the ancient-modern Chinese translation task  One of the main reasons is the limited high-quality parallel data resource 
The most popular method of acquiring translation examples is bilingual text alignment BIBREF5   This kind of method can be classified into two types: lexical-based and statistical-based  The lexical-based approaches BIBREF6 , BIBREF7 focus on lexical information, which utilize the bilingual dictionary BIBREF8 , BIBREF9 or lexical features  Meanwhile, the statistical-based approaches BIBREF10 , BIBREF11 rely on statistical information, such as sentence length ratio in two languages and align mode probability 
However, these methods are designed for other bilingual language pairs that are written in different language characters (e g  English-French, Chinese-Japanese)  The ancient-modern Chinese has some characteristics that are quite different from other language pairs  For example, ancient and modern Chinese are both written in Chinese characters, but ancient Chinese is highly concise and its syntactical structure is different from modern Chinese  The traditional methods do not take these characteristics into account  In this paper, we propose an effective ancient-modern Chinese text alignment method at the level of clause based on the characteristics of these two languages  The proposed method combines both lexical-based information and statistical-based information, which achieves 94 2 F1-score on Test set  Recently, a simple longest common subsequence based approach for ancient-modern Chinese sentence alignment is proposed in BIBREF12   Our experiments showed that our proposed alignment approach performs much better than their method 
We apply the proposed method to create a large translation parallel corpus which contains INLINEFORM0 1 24M bilingual sentence pairs  To our best knowledge, this is the first large high-quality ancient-modern Chinese dataset  Furthermore, we test SMT models and various NMT models on the created dataset and provide a strong baseline for this task 
Overview
There are four steps to build the ancient-modern Chinese translation dataset: (i) The parallel corpus crawling and cleaning  (ii) The paragraph alignment  (iii) The clause alignment based on aligned paragraphs  (iv) Augmenting data by merging aligned adjacent clauses  The most critical step is the third step 
Clause Alignment
In the clause alignment step, we combine both statistical-based and lexical-based information to measure the score for each possible clause alignment between ancient and modern Chinese strings  The dynamic programming is employed to further find overall optimal alignment paragraph by paragraph  According to the characteristics of the ancient and modern Chinese languages, we consider the following factors to measure the alignment score INLINEFORM0 between a bilingual clause pair:
Lexical Matching  The lexical matching score is used to calculate the matching coverage of the ancient clause INLINEFORM0   It contains two parts: exact matching and dictionary matching  An ancient Chinese character usually corresponds to one or more modern Chinese words  In the first part, we carry out Chinese Word segmentation to the modern Chinese clause INLINEFORM1   Then we match the ancient characters and modern words in the order from left to right  In further matching, the words that have been matched will be deleted from the original clauses 
However, some ancient characters do not appear in its corresponding modern Chinese words  An ancient Chinese dictionary is employed to address this issue  We preprocess the ancient Chinese dictionary and remove the stop words  In this dictionary matching step, we retrieve the dictionary definition of each unmatched ancient character and use it to match the remaining modern Chinese words  To reduce the impact of universal word matching, we use Inverse Document Frequency (IDF) to weight the matching words  The lexical matching score is calculated as: DISPLAYFORM0
The above equation is used to calculate the matching coverage of the ancient clause INLINEFORM0   The first term of equation ( EQREF8 ) represents exact matching score  INLINEFORM1 denotes the length of INLINEFORM2 , INLINEFORM3 denotes each ancient character in INLINEFORM4 , and the indicator function INLINEFORM5 indicates whether the character INLINEFORM6 can match the words in the clause INLINEFORM7   The second term is dictionary matching score  Here INLINEFORM8 and INLINEFORM9 represent the remaining unmatched strings of INLINEFORM10 and INLINEFORM11 , respectively  INLINEFORM12 denotes the INLINEFORM13 -th character in the dictionary definition of the INLINEFORM14 and its IDF score is denoted as INLINEFORM15   The INLINEFORM16 is a predefined parameter which is used to normalize the IDF score  We tuned the value of this parameter on the Dev set 
Statistical Information  Similar to BIBREF11 and BIBREF6 , the statistical information contains alignment mode and length information  There are many alignment modes between ancient and modern Chinese languages  If one ancient Chinese clause aligns two adjacent modern Chinese clauses, we call this alignment as 1-2 alignment mode  We show some examples of different alignment modes in Figure FIGREF9   In this paper, we only consider 1-0, 0-1, 1-1, 1-2, 2-1 and 2-2 alignment modes which account for INLINEFORM0 of the Dev set  We estimate the probability Pr INLINEFORM1 n-m INLINEFORM2 of each alignment mode n-m on the Dev set  To utilize length information, we make an investigation on length correlation between these two languages  Based on the assumption of BIBREF11 that each character in one language gives rise to a random number of characters in the other language and those random variables INLINEFORM3 are independent and identically distributed with a normal distribution, we estimate the mean INLINEFORM4 and standard deviation INLINEFORM5 from the paragraph aligned parallel corpus  Given a clause pair INLINEFORM6 , the statistical information score can be calculated by: DISPLAYFORM0
where INLINEFORM0 denotes the normal distribution probability density function 
Edit Distance  Because ancient and modern Chinese are both written in Chinese characters, we also consider using the edit distance  It is a way of quantifying the dissimilarity between two strings by counting the minimum number of operations (insertion, deletion, and substitution) required to transform one string into the other  Here we define the edit distance score as: DISPLAYFORM0
Dynamic Programming  The overall alignment score for each possible clause alignment is as follows: DISPLAYFORM0
Here INLINEFORM0 and INLINEFORM1 are pre-defined interpolation factors  We use dynamic programming to find the overall optimal alignment paragraph by paragraph  Let INLINEFORM2 be total alignment scores of aligning the first to INLINEFORM3 -th ancient Chinese clauses with the first to to INLINEFORM4 -th modern Chinese clauses, and the recurrence then can be described as follows: DISPLAYFORM0
Where INLINEFORM0 denotes concatenate clause INLINEFORM1 to clause INLINEFORM2   As we discussed above, here we only consider 1-0, 0-1, 1-1, 1-2, 2-1 and 2-2 alignment modes 
Ancient-Modern Chinese Dataset
Data Collection  To build the large ancient-modern Chinese dataset, we collected 1 7K bilingual ancient-modern Chinese articles from the internet  More specifically, a large part of the ancient Chinese data we used come from ancient Chinese history records in several dynasties (about 1000BC-200BC) and articles written by celebrities of that era  They used plain and accurate words to express what happened at that time, and thus ensure the generality of the translated materials 
Paragraph Alignment  To further ensure the quality of the new dataset, the work of paragraph alignment is manually completed  After data cleaning and manual paragraph alignment, we obtained 35K aligned bilingual paragraphs 
Clause Alignment  We applied our clause alignment algorithm on the 35K aligned bilingual paragraphs and obtained 517K aligned bilingual clauses  The reason we use clause alignment algorithm instead of sentence alignment is because we can construct more aligned sentences more flexibly and conveniently  To be specific, we can get multiple additional sentence level bilingual pairs by “data augmentation” 
Data Augmentation  We augmented the data in the following way: Given an aligned clause pair, we merged its adjacent clause pairs as a new sample pair  For example, suppose we have three adjacent clause level bilingual pairs: ( INLINEFORM0 , INLINEFORM1 ), ( INLINEFORM2 , INLINEFORM3 ), and ( INLINEFORM4 , INLINEFORM5 )  We can get some additional sentence level bilingual pairs, such as: ( INLINEFORM6 , INLINEFORM7 ) and ( INLINEFORM8 , INLINEFORM9 )  Here INLINEFORM10 , INLINEFORM11 , and INLINEFORM12 are adjacent clauses in the original paragraph, and INLINEFORM13 denotes concatenate clause INLINEFORM14 to clause INLINEFORM15   The advantage of using this data augmentation method is that compared with only using ( INLINEFORM16 , INLINEFORM17 ) as the training data, we can also use ( INLINEFORM18 , INLINEFORM19 ) and ( INLINEFORM20 , INLINEFORM21 ) as the training data, which can provide richer supervision information for the model and make the model learn the align information between the source language and the target language better  After the data augmentation, we filtered the sentences which are longer than 50 or contain more than four clause pairs 
Dataset Creation  Finally, we split the dataset into three sets: training (Train), development (Dev) and testing (Test)  Note that the unaugmented dataset contains 517K aligned bilingual clause pairs from 35K aligned bilingual paragraphs  To keep all the sentences in different sets come from different articles, we split the 35K aligned bilingual paragraphs into Train, Dev and Test sets following these ratios respectively: 80%, 10%, 10%  Before data augmentation, the unaugmented Train set contains INLINEFORM0 aligned bilingual clause pairs from 28K aligned bilingual paragraphs  Then we augmented the Train, Dev and Test sets respectively  Note that the augmented Train, Dev and Test sets also contain the unaugmented data  The statistical information of the three data sets is shown in Table TABREF17   We show some examples of data in Figure FIGREF14  
RNN-based NMT model
We first briefly introduce the RNN based Neural Machine Translation (RNN-based NMT) model  The RNN-based NMT with attention mechanism BIBREF0 has achieved remarkable performance on many translation tasks
Context 3: Introduction
As social media, specially Twitter, takes on an influential role in presidential elections in the U S , natural language processing of political tweets BIBREF0 has the potential to help with nowcasting and forecasting of election results as well as identifying the main issues with a candidate – tasks of much interest to journalists, political scientists, and campaign organizers BIBREF1  As a methodology to obtain training data for a machine learning system that analyzes political tweets, BIBREF2 devised a crowdsourcing scheme with variable crowdworker numbers based on the difficulty of the annotation task  They provided a dataset of tweets where the sentiments towards political candidates were labeled both by experts in political communication and by crowdworkers who were likely not domain experts  BIBREF2 revealed that crowdworkers can match expert performance relatively accurately and in a budget-efficient manner  Given this result, the authors envisioned future work in which groundtruth labels would be crowdsourced for a large number of tweets and then used to design an automated NLP tool for political tweet analysis 
The question we address here is: How accurate are existing NLP tools for political tweet analysis? These tools would provide a baseline performance that any new machine learning system for political tweet analysis would compete against  We here explore whether existing NLP systems can answer the questions "What sentiment?" and "Towards whom?" accurately for the dataset of political tweets provided by BIBREF2  In our analysis, we include NLP tools with publicly-available APIs, even if the tools were not specifically designed for short texts like tweets, and, in particular, political tweets 
Our experiments reveal that the task of entity-level sentiment analysis is difficult for existing tools to answer accurately while the recognition of the entity, here, which politician, was easier 
NLP Toolkits
NLP toolkits typically have the following capabilities: tokenization, part-of-speech (PoS) tagging, chunking, named entity recognition and sentiment analysis  In a study by BIBREF3, it is shown that the well-known NLP toolkits NLTK BIBREF4, Stanford CoreNLP BIBREF5, and TwitterNLP BIBREF6 have tokenization, PoS tagging and NER modules in their pipelines  There are two main approaches for NER: (1) rule-based and (2) statistical or machine learning based  The most ubiquitous algorithms for sequence tagging use Hidden Markov Models BIBREF7, Maximum Entropy Markov Models BIBREF7, BIBREF8, or Conditional Random Fields BIBREF9  Recent works BIBREF10, BIBREF11 have used recurrent neural networks with attention modules for NER 
Sentiment detection tools like SentiStrength BIBREF12 and TensiStrength BIBREF13 are rule-based tools, relying on various dictionaries of emoticons, slangs, idioms, and ironic phrases, and set of rules that can detect the sentiment of a sentence overall or a targeted sentiment  Given a list of keywords, TensiStrength (similar to SentiStrength) reports the sentiment towards selected entities in a sentence, based on five levels of relaxation and five levels of stress 
Among commercial NLP toolkits (e g , BIBREF14, BIBREF15, BIBREF16), we selected BIBREF17 and BIBREF18 for our experiments, which, to the best of our knowledge, are the only publicly accessible commercial APIs for the task of entity-level sentiment analysis that is agnostic to the text domain  We also report results of TensiStrength BIBREF13, TwitterNLP BIBREF6, BIBREF19, CogComp-NLP BIBREF20, and Stanford NLP NER BIBREF21 
Dataset and Analysis Methodology
We used the 1,000-tweet dataset by BIBREF2 that contains the named-entities labels and entity-level sentiments for each of the four 2016 presidential primary candidates Bernie Sanders, Donald Trump, Hillary Clinton, and Ted Cruz, provided by crowdworkers, and by experts in political communication, whose labels are considered groundtruth  The crowdworkers were located in the US and hired on the BIBREF22 platform  For the task of entity-level sentiment analysis, a 3-scale rating of "negative," "neutral," and "positive" was used by the annotators 
BIBREF2 proposed a decision tree approach for computing the number of crowdworkers who should analyze a tweet based on the difficulty of the task  Tweets are labeled by 2, 3, 5, or 7 workers based on the difficulty of the task and the level of disagreement between the crowdworkers  The model computes the number of workers based on how long a tweet is, the presence of a link in a tweet, and the number of present sarcasm signals  Sarcasm is often used in political tweets and causes disagreement between the crowdworkers  The tweets that are deemed to be sarcastic by the decision tree model, are expected to be more difficult to annotate, and hence are allocated more crowdworkers to work on 
We conducted two sets of experiments  In the first set, we used BIBREF23, BIBREF17, and BIBREF18, for entity-level sentiment analysis; in the second set, BIBREF17, BIBREF19, BIBREF24, BIBREF25, and BIBREF26, BIBREF18 for named-entity recognition 
In the experiments that we conducted with TwitterNLP for named-entity recognition, we worked with the default values of the model  Furthermore, we selected the 3-class Stanford NER model, which uses the classes “person,” “organization,” and “location” because it resulted in higher accuracy compared to the 7-class model  For CogComp-NLP NER we used Ontonotes 5 0 NER model BIBREF27  For spaCy NER we used the `en_core_web_lg' model 
We report the experimental results for our two tasks in terms of the correct classification rate (CCR)  For sentiment analysis, we have a three-class problem (positive, negative, and neutral), where the classes are mutually exclusive  The CCR, averaged for a set of tweets, is defined to be the number of correctly-predicted sentiments over the number of groundtruth sentiments in these tweets  For NER, we consider that each tweet may reference up to four candidates, i e , targeted entities  The CCR, averaged for a set of tweets, is the number of correctly predicted entities (candidates) over the number of groundtruth entities (candidates) in this set 
Results and Discussion
The dataset of 1,000 randomly selected tweets contains more than twice as many tweets about Trump than about the other candidates

ground_truth: the Carnegie Mellon Pronouncing Dictionary BIBREF12, the multilingual pronunciation corpus collected by deri2016grapheme , ranscriptions extracted from Wiktionary

multilingual pronunciation corpus collected by deri2016grapheme

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Where does the ancient Chinese dataset come from?
Context 2:  It consists of encoder and decoder part 
We firstly introduce the encoder part  The input word sequence of source language are individually mapped into a INLINEFORM0 -dimensional vector space INLINEFORM1   Then a bi-directional RNN BIBREF15 with GRU BIBREF16 or LSTM BIBREF17 cell converts these vectors into a sequences of hidden states INLINEFORM2  
For the decoder part, another RNN is used to generate target sequence INLINEFORM0   The attention mechanism BIBREF0 , BIBREF18 is employed to allow the decoder to refer back to the hidden state sequence and focus on a particular segment  The INLINEFORM1 -th hidden state INLINEFORM2 of decoder part is calculated as: DISPLAYFORM0
Here g INLINEFORM0 is a linear combination of attended context vector c INLINEFORM1 and INLINEFORM2 is the word embedding of (i-1)-th target word: DISPLAYFORM0
The attended context vector c INLINEFORM0 is computed as a weighted sum of the hidden states of the encoder: DISPLAYFORM0
The probability distribution vector of the next word INLINEFORM0 is generated according to the following: DISPLAYFORM0
We take this model as the basic RNN-based NMT model in the following experiments 
Transformer-NMT
Recently, the Transformer model BIBREF4 has made remarkable progress in machine translation  This model contains a multi-head self-attention encoder and a multi-head self-attention decoder 
As proposed by BIBREF4 , an attention function maps a query and a set of key-value pairs to an output, where the queries INLINEFORM0 , keys INLINEFORM1 , and values INLINEFORM2 are all vectors  The input consists of queries and keys of dimension INLINEFORM3 , and values of dimension INLINEFORM4   The attention function is given by: DISPLAYFORM0
Multi-head attention mechanism projects queries, keys and values to INLINEFORM0 different representation subspaces and calculates corresponding attention  The attention function outputs are concatenated and projected again before giving the final output  Multi-head attention allows the model to attend to multiple features at different positions 
The encoder is composed of a stack of INLINEFORM0 identical layers  Each layer has two sub-layers: multi-head self-attention mechanism and position-wise fully connected feed-forward network  Similarly, the decoder is also composed of a stack of INLINEFORM1 identical layers  In addition to the two sub-layers in each encoder layer, the decoder contains a third sub-layer which performs multi-head attention over the output of the encoder stack (see more details in BIBREF4 ) 
Experiments
Our experiments revolve around the following questions: Q1: As we consider three factors for clause alignment, do all these factors help? How does our method compare with previous methods? Q2: How does the NMT and SMT models perform on this new dataset we build?
Clause Alignment Results (Q1)
In order to evaluate our clause alignment algorithm, we manually aligned bilingual clauses from 37 bilingual ancient-modern Chinese articles, and finally got 4K aligned bilingual clauses as the Test set and 2K clauses as the Dev set 
Metrics  We used F1-score and precision score as the evaluation metrics  Suppose that we get INLINEFORM0 bilingual clause pairs after running the algorithm on the Test set, and there are INLINEFORM1 bilingual clause pairs of these INLINEFORM2 pairs are in the ground truth of the Test set, the precision score is defined as INLINEFORM3 (the algorithm gives INLINEFORM4 outputs, INLINEFORM5 of which are correct)  And suppose that the ground truth of the Test set contains INLINEFORM6 bilingual clause pairs, the recall score is INLINEFORM7 (there are INLINEFORM8 ground truth samples, INLINEFORM9 of which are output by the algorithm), then the F1-score is INLINEFORM10  
Baselines  Since the related work BIBREF10 , BIBREF11 can be seen as the ablation cases of our method (only statistical score INLINEFORM0 with dynamic programming), we compared the full proposed method with its variants on the Test set for ablation study  In addition, we also compared our method with the longest common subsequence (LCS) based approach proposed by BIBREF12   To the best of our knowledge, BIBREF12 is the latest related work which are designed for Ancient-Modern Chinese alignment 
Hyper-parameters  For the proposed method, we estimated INLINEFORM0 and INLINEFORM1 on all aligned paragraphs  The probability Pr INLINEFORM2 n-m INLINEFORM3 of each alignment mode n-m was estimated on the Dev set  For the hyper-parameters INLINEFORM4 , INLINEFORM5 and INLINEFORM6 , the grid search was applied to tune them on the Dev set  In order to show the effect of hyper-parameters INLINEFORM7 , INLINEFORM8 , and INLINEFORM9 , we reported the results of various hyper-parameters on the Dev set in Table TABREF26   Based on the results of grid search on the Dev set, we set INLINEFORM10 , INLINEFORM11 , and INLINEFORM12 in the following experiment  The Jieba Chinese text segmentation is employed for modern Chinese word segmentation 
Results  The results on the Test set are shown in Table TABREF28 , the abbreviation w/o means removing a particular part from the setting  From the results, we can see that the lexical matching score is the most important among these three factors, and statistical information score is more important than edit distance score  Moreover, the dictionary term in lexical matching score significantly improves the performance  From these results, we obtain the best setting that involves all these three factors  We used this setting for dataset creation  Furthermore, the proposed method performs much better than LCS BIBREF12  
Translation Results (Q2)
In this experiment, we analyzed and compared the performance of the SMT and various NMT models on our built dataset  To verify the effectiveness of our data augmented method  We trained the NMT and SMT models on both unaugmented dataset (including 0 46M training pairs) and augmented dataset, and test all the models on the same Test set which is augmented  The models to be tested and their configurations are as follows:
SMT  The state-of-art Moses toolkit BIBREF19 was used to train SMT model  We used KenLM BIBREF20 to train a 5-gram language model, and the GIZA++ toolkit to align the data 
RNN-based NMT  The basic RNN-based NMT model is based on BIBREF0 which is introduced above  Both the encoder and decoder used 2-layer RNN with 1024 LSTM cells, and the encoder is a bi-directional RNN  The batch size, threshold of element-wise gradient clipping and initial learning rate of Adam optimizer BIBREF21 were set to 128, 5 0 and 0 001  When trained the model on augmented dataset, we used 4-layer RNN  Several techniques were investigated to train the model, including layer-normalization BIBREF22 , RNN-dropout BIBREF23 , and learning rate decay BIBREF1   The hyper-parameters were chosen empirically and adjusted in the Dev set  Furthermore, we tested the basic NMT model with several techniques, such as target language reversal BIBREF24 (reversing the order of the words in all target sentences, but not source sentences), residual connection BIBREF25 and pre-trained word2vec BIBREF26   For word embedding pre-training, we collected an external ancient corpus which contains INLINEFORM0 134M tokens 
Transformer-NMT  We also trained the Transformer model BIBREF4 which is a strong baseline of NMT on both augmented and unaugmented parallel corpus  The training configuration of the Transformer model is shown in Table TABREF32   The hyper-parameters are set based on the settings in the paper BIBREF4 and the sizes of our training sets 
For the evaluation, we used the average of 1 to 4 gram BLEUs multiplied by a brevity penalty BIBREF27 which computed by multi-bleu perl in Moses as metrics  The results are reported in Table TABREF34   For RNN-based NMT, we can see that target language reversal, residual connection, and word2vec can further improve the performance of the basic RNN-based NMT model  However, we find that word2vec and reversal tricks seem no obvious improvement when trained the RNN-based NMT and Transformer models on augmented parallel corpus  For SMT, it performs better than NMT models when they were trained on the unaugmented dataset  Nevertheless, when trained on the augmented dataset, both the RNN-based NMT model and Transformer based NMT model outperform the SMT model  In addition, as with other translation tasks BIBREF4 , the Transformer also performs better than RNN-based NMT 
Because the Test set contains both augmented and unaugmented data, it is not surprising that the RNN-based NMT model and Transformer based NMT model trained on unaugmented data would perform poorly  In order to further verify the effect of data augmentation, we report the test results of the models on only unaugmented test data (including 48K test pairs) in Table TABREF35   From the results, it can be seen that the data augmentation can still improve the models 
Analysis
The generated samples of various models are shown in Figure FIGREF36   Besides BLEU scores, we analyze these examples from a human perspective and draw some conclusions  At the same time, we design different metrics and evaluate on the whole Test set to support our conclusions as follows:
On the one hand, we further compare the translation results from the perspective of people  We find that although the original meaning can be basically translated by SMT, its translation results are less smooth when compared with the other two NMT models (RNN-based NMT and Transformer)  For example, the translations of SMT are usually lack of auxiliary words, conjunctions and function words, which is not consistent with human translation habits  To further confirm this conclusion, the average length of the translation results of the three models are measured (RNN-based NMT:17 12, SMT:15 50, Transformer:16 78, Reference:16 47)  We can see that the average length of the SMT outputs is shortest, and the length gaps between the SMT outputs and the references are largest  Meanwhile, the average length of the sentences translated by Transformer is closest to the average length of references  These results indirectly verify our point of view, and show that the NMT models perform better than SMT in this task 
On the other hand, there still exists some problems to be solved  We observe that translating proper nouns and personal pronouns (such as names, place names and ancient-specific appellations) is very difficult for all of these models  For instance, the ancient Chinese appellation `Zhen' should be translated into `Wo' in modern Chinese  Unfortunately, we calculate the accurate rate of some special words (such as `Zhen',`Chen' and `Gua'), and find that this rate is very low (the accurate rate of translating `Zhen' are: RNN-based NMT:0 14, SMT:0 16, Transformer:0 05)  We will focus on this issue in the future 
Conclusion and Future Work
We propose an effective ancient-modern Chinese clause alignment method which achieves 94 2 F1-score on Test set  Based on it, we build a large scale parallel corpus which contains INLINEFORM0 1 24M bilingual sentence pairs  To our best knowledge, this is the first large high-quality ancient-modern Chinese dataset  In addition, we test the performance of the SMT and various NMT models on our built dataset and provide a strong NMT baseline for this task which achieves 27 16 BLEU score (4-gram)  We further analyze the performance of the SMT and various NMT models and summarize some specific problems that machine translation models will encounter when translating ancient Chinese 
For the future work, firstly, we are going to expand the dataset using the proposed method continually  Secondly, we will focus on solving the problem of proper noun translation and improve the translation system according to the features of ancient Chinese translation  Finally, we plan to introduce some techniques of statistical translation into neural machine translation to improve the performance 
This work is supported by National Natural Science Fund for Distinguished Young Scholar (Grant No  61625204) and partially supported by the State Key Program of National Science Foundation of China (Grant Nos  61836006 and 61432014) 

ground_truth: ancient Chinese history records in several dynasties (about 1000BC-200BC) and articles written by celebrities of that era

Ancient Chinese history records in several dynasties and articles written by celebrities during 1000BC-200BC collected from the internet 

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Which toolkits do they use?
Context 3:  In the named-entity recognition experiment, the average CCR of crowdworkers was 98 6%, while the CCR of the automated systems ranged from 77 2% to 96 7%  For four of the automated systems, detecting the entity Trump was more difficult than the other entities (e g , spaCy 72 7% for the entity Trump vs  above 91% for the other entities)  An example of incorrect NER is shown in Figure FIGREF1 top  The difficulties the automated tools had in NER may be explained by the fact that the tools were not trained on tweets, except for TwitterNLP, which was not in active development when the data was created BIBREF1 
In the sentiment analysis experiments, we found that a tweet may contain multiple sentiments  The groundtruth labels contain 210 positive sentiments, 521 neutral sentiments, and 305 negative sentiments to the candidates  We measured the CCR, across all tweets, to be 31 7% for Rosette Text Analytics, 43 2% for Google Cloud, 44 2% for TensiStrength, and 74 7% for the crowdworkers  This means the difference between the performance of the tools and the crowdworkers is significant – more than 30 percent points 
Crowdworkers correctly identified 62% of the neutral, 85% of the positive, and 92% of the negative sentiments  Google Cloud correctly identified 88% of the neutral sentiments, but only 3% of the positive, and 19% of the negative sentiments  TensiStrength correctly identified 87 2% of the neutral sentiments, but 10 5% of the positive, and 8 1% of the negative sentiments  Rosette Text Analytics correctly identified 22 7% of neutral sentiments, 38 1% of negative sentiments and 40 9% of positive sentiments  The lowest and highest CCR pertains to tweets about Trump and Sanders for both Google Cloud and TensiStrength, Trump and Clinton for Rosette Text Analytics, and Clinton and Cruz for crowdworkers  An example of incorrect ELS analysis is shown in Figure FIGREF1 bottom 
Conclusions and Future Work
Our results show that existing NLP systems cannot accurately perform sentiment analysis of political tweets in the dataset we experimented with  Labeling by humans, even non-expert crowdworkers, yields accuracy results that are well above the results of existing automated NLP systems  In future work we will therefore use a crowdworker-labeled dataset to train a new machine-learning based NLP system for tweet analysis  We will ensure that the training data is balanced among classes  Our plan is to use state-of-the-art deep neural networks and compare their performance for entity-level sentiment analysis of political tweets 
Acknowledgments
Partial support of this work by the Hariri Institute for Computing and Computational Science & Engineering at Boston University (to L G ) and a Google Faculty Research Award (to M B  and L G ) is gratefully acknowledged  Additionally, we would like to thank Daniel Khashabi for his help in running the CogComp-NLP Python API and Mike Thelwal for his help with TensiStrength  We are also grateful to the Stanford NLP group for clarifying some of the questions we had with regards to the Stanford NER tool 

ground_truth: BIBREF17, BIBREF18, TensiStrength BIBREF13, TwitterNLP BIBREF6, BIBREF19, CogComp-NLP BIBREF20, Stanford NLP NER BIBREF21

BIBREF23, BIBREF17, BIBREF18, BIBREF19, BIBREF24, BIBREF25, BIBREF26