prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What dataset did they use?
Context 1: Introduction
Automatic classification of sentiment has mainly focused on categorizing tweets in either two (binary sentiment analysis) or three (ternary sentiment analysis) categories BIBREF0   In this work we study the problem of fine-grained sentiment classification where tweets are classified according to a five-point scale ranging from VeryNegative to VeryPositive  To illustrate this, Table TABREF3 presents examples of tweets associated with each of these categories  Five-point scales are widely adopted in review sites like Amazon and TripAdvisor, where a user's sentiment is ordered with respect to its intensity  From a sentiment analysis perspective, this defines a classification problem with five categories  In particular, Sebastiani et al  BIBREF1 defined such classification problems whose categories are explicitly ordered to be ordinal classification problems  To account for the ordering of the categories, learners are penalized according to how far from the true class their predictions are 
Although considering different scales, the various settings of sentiment classification are related  First, one may use the same feature extraction and engineering approaches to represent the text spans such as word membership in lexicons, morpho-syntactic statistics like punctuation or elongated word counts BIBREF2 , BIBREF3   Second, one would expect that knowledge from one task can be transfered to the others and this would benefit the performance  Knowing that a tweet is “Positive” in the ternary setting narrows the classification decision between the VeryPositive and Positive categories in the fine-grained setting  From a research perspective this raises the question of whether and how one may benefit when tackling such related tasks and how one can transfer knowledge from one task to another during the training phase 
Our focus in this work is to exploit the relation between the sentiment classification settings and demonstrate the benefits stemming from combining them  To this end, we propose to formulate the different classification problems as a multitask learning problem and jointly learn them  Multitask learning BIBREF4 has shown great potential in various domains and its benefits have been empirically validated BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 using different types of data and learning approaches  An important benefit of multitask learning is that it provides an elegant way to access resources developed for similar tasks  By jointly learning correlated tasks, the amount of usable data increases  For instance, while for ternary classification one can label data using distant supervision with emoticons BIBREF9 , there is no straightforward way to do so for the fine-grained problem  However, the latter can benefit indirectly, if the ternary and fine-grained tasks are learned jointly 
The research question that the paper attempts to answer is the following: Can twitter sentiment classification problems, and fine-grained sentiment classification in particular, benefit from multitask learning? To answer the question, the paper brings the following two main contributions: (i) we show how jointly learning the ternary and fine-grained sentiment classification problems in a multitask setting improves the state-of-the-art performance, and (ii) we demonstrate that recurrent neural networks outperform models previously proposed without access to huge corpora while being flexible to incorporate different sources of data 
Multitask Learning for Twitter Sentiment Classification
In his work, Caruana BIBREF4 proposed a multitask approach in which a learner takes advantage of the multiplicity of interdependent tasks while jointly learning them  The intuition is that if the tasks are correlated, the learner can learn a model jointly for them while taking into account the shared information which is expected to improve its generalization ability  People express their opinions online on various subjects (events, products  ), on several languages and in several styles (tweets, paragraph-sized reviews  ), and it is exactly this variety that motivates the multitask approaches  Specifically for Twitter for instance, the different settings of classification like binary, ternary and fine-grained are correlated since their difference lies in the sentiment granularity of the classes which increases while moving from binary to fine-grained problems 
There are two main decisions to be made in our approach: the learning algorithm, which learns a decision function, and the data representation  With respect to the former, neural networks are particularly suitable as one can design architectures with different properties and arbitrary complexity  Also, as training neural network usually relies on back-propagation of errors, one can have shared parts of the network trained by estimating errors on the joint tasks and others specialized for particular tasks  Concerning the data representation, it strongly depends on the data type available  For the task of sentiment classification of tweets with neural networks, distributed embeddings of words have shown great potential  Embeddings are defined as low-dimensional, dense representations of words that can be obtained in an unsupervised fashion by training on large quantities of text BIBREF10  
Concerning the neural network architecture, we focus on Recurrent Neural Networks (RNNs) that are capable of modeling short-range and long-range dependencies like those exhibited in sequence data of arbitrary length like text  While in the traditional information retrieval paradigm such dependencies are captured using INLINEFORM0 -grams and skip-grams, RNNs learn to capture them automatically BIBREF11   To circumvent the problems with capturing long-range dependencies and preventing gradients from vanishing, the long short-term memory network (LSTM) was proposed BIBREF12   In this work, we use an extended version of LSTM called bidirectional LSTM (biLSTM)  While standard LSTMs access information only from the past (previous words), biLSTMs capture both past and future information effectively BIBREF13 , BIBREF11   They consist of two LSTM networks, for propagating text forward and backwards with the goal being to capture the dependencies better  Indeed, previous work on multitask learning showed the effectiveness of biLSTMs in a variety of problems: BIBREF14 tackled sequence prediction, while BIBREF6 and BIBREF15 used biLSTMs for Named Entity Recognition and dependency parsing respectively 
Figure FIGREF2 presents the architecture we use for multitask learning  In the top-left of the figure a biLSTM network (enclosed by the dashed line) is fed with embeddings INLINEFORM0 that correspond to the INLINEFORM1 words of a tokenized tweet  Notice, as discussed above, the biLSTM consists of two LSTMs that are fed with the word sequence forward and backwards  On top of the biLSTM network one (or more) hidden layers INLINEFORM2 transform its output  The output of INLINEFORM3 is led to the softmax layers for the prediction step  There are INLINEFORM4 softmax layers and each is used for one of the INLINEFORM5 tasks of the multitask setting  In tasks such as sentiment classification, additional features like membership of words in sentiment lexicons or counts of elongated/capitalized words can be used to enrich the representation of tweets before the classification step BIBREF3   The lower part of the network illustrates how such sources of information can be incorporated to the process  A vector “Additional Features” for each tweet is transformed from the hidden layer(s) INLINEFORM6 and then is combined by concatenation with the transformed biLSTM output in the INLINEFORM7 layer 
Experimental setup
Our goal is to demonstrate how multitask learning can be successfully applied on the task of sentiment classification of tweets  The particularities of tweets are to be short and informal text spans  The common use of abbreviations, creative language etc , makes the sentiment classification problem challenging  To validate our hypothesis, that learning the tasks jointly can benefit the performance, we propose an experimental setting where there are data from two different twitter sentiment classification problems: a fine-grained and a ternary  We consider the fine-grained task to be our primary task as it is more challenging and obtaining bigger datasets, e g  by distant supervision, is not straightforward and, hence we report the performance achieved for this task 
Ternary and fine-grained sentiment classification were part of the SemEval-2016 “Sentiment Analysis in Twitter” task BIBREF16   We use the high-quality datasets the challenge organizers released  The dataset for fine-grained classification is split in training, development, development_test and test parts  In the rest, we refer to these splits as train, development and test, where train is composed by the training and the development instances  Table TABREF7 presents an overview of the data  As discussed in BIBREF16 and illustrated in the Table, the fine-grained dataset is highly unbalanced and skewed towards the positive sentiment: only INLINEFORM0 of the training examples are labeled with one of the negative classes 
Feature representation We report results using two different feature sets  The first one, dubbed nbow, is a neural bag-of-words that uses text embeddings to generate low-dimensional, dense representations of the tweets  To construct the nbow representation, given the word embeddings dictionary where each word is associated with a vector, we apply the average compositional function that averages the embeddings of the words that compose a tweet  Simple compositional functions like average were shown to be robust and efficient in previous work BIBREF17   Instead of training embeddings from scratch, we use the pre-trained on tweets GloVe embeddings of BIBREF10   In terms of resources required, using only nbow is efficient as it does not require any domain knowledge  However, previous research on sentiment analysis showed that using extra resources, like sentiment lexicons, can benefit significantly the performance BIBREF3 , BIBREF2   To validate this and examine at which extent neural networks and multitask learning benefit from such features we evaluate the models using an augmented version of nbow, dubbed nbow+  The feature space of the latter, is augmented using 1,368 extra features consisting mostly of counts of punctuation symbols ('!?#@'), emoticons, elongated words and word membership features in several sentiment lexicons  Due to space limitations, for a complete presentation of these features, we refer the interested reader to BIBREF2 , whose open implementation we used to extract them 
Evaluation measure To reproduce the setting of the SemEval challenges BIBREF16 , we optimize our systems using as primary measure the macro-averaged Mean Absolute Error ( INLINEFORM0 ) given by: INLINEFORM1
where INLINEFORM0 is the number of categories, INLINEFORM1 is the set of instances whose true class is INLINEFORM2 , INLINEFORM3 is the true label of the instance INLINEFORM4 and INLINEFORM5 the predicted label  The measure penalizes decisions far from the true ones and is macro-averaged to account for the fact that the data are unbalanced  Complementary to INLINEFORM6 , we report the performance achieved on the micro-averaged INLINEFORM7 measure, which is a commonly used measure for classification 
The models To evaluate the multitask learning approach, we compared it with several other models  Support Vector Machines (SVMs) are maximum margin classification algorithms that have been shown to achieve competitive performance in several text classification problems BIBREF16   SVM INLINEFORM0 stands for an SVM with linear kernel and an one-vs-rest approach for the multi-class problem  Also, SVM INLINEFORM1 is an SVM with linear kernel that employs the crammer-singer strategy BIBREF18 for the multi-class problem  Logistic regression (LR) is another type of linear classification method, with probabilistic motivation  Again, we use two types of Logistic Regression depending on the multi-class strategy: LR INLINEFORM2 that uses an one-vs-rest approach and multinomial Logistic Regression also known as the MaxEnt classifier that uses a multinomial criterion 
Both SVMs and LRs as discussed above treat the problem as a multi-class one, without considering the ordering of the classes  For these four models, we tuned the hyper-parameter INLINEFORM0 that controls the importance of the L INLINEFORM1 regularization part in the optimization problem with grid-search over INLINEFORM2 using 10-fold cross-validation in the union of the training and development data and then retrained the models with the selected values  Also, to account for the unbalanced classification problem we used class weights to penalize more the errors made on the rare classes  These weights were inversely proportional to the frequency of each class  For the four models we used the implementations of Scikit-learn BIBREF19  
For multitask learning we use the architecture shown in Figure FIGREF2 , which we implemented with Keras BIBREF20   The embeddings are initialized with the 50-dimensional GloVe embeddings while the output of the biLSTM network is set to dimension 50  The activation function of the hidden layers is the hyperbolic tangent  The weights of the layers were initialized from a uniform distribution, scaled as described in BIBREF21   We used the Root Mean Square Propagation optimization method  We used dropout for regularizing the network  We trained the network using batches of 128 examples as follows: before selecting the batch, we perform a Bernoulli trial with probability INLINEFORM0 to select the task to train for  With probability INLINEFORM1 we pick a batch for the fine-grained sentiment classification problem, while with probability INLINEFORM2 we pick a batch for the ternary problem  As shown in Figure FIGREF2 , the error is backpropagated until the embeddings, that we fine-tune during the learning process  Notice also that the weights of the network until the layer INLINEFORM3 are shared and therefore affected by both tasks 
To tune the neural network hyper-parameters we used 5-fold cross validation  We tuned the probability INLINEFORM0 of dropout after the hidden layers INLINEFORM1 and for the biLSTM for INLINEFORM2 , the size of the hidden layer INLINEFORM3 and the probability INLINEFORM4 of the Bernoulli trials from INLINEFORM5   During training, we monitor the network's performance on the development set and apply early stopping if the performance on the validation set does not improve for 5 consecutive epochs 
Experimental results Table TABREF9 illustrates the performance of the models for the different data representations  The upper part of the Table summarizes the performance of the baselines  The entry “Balikas et al ” stands for the winning system of the 2016 edition of the challenge BIBREF2 , which to the best of our knowledge holds the state-of-the-art  Due to the stochasticity of training the biLSTM models, we repeat the experiment 10 times and report the average and the standard deviation of the performance achieved 
Several observations can be made from the table  First notice that, overall, the best performance is achieved by the neural network architecture that uses multitask learning  This entails that the system makes use of the available resources efficiently and improves the state-of-the-art performance  In conjunction with the fact that we found the optimal probability INLINEFORM0 , this highlights the benefits of multitask learning over single task learning  Furthermore, as described above, the neural network-based models have only access to the training data as the development are hold for early stopping  On the other hand, the baseline systems were retrained on the union of the train and development sets  Hence, even with fewer resources available for training on the fine-grained problem, the neural networks outperform the baselines  We also highlight the positive effect of the additional features that previous research proposed  Adding the features both in the baselines and in the biLSTM-based architectures improves the INLINEFORM1 scores by several points 
Lastly, we compare the performance of the baseline systems with the performance of the state-of-the-art system of BIBREF2   While BIBREF2 uses n-grams (and character-grams) with INLINEFORM0 , the baseline systems (SVMs, LRs) used in this work use the nbow+ representation, that relies on unigrams  Although they perform on par, the competitive performance of nbow highlights the potential of distributed representations for short-text classification  Further, incorporating structure and distributed representations leads to the gains of the biLSTM network, in the multitask and single task setting 
Similar observations can be drawn from Figure FIGREF10 that presents the INLINEFORM0 scores  Again, the biLSTM network with multitask learning achieves the best performance  It is also to be noted that although the two evaluation measures are correlated in the sense that the ranking of the models is the same, small differences in the INLINEFORM1 have large effect on the scores of the INLINEFORM2 measure 
Conclusion
In this paper, we showed that by jointly learning the tasks of ternary and fine-grained classification with a multitask learning model, one can greatly improve the performance on the second  This opens several avenues for future research  Since sentiment is expressed in different textual types like tweets and paragraph-sized reviews, in different languages (English, German,   ) and in different granularity levels (binary, ternary,  ) one can imagine multitask approaches that could benefit from combining such resources  Also, while we opted for biLSTM networks here, one could use convolutional neural networks or even try to combine different types of networks and tasks to investigate the performance effect of multitask learning  Lastly, while our approach mainly relied on the foundations of BIBREF4 , the internal mechanisms and the theoretical guarantees of multitask learning remain to be better understood 
Acknowledgements
This work is partially supported by the CIFRE N 28/2015 
Context 3: Introduction
This paper describes our approach and results for Task 2 of the CoNLL–SIGMORPHON 2018 shared task on universal morphological reinflection BIBREF0   The task is to generate an inflected word form given its lemma and the context in which it occurs 
Morphological (re)inflection from context is of particular relevance to the field of computational linguistics: it is compelling to estimate how well a machine-learned system can capture the morphosyntactic properties of a word given its context, and map those properties to the correct surface form for a given lemma 
There are two tracks of Task 2 of CoNLL–SIGMORPHON 2018: in Track 1 the context is given in terms of word forms, lemmas and morphosyntactic descriptions (MSD); in Track 2 only word forms are available  See Table TABREF1 for an example  Task 2 is additionally split in three settings based on data size: high, medium and low, with high-resource datasets consisting of up to 70K instances per language, and low-resource datasets consisting of only about 1K instances 
The baseline provided by the shared task organisers is a seq2seq model with attention (similar to the winning system for reinflection in CoNLL–SIGMORPHON 2016, BIBREF1 ), which receives information about context through an embedding of the two words immediately adjacent to the target form  We use this baseline implementation as a starting point and achieve the best overall accuracy of 49 87 on Task 2 by introducing three augmentations to the provided baseline system: (1) We use an LSTM to encode the entire available context; (2) We employ a multi-task learning approach with the auxiliary objective of MSD prediction; and (3) We train the auxiliary component in a multilingual fashion, over sets of two to three languages 
In analysing the performance of our system, we found that encoding the full context improves performance considerably for all languages: 11 15 percentage points on average, although it also highly increases the variance in results  Multi-task learning, paired with multilingual training and subsequent monolingual finetuning, scored highest for five out of seven languages, improving accuracy by another 9 86% on average 
System Description
Our system is a modification of the provided CoNLL–SIGMORPHON 2018 baseline system, so we begin this section with a reiteration of the baseline system architecture, followed by a description of the three augmentations we introduce 
Baseline
The CoNLL–SIGMORPHON 2018 baseline is described as follows:
The system is an encoder-decoder on character sequences  It takes a lemma as input and generates a word form  The process is conditioned on the context of the lemma [   ] The baseline treats the lemma, word form and MSD of the previous and following word as context in track 1  In track 2, the baseline only considers the word forms of the previous and next word  [   ] The baseline system concatenates embeddings for context word forms, lemmas and MSDs into a context vector  The baseline then computes character embeddings for each character in the input lemma  Each of these is concatenated with a copy of the context vector  The resulting sequence of vectors is encoded using an LSTM encoder  Subsequently, an LSTM decoder generates the characters in the output word form using encoder states and an attention mechanism 
To that we add a few details regarding model size and training schedule:
the number of LSTM layers is one;
embedding size, LSTM layer size and attention layer size is 100;
models are trained for 20 epochs;
on every epoch, training data is subsampled at a rate of 0 3;
LSTM dropout is applied at a rate 0 3;
context word forms are randomly dropped at a rate of 0 1;
the Adam optimiser is used, with a default learning rate of 0 001; and
trained models are evaluated on the development data (the data for the shared task comes already split in train and dev sets) 
Our system
Here we compare and contrast our system to the baseline system  A diagram of our system is shown in Figure FIGREF4  
The idea behind this modification is to provide the encoder with access to all morpho-syntactic cues present in the sentence  In contrast to the baseline, which only encodes the immediately adjacent context of a target word, we encode the entire context  All context word forms, lemmas, and MSD tags (in Track 1) are embedded in their respective high-dimensional spaces as before, and their embeddings are concatenated  However, we now reduce the entire past context to a fixed-size vector by encoding it with a forward LSTM, and we similarly represent the future context by encoding it with a backwards LSTM 
We introduce an auxiliary objective that is meant to increase the morpho-syntactic awareness of the encoder and to regularise the learning process—the task is to predict the MSD tag of the target form  MSD tag predictions are conditioned on the context encoding, as described in UID15   Tags are generated with an LSTM one component at a time, e g  the tag PRO;NOM;SG;1 is predicted as a sequence of four components, INLINEFORM0 PRO, NOM, SG, 1 INLINEFORM1  
For every training instance, we backpropagate the sum of the main loss and the auxiliary loss without any weighting 
As MSD tags are only available in Track 1, this augmentation only applies to this track 
The parameters of the entire MSD (auxiliary-task) decoder are shared across languages 
Since a grouping of the languages based on language family would have left several languages in single-member groups (e g  Russian is the sole representative of the Slavic family), we experiment with random groupings of two to three languages  Multilingual training is performed by randomly alternating between languages for every new minibatch  We do not pass any information to the auxiliary decoder as to the source language of the signal it is receiving, as we assume abstract morpho-syntactic features are shared across languages 
After 20 epochs of multilingual training, we perform 5 epochs of monolingual finetuning for each language  For this phase, we reduce the learning rate to a tenth of the original learning rate, i e  0 0001, to ensure that the models are indeed being finetuned rather than retrained 
We keep all hyperparameters the same as in the baseline

ground_truth:  high-quality datasets  from SemEval-2016 “Sentiment Analysis in Twitter” task

 SemEval-2016 “Sentiment Analysis in Twitter”

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. what language does this paper focus on?
Context 2: Introduction
Text simplification aims to reduce the lexical and structural complexity of a text, while still retaining the semantic meaning, which can help children, non-native speakers, and people with cognitive disabilities, to understand text better  One of the methods of automatic text simplification can be generally divided into three categories: lexical simplification (LS) BIBREF0 , BIBREF1 , rule-based BIBREF2 , and machine translation (MT) BIBREF3 , BIBREF4   LS is mainly used to simplify text by substituting infrequent and difficult words with frequent and easier words  However, there are several challenges for the LS approach: a great number of transformation rules are required for reasonable coverage and should be applied based on the specific context; third, the syntax and semantic meaning of the sentence is hard to retain  Rule-based approaches use hand-crafted rules for lexical and syntactic simplification, for example, substituting difficult words in a predefined vocabulary  However, such approaches need a lot of human-involvement to manually define these rules, and it is impossible to give all possible simplification rules  MT-based approach has attracted great attention in the last several years, which addresses text simplification as a monolingual machine translation problem translating from 'ordinary' and 'simplified' sentences 
In recent years, neural Machine Translation (NMT) is a newly-proposed deep learning approach and achieves very impressive results BIBREF5 , BIBREF6 , BIBREF7   Unlike the traditional phrased-based machine translation system which operates on small components separately, NMT system is being trained end-to-end, without the need to have external decoders, language models or phrase tables  Therefore, the existing architectures in NMT are used for text simplification BIBREF8 , BIBREF4   However, most recent work using NMT is limited to the training data that are scarce and expensive to build  Language models trained on simplified corpora have played a central role in statistical text simplification BIBREF9 , BIBREF10   One main reason is the amount of available simplified corpora typically far exceeds the amount of parallel data  The performance of models can be typically improved when trained on more data  Therefore, we expect simplified corpora to be especially helpful for NMT models 
In contrast to previous work, which uses the existing NMT models, we explore strategy to include simplified training corpora in the training process without changing the neural network architecture  We first propose to pair simplified training sentences with synthetic ordinary sentences during training, and treat this synthetic data as additional training data  We obtain synthetic ordinary sentences through back-translation, i e  an automatic translation of the simplified sentence into the ordinary sentence BIBREF11   Then, we mix the synthetic data into the original (simplified-ordinary) data to train NMT model  Experimental results on two publicly available datasets show that we can improve the text simplification quality of NMT models by mixing simplified sentences into the training set over NMT model only using the original training data 
Related Work
Automatic TS is a complicated natural language processing (NLP) task, which consists of lexical and syntactic simplification levels BIBREF12   It has attracted much attention recently as it could make texts more accessible to wider audiences, and used as a pre-processing step, improve performances of various NLP tasks and systems BIBREF13 , BIBREF14 , BIBREF15   Usually, hand-crafted, supervised, and unsupervised methods based on resources like English Wikipedia and Simple English Wikipedia (EW-SEW) BIBREF10 are utilized for extracting simplification rules  It is very easy to mix up the automatic TS task and the automatic summarization task BIBREF3 , BIBREF16 , BIBREF6   TS is different from text summarization as the focus of text summarization is to reduce the length and redundant content 
At the lexical level, lexical simplification systems often substitute difficult words using more common words, which only require a large corpus of regular text to obtain word embeddings to get words similar to the complex word BIBREF1 , BIBREF9   Biran et al  BIBREF0 adopted an unsupervised method for learning pairs of complex and simpler synonyms from a corpus consisting of Wikipedia and Simple Wikipedia  At the sentence level, a sentence simplification model was proposed by tree transformation based on statistical machine translation (SMT) BIBREF3   Woodsend and Lapata BIBREF17 presented a data-driven model based on a quasi-synchronous grammar, a formalism that can naturally capture structural mismatches and complex rewrite operations  Wubben et al  BIBREF18 proposed a phrase-based machine translation (PBMT) model that is trained on ordinary-simplified sentence pairs  Xu et al  BIBREF19 proposed a syntax-based machine translation model using simplification-specific objective functions and features to encourage simpler output 
Compared with SMT, neural machine translation (NMT) has shown to produce state-of-the-art results BIBREF5 , BIBREF7   The central approach of NMT is an encoder-decoder architecture implemented by recurrent neural networks, which can represent the input sequence as a vector, and then decode that vector into an output sequence  Therefore, NMT models were used for text simplification task, and achieved good results BIBREF8 , BIBREF4 , BIBREF20   The main limitation of the aforementioned NMT models for text simplification depended on the parallel ordinary-simplified sentence pairs  Because ordinary-simplified sentence pairs are expensive and time-consuming to build, the available largest data is EW-SEW that only have 296,402 sentence pairs  The dataset is insufficiency for NMT model if we want to NMT model can obtain the best parameters  Considering simplified data plays an important role in boosting fluency for phrase-based text simplification, and we investigate the use of simplified data for text simplification  We are the first to show that we can effectively adapt neural translation models for text simplifiation with simplified corpora 
Simplified Corpora
We collected a simplified dataset from Simple English Wikipedia that are freely available, which has been previously used for many text simplification methods BIBREF0 , BIBREF10 , BIBREF3   The simple English Wikipedia is pretty easy to understand than normal English Wikipedia  We downloaded all articles from Simple English Wikipedia  For these articles, we removed stubs, navigation pages and any article that consisted of a single sentence  We then split them into sentences with the Stanford CorNLP BIBREF21 , and deleted these sentences whose number of words are smaller than 10 or large than 40  After removing repeated sentences, we chose 600K sentences as the simplified data with 11 6M words, and the size of vocabulary is 82K 
Text Simplification using Neural Machine Translation
Our work is built on attention-based NMT BIBREF5 as an encoder-decoder network with recurrent neural networks (RNN), which simultaneously conducts dynamic alignment and generation of the target simplified sentence 
The encoder uses a bidirectional RNN that consists of forward and backward RNN  Given a source sentence INLINEFORM0 , the forward RNN and backward RNN calculate forward hidden states INLINEFORM1 and backward hidden states INLINEFORM2 , respectively  The annotation vector INLINEFORM3 is obtained by concatenating INLINEFORM4 and INLINEFORM5  
The decoder is a RNN that predicts a target simplificated sentence with Gated Recurrent Unit (GRU) BIBREF22   Given the previously generated target (simplified) sentence INLINEFORM0 , the probability of next target word INLINEFORM1 is DISPLAYFORM0
where INLINEFORM0 is a non-linear function, INLINEFORM1 is the embedding of INLINEFORM2 , and INLINEFORM3 is a decoding state for time step INLINEFORM4  
State INLINEFORM0 is calculated by DISPLAYFORM0
where INLINEFORM0 is the activation function GRU 
The INLINEFORM0 is the context vector computed as a weighted annotation INLINEFORM1 , computed by DISPLAYFORM0
where the weight INLINEFORM0 is computed by DISPLAYFORM0 DISPLAYFORM1
where INLINEFORM0 , INLINEFORM1 and INLINEFORM2 are weight matrices  The training objective is to maximize the likelihood of the training data  Beam search is employed for decoding 
Synthetic Simplified Sentences
We train an auxiliary system using NMT model from the simplified sentence to the ordinary sentence, which is first trained on the available parallel data  For leveraging simplified sentences to improve the quality of NMT model for text simplification, we propose to adapt the back-translation approach proposed by Sennrich et al  BIBREF11 to our scenario  More concretely, Given one sentence in simplified sentences, we use the simplified-ordinary system in translate mode with greedy decoding to translate it to the ordinary sentences, which is denoted as back-translation  This way, we obtain a synthetic parallel simplified-ordinary sentences  Both the synthetic sentences and the available parallel data are used as training data for the original NMT system 
Evaluation
We evaluate the performance of text simplification using neural machine translation on available parallel sentences and additional simplified sentences 
Dataset  We use two simplification datasets (WikiSmall and WikiLarge)  WikiSmall consists of ordinary and simplified sentences from the ordinary and simple English Wikipedias, which has been used as benchmark for evaluating text simplification BIBREF17 , BIBREF18 , BIBREF8   The training set has 89,042 sentence pairs, and the test set has 100 pairs  WikiLarge is also from Wikipedia corpus whose training set contains 296,402 sentence pairs BIBREF19 , BIBREF20   WikiLarge includes 8 (reference) simplifications for 2,359 sentences split into 2,000 for development and 359 for testing 
Metrics  Three metrics in text simplification are chosen in this paper  BLEU BIBREF5 is one traditional machine translation metric to assess the degree to which translated simplifications differed from reference simplifications  FKGL measures the readability of the output BIBREF23   A small FKGL represents simpler output  SARI is a recent text-simplification metric by comparing the output against the source and reference simplifications BIBREF20  
We evaluate the output of all systems using human evaluation  The metric is denoted as Simplicity BIBREF8   The three non-native fluent English speakers are shown reference sentences and output sentences  They are asked whether the output sentence is much simpler (+2), somewhat simpler (+1), equally (0), somewhat more difficult (-1), and much more difficult (-2) than the reference sentence 
Methods  We use OpenNMT BIBREF24 as the implementation of the NMT system for all experiments BIBREF5   We generally follow the default settings and training procedure described by Klein et al (2017)  We replace out-of-vocabulary words with a special UNK symbol  At prediction time, we replace UNK words with the highest probability score from the attention layer  OpenNMT system used on parallel data is the baseline system  To obtain a synthetic parallel training set, we back-translate a random sample of 100K sentences from the collected simplified corpora  OpenNMT used on parallel data and synthetic data is our model  The benchmarks are run on a Intel(R) Core(TM) i7-5930K CPU@3 50GHz, 32GB Mem, trained on 1 GPU GeForce GTX 1080 (Pascal) with CUDA v  8 0 
We choose three statistical text simplification systems  PBMT-R is a phrase-based method with a reranking post-processing step BIBREF18   Hybrid performs sentence splitting and deletion operations based on discourse representation structures, and then simplifies sentences with PBMT-R BIBREF25   SBMT-SARI BIBREF19 is syntax-based translation model using PPDB paraphrase database BIBREF26 and modifies tuning function (using SARI)  We choose two neural text simplification systems  NMT is a basic attention-based encoder-decoder model which uses OpenNMT framework to train with two LSTM layers, hidden states of size 500 and 500 hidden units, SGD optimizer, and a dropout rate of 0 3 BIBREF8   Dress is an encoder-decoder model coupled with a deep reinforcement learning framework, and the parameters are chosen according to the original paper BIBREF20   For the experiments with synthetic parallel data, we back-translate a random sample of 60 000 sentences from the collected simplified sentences into ordinary sentences  Our model is trained on synthetic data and the available parallel data, denoted as NMT+synthetic 
Results  Table 1 shows the results of all models on WikiLarge dataset  We can see that our method (NMT+synthetic) can obtain higher BLEU, lower FKGL and high SARI compared with other models, except Dress on FKGL and SBMT-SARI on SARI  It verified that including synthetic data during training is very effective, and yields an improvement over our baseline NMF by 2 11 BLEU, 1 7 FKGL and 1 07 SARI  We also substantially outperform Dress, who previously reported SOTA result  The results of our human evaluation using Simplicity are also presented in Table 1  NMT on synthetic data is significantly better than PBMT-R, Dress, and SBMT-SARI on Simplicity  It indicates that our method with simplified data is effective at creating simpler output 
Results on WikiSmall dataset are shown in Table 2  We see substantial improvements (6 37 BLEU) than NMT from adding simplified training data with synthetic ordinary sentences  Compared with statistical machine translation models (PBMT-R, Hybrid, SBMT-SARI), our method (NMT+synthetic) still have better results, but slightly worse FKGL and SARI  Similar to the results in WikiLarge, the results of our human evaluation using Simplicity outperforms the other models  In conclusion, Our method produces better results comparing with the baselines, which demonstrates the effectiveness of adding simplified training data 
Conclusion
In this paper, we propose one simple method to use simplified corpora during training of NMT systems, with no changes to the network architecture  In the experiments on two datasets, we achieve substantial gains in all tasks, and new SOTA results, via back-translation of simplified sentences into the ordinary sentences, and treating this synthetic data as additional training data  Because we do not change the neural network architecture to integrate simplified corpora, our method can be easily applied to other Neural Text Simplification (NTS) systems  We expect that the effectiveness of our method not only varies with the quality of the NTS system used for back-translation, but also depends on the amount of available parallel and simplified corpora  In the paper, we have only utilized data from Wikipedia for simplified sentences  In the future, many other text sources are available and the impact of not only size, but also of domain should be investigated 
Context 3:  Training data is split 90:10 for training and validation  We train our models for 50 epochs, adding early stopping with a tolerance of five epochs of no improvement in the validation loss  We do not subsample from the training data 
We train models for 50 different random combinations of two to three languages in Track 1, and 50 monolingual models for each language in Track 2  Instead of picking the single model that performs best on the development set and thus risking to select a model that highly overfits that data, we use an ensemble of the five best models, and make the final prediction for a given target form with a majority vote over the five predictions 
Results and Discussion
Test results are listed in Table TABREF17   Our system outperforms the baseline for all settings and languages in Track 1 and for almost all in Track 2—only in the high resource setting is our system not definitively superior to the baseline 
Interestingly, our results in the low resource setting are often higher for Track 2 than for Track 1, even though contextual information is less explicit in the Track 2 data and the multilingual multi-tasking approach does not apply to this track  We interpret this finding as an indicator that a simpler model with fewer parameters works better in a setting of limited training data  Nevertheless, we focus on the low resource setting in the analysis below due to time limitations  As our Track 1 results are still substantially higher than the baseline results, we consider this analysis valid and insightful 
Ablation Study
We analyse the incremental effect of the different features in our system, focusing on the low-resource setting in Track 1 and using development data 
Encoding the entire context with an LSTM highly increases the variance of the observed results  So we trained fifty models for each language and each architecture  Figure FIGREF23 visualises the means and standard deviations over the trained models  In addition, we visualise the average accuracy for the five best models for each language and architecture, as these are the models we use in the final ensemble prediction  Below we refer to these numbers only 
The results indicate that encoding the full context with an LSTM highly enhances the performance of the model, by 11 15% on average  This observation explains the high results we obtain also for Track 2 
Adding the auxiliary objective of MSD prediction has a variable effect: for four languages (de, en, es, and sv) the effect is positive, while for the rest it is negative  We consider this to be an issue of insufficient data for the training of the auxiliary component in the low resource setting we are working with 
We indeed see results improving drastically with the introduction of multilingual training, with multilingual results being 7 96% higher than monolingual ones on average 
We studied the five best models for each language as emerging from the multilingual training (listed in Table TABREF27 ) and found no strong linguistic patterns  The en–sv pairing seems to yield good models for these languages, which could be explained in terms of their common language family and similar morphology  The other natural pairings, however, fr–es, and de–sv, are not so frequent among the best models for these pairs of languages 
Finally, monolingual finetuning improves accuracy across the board, as one would expect, by 2 72% on average 
The final observation to be made based on this breakdown of results is that the multi-tasking approach paired with multilingual training and subsequent monolingual finetuning outperforms the other architectures for five out of seven languages: de, en, fr, ru and sv  For the other two languages in the dataset, es and fi, the difference between this approach and the approach that emerged as best for them is less than 1%  The overall improvement of the multilingual multi-tasking approach over the baseline is 18 30% 
Error analysis
Here we study the errors produced by our system on the English test set to better understand the remaining shortcomings of the approach  A small portion of the wrong predictions point to an incorrect interpretation of the morpho-syntactic conditioning of the context, e g  the system predicted plan instead of plans in the context Our _ include raising private capital  The majority of wrong predictions, however, are nonsensical, like bomb for job, fify for fixing, and gnderrate for understand  This observation suggests that generally the system did not learn to copy the characters of lemma into inflected form, which is all it needs to do in a large number of cases  This issue could be alleviated with simple data augmentation techniques that encourage autoencoding BIBREF2  
MSD prediction
Figure FIGREF32 summarises the average MSD-prediction accuracy for the multi-tasking experiments discussed above  Accuracy here is generally higher than on the main task, with the multilingual finetuned setup for Spanish and the monolingual setup for French scoring best: 66 59% and 65 35%, respectively  This observation illustrates the added difficulty of generating the correct surface form even when the morphosyntactic description has been identified correctly 
We observe some correlation between these numbers and accuracy on the main task: for de, en, ru and sv, the brown, pink and blue bars here pattern in the same way as the corresponding INLINEFORM0 's in Figure FIGREF23   One notable exception to this pattern is fr where inflection gains a lot from multilingual training, while MSD prediction suffers greatly  Notice that the magnitude of change is not always the same, however, even when the general direction matches: for ru, for example, multilingual training benefits inflection much more than in benefits MSD prediction, even though the MSD decoder is the only component that is actually shared between languages  This observation illustrates the two-fold effect of multi-task training: an auxiliary task can either inform the main task through the parameters the two tasks share, or it can help the main task learning through its regularising effect 
Related Work
Our system is inspired by previous work on multi-task learning and multi-lingual learning, mainly building on two intuitions: (1) jointly learning related tasks tends to be beneficial BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 ; and (2) jointly learning related languages in an MTL-inspired framework tends to be beneficial BIBREF8 , BIBREF9 , BIBREF10   In the context of computational morphology, multi-lingual approaches have previously been employed for morphological reinflection BIBREF2 and for paradigm completion BIBREF11   In both of these cases, however, the available datasets covered more languages, 40 and 21, respectively, which allowed for linguistically-motivated language groupings and for parameter sharing directly on the level of characters  BIBREF10 explore parameter sharing between related languages for dependency parsing, and find that sharing is more beneficial in the case of closely related languages 
Conclusions
In this paper we described our system for the CoNLL–SIGMORPHON 2018 shared task on Universal Morphological Reinflection, Task 2, which achieved the best performance out of all systems submitted, an overall accuracy of 49 87  We showed in an ablation study that this is due to three core innovations, which extend a character-based encoder-decoder model: (1) a wide context window, encoding the entire available context; (2) multi-task learning with the auxiliary task of MSD prediction, which acts as a regulariser; (3) a multilingual approach, exploiting information across languages  In future work we aim to gain better understanding of the increase in variance of the results introduced by each of our modifications and the reasons for the varying effect of multi-task learning for different languages 
Acknowledgements
We gratefully acknowledge the support of the NVIDIA Corporation with the donation of the Titan Xp GPU used for this research 

ground_truth: English

Simple English

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What architecture does the encoder have?
Context 4: 1$ for sentence classification based on dev (internal, fold1) scores  We use BIO tagging scheme of NER in FLC task  For CNN, we follow DBLP:conf/emnlp/Kim14 with filter-sizes of [2, 3, 4, 5, 6], 128 filters and 16 batch-size  We compute binary-F1and macro-F1 BIBREF12 in SLC and FLC, respectively on dev (internal) 
Experiments and Evaluation ::: Results: Sentence-Level Propaganda
Table TABREF10 shows the scores on dev (internal and external) for SLC task  Observe that the pre-trained embeddings (FastText or BERT) outperform TF-IDF vector representation  In row r2, we apply logistic regression classifier with BERTSentEmb that leads to improved scores over FastTextSentEmb  Subsequently, we augment the sentence vector with additional features that improves F1 on dev (external), however not dev (internal)  Next, we initialize CNN by FastTextWordEmb or BERTWordEmb and augment the last hidden layer (before classification) with BERTSentEmb and feature vectors, leading to gains in F1 for both the dev sets  Further, we fine-tune BERT and apply different thresholds in relaxing the decision boundary, where $\tau \ge 0 35$ is found optimal 
We choose the three different models in the ensemble: Logistic Regression, CNN and BERT on fold1 and subsequently an ensemble+ of r3, r6 and r12 from each fold1-5 (i e , 15 models) to obtain predictions for dev (external)  We investigate different ensemble schemes (r17-r19), where we observe that the relax-voting improves recall and therefore, the higher F1 (i e , 0 673)  In postprocess step, we check for repetition propaganda technique by computing cosine similarity between the current sentence and its preceding $w=10$ sentence vectors (i e , BERTSentEmb) in the document  If the cosine-similarity is greater than $\lambda \in \lbrace  99,  95\rbrace $, then the current sentence is labeled as propaganda due to repetition  Comparing r19 and r21, we observe a gain in recall, however an overall decrease in F1 applying postprocess 
Finally, we use the configuration of r19 on the test set  The ensemble+ of (r4, r7 r12) was analyzed after test submission  Table TABREF9 (SLC) shows that our submission is ranked at 4th position 
Experiments and Evaluation ::: Results: Fragment-Level Propaganda
Table TABREF11 shows the scores on dev (internal and external) for FLC task  Observe that the features (i e , polarity, POS and NER in row II) when introduced in LSTM-CRF improves F1  We run multi-grained LSTM-CRF without BERTSentEmb (i e , row III) and with it (i e , row IV), where the latter improves scores on dev (internal), however not on dev (external)  Finally, we perform multi-tasking with another auxiliary task of PFD  Given the scores on dev (internal and external) using different configurations (rows I-V), it is difficult to infer the optimal configuration  Thus, we choose the two best configurations (II and IV) on dev (internal) set and build an ensemble+ of predictions (discussed in section SECREF6), leading to a boost in recall and thus an improved F1 on dev (external) 
Finally, we use the ensemble+ of (II and IV) from each of the folds 1-3, i e , $|{\mathcal {M}}|=6$ models to obtain predictions on test  Table TABREF9 (FLC) shows that our submission is ranked at 3rd position 
Conclusion and Future Work
Our system (Team: MIC-CIS) explores different neural architectures (CNN, BERT and LSTM-CRF) with linguistic, layout and topical features to address the tasks of fine-grained propaganda detection  We have demonstrated gains in performance due to the features, ensemble schemes, multi-tasking and multi-granularity architectures  Compared to the other participating systems, our submissions are ranked 3rd and 4th in FLC and SLC tasks, respectively 
In future, we would like to enrich BERT models with linguistic, layout and topical features during their fine-tuning  Further, we would also be interested in understanding and analyzing the neural network learning, i e , extracting salient fragments (or key-phrases) in the sentence that generate propaganda, similar to pankajgupta:2018LISA in order to promote explainable AI 

ground_truth: LSTM

LSTM