prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What was the baseline used?
Context 1: Introduction
Writing errors can occur in many different forms – from relatively simple punctuation and determiner errors, to mistakes including word tense and form, incorrect collocations and erroneous idioms  Automatically identifying all of these errors is a challenging task, especially as the amount of available annotated data is very limited  Rei2016 showed that while some error detection algorithms perform better than others, it is additional training data that has the biggest impact on improving performance 
Being able to generate realistic artificial data would allow for any grammatically correct text to be transformed into annotated examples containing writing errors, producing large amounts of additional training examples  Supervised error generation systems would also provide an efficient method for anonymising the source corpus – error statistics from a private corpus can be aggregated and applied to a different target text, obscuring sensitive information in the original examination scripts  However, the task of creating incorrect data is somewhat more difficult than might initially appear – naive methods for error generation can create data that does not resemble natural errors, thereby making downstream systems learn misleading or uninformative patterns 
Previous work on artificial error generation (AEG) has focused on specific error types, such as prepositions and determiners BIBREF0 , BIBREF1 , or noun number errors BIBREF2   Felice2014a investigated the use of linguistic information when generating artificial data for error correction, but also restricting the approach to only five error types  There has been very limited research on generating artificial data for all types, which is important for general-purpose error detection systems  For example, the error types investigated by Felice2014a cover only 35 74% of all errors present in the CoNLL 2014 training dataset, providing no additional information for the majority of errors 
In this paper, we investigate two supervised approaches for generating all types of artificial errors  We propose a framework for generating errors based on statistical machine translation (SMT), training a model to translate from correct into incorrect sentences  In addition, we describe a method for learning error patterns from an annotated corpus and transplanting them into error-free text  We evaluate the effect of introducing artificial data on two error detection benchmarks  Our results show that each method provides significant improvements over using only the available training set, and a combination of both gives an absolute improvement of 4 3% in INLINEFORM0 , without requiring any additional annotated data 
Error Generation Methods
We investigate two alternative methods for AEG  The models receive grammatically correct text as input and modify certain tokens to produce incorrect sequences  The alternative versions of each sentence are aligned using Levenshtein distance, allowing us to identify specific words that need to be marked as errors  While these alignments are not always perfect, we found them to be sufficient for practical purposes, since alternative alignments of similar sentences often result in the same binary labeling  Future work could explore more advanced alignment methods, such as proposed by felice-bryant-briscoe 
In Section SECREF4 , this automatically labeled data is then used for training error detection models 
Machine Translation
We treat AEG as a translation task – given a correct sentence as input, the system would learn to translate it to contain likely errors, based on a training corpus of parallel data  Existing SMT approaches are already optimised for identifying context patterns that correspond to specific output sequences, which is also required for generating human-like errors  The reverse of this idea, translating from incorrect to correct sentences, has been shown to work well for error correction tasks BIBREF2 , BIBREF3 , and round-trip translation has also been shown to be promising for correcting grammatical errors BIBREF4  
Following previous work BIBREF2 , BIBREF5 , we build a phrase-based SMT error generation system  During training, error-corrected sentences in the training data are treated as the source, and the original sentences written by language learners as the target  Pialign BIBREF6 is used to create a phrase translation table directly from model probabilities  In addition to default features, we add character-level Levenshtein distance to each mapping in the phrase table, as proposed by Felice:2014-CoNLL  Decoding is performed using Moses BIBREF7 and the language model used during decoding is built from the original erroneous sentences in the learner corpus  The IRSTLM Toolkit BIBREF8 is used for building a 5-gram language model with modified Kneser-Ney smoothing BIBREF9  
Pattern Extraction
We also describe a method for AEG using patterns over words and part-of-speech (POS) tags, extracting known incorrect sequences from a corpus of annotated corrections  This approach is based on the best method identified by Felice2014a, using error type distributions; while they covered only 5 error types, we relax this restriction and learn patterns for generating all types of errors 
The original and corrected sentences in the corpus are aligned and used to identify short transformation patterns in the form of (incorrect phrase, correct phrase)  The length of each pattern is the affected phrase, plus up to one token of context on both sides  If a word form changes between the incorrect and correct text, it is fully saved in the pattern, otherwise the POS tags are used for matching 
For example, the original sentence `We went shop on Saturday' and the corrected version `We went shopping on Saturday' would produce the following pattern:
(VVD shop_VV0 II, VVD shopping_VVG II)
After collecting statistics from the background corpus, errors can be inserted into error-free text  The learned patterns are now reversed, looking for the correct side of the tuple in the input sentence  We only use patterns with frequency INLINEFORM0 , which yields a total of 35,625 patterns from our training data  For each input sentence, we first decide how many errors will be generated (using probabilities from the background corpus) and attempt to create them by sampling from the collection of applicable patterns  This process is repeated until all the required errors have been generated or the sentence is exhausted  During generation, we try to balance the distribution of error types as well as keeping the same proportion of incorrect and correct sentences as in the background corpus BIBREF10   The required POS tags were generated with RASP BIBREF11 , using the CLAWS2 tagset 
Error Detection Model
We construct a neural sequence labeling model for error detection, following the previous work BIBREF12 , BIBREF13   The model receives a sequence of tokens as input and outputs a prediction for each position, indicating whether the token is correct or incorrect in the current context  The tokens are first mapped to a distributed vector space, resulting in a sequence of word embeddings  Next, the embeddings are given as input to a bidirectional LSTM BIBREF14 , in order to create context-dependent representations for every token  The hidden states from forward- and backward-LSTMs are concatenated for each word position, resulting in representations that are conditioned on the whole sequence  This concatenated vector is then passed through an additional feedforward layer, and a softmax over the two possible labels (correct and incorrect) is used to output a probability distribution for each token  The model is optimised by minimising categorical cross-entropy with respect to the correct labels  We use AdaDelta BIBREF15 for calculating an adaptive learning rate during training, which accounts for a higher baseline performance compared to previous results 
Evaluation
We trained our error generation models on the public FCE training set BIBREF16 and used them to generate additional artificial training data  Grammatically correct text is needed as the starting point for inserting artificial errors, and we used two different sources: 1) the corrected version of the same FCE training set on which the system is trained (450K tokens), and 2) example sentences extracted from the English Vocabulary Profile (270K tokens)   While there are other text corpora that could be used (e g , Wikipedia and news articles), our development experiments showed that keeping the writing style and vocabulary close to the target domain gives better results compared to simply including more data 
We evaluated our detection models on three benchmarks: the FCE test data (41K tokens) and the two alternative annotations of the CoNLL 2014 Shared Task dataset (30K tokens) BIBREF3   Each artificial error generation system was used to generate 3 different versions of the artificial data, which were then combined with the original annotated dataset and used for training an error detection system  Table TABREF1 contains example sentences from the error generation systems, highlighting each of the edits that are marked as errors 
The error detection results can be seen in Table TABREF4   We use INLINEFORM0 as the main evaluation measure, which was established as the preferred measure for error correction and detection by the CoNLL-14 shared task BIBREF3   INLINEFORM1 calculates a weighted harmonic mean of precision and recall, which assigns twice as much importance to precision – this is motivated by practical applications, where accurate predictions from an error detection system are more important compared to coverage  For comparison, we also report the performance of the error detection system by Rei2016, trained using the same FCE dataset 
The results show that error detection performance is substantially improved by making use of artificially generated data, created by any of the described methods  When comparing the error generation system by Felice2014a (FY14) with our pattern-based (PAT) and machine translation (MT) approaches, we see that the latter methods covering all error types consistently improve performance  While the added error types tend to be less frequent and more complicated to capture, the added coverage is indeed beneficial for error detection  Combining the pattern-based approach with the machine translation system (Ann+PAT+MT) gave the best overall performance on all datasets  The two frameworks learn to generate different types of errors, and taking advantage of both leads to substantial improvements in error detection 
We used the Approximate Randomisation Test BIBREF17 , BIBREF18 to calculate statistical significance and found that the improvement for each of the systems using artificial data was significant over using only manual annotation  In addition, the final combination system is also significantly better compared to the Felice2014a system, on all three datasets  While Rei2016 also report separate experiments that achieve even higher performance, these models were trained on a considerably larger proprietary corpus  In this paper we compare error detection frameworks trained on the same publicly available FCE dataset, thereby removing the confounding factor of dataset size and only focusing on the model architectures 
The error generation methods can generate alternative versions of the same input text – the pattern-based method randomly samples the error locations, and the SMT system can provide an n-best list of alternative translations  Therefore, we also investigated the combination of multiple error-generated versions of the input files when training error detection models  Figure FIGREF6 shows the INLINEFORM0 score on the development set, as the training data is increased by using more translations from the n-best list of the SMT system  These results reveal that allowing the model to see multiple alternative versions of the same file gives a distinct improvement – showing the model both correct and incorrect variations of the same sentences likely assists in learning a discriminative model 
Related Work
Our work builds on prior research into AEG  Brockett2006 constructed regular expressions for transforming correct sentences to contain noun number errors  Rozovskaya2010a learned confusion sets from an annotated corpus in order to generate preposition errors  Foster2009 devised a tool for generating errors for different types using patterns provided by the user or collected automatically from an annotated corpus  However, their method uses a limited number of edit operations and is thus unable to generate complex errors  Cahill2013 compared different training methodologies and showed that artificial errors helped correct prepositions  Felice2014a learned error type distributions for generating five types of errors, and the system in Section SECREF3 is an extension of this model  While previous work focused on generating a specific subset of error types, we explored two holistic approaches to AEG and showed that they are able to significantly improve error detection performance 
Conclusion
This paper investigated two AEG methods, in order to create additional training data for error detection  First, we explored a method using textual patterns learned from an annotated corpus, which are used for inserting errors into correct input text  In addition, we proposed formulating error generation as an MT framework, learning to translate from grammatically correct to incorrect sentences 
The addition of artificial data to the training process was evaluated on three error detection annotations, using the FCE and CoNLL 2014 datasets  Making use of artificial data provided improvements for all data generation methods  By relaxing the type restrictions and generating all types of errors, our pattern-based method consistently outperformed the system by Felice2014a  The combination of the pattern-based method with the machine translation approach gave further substantial improvements and the best performance on all datasets 
Context 2: Introduction
Deep Neural Networks (DNN) have been widely employed in industry for solving various Natural Language Processing (NLP) tasks, such as text classification, sequence labeling, question answering, etc  However, when engineers apply DNN models to address specific NLP tasks, they often face the following challenges 
The above challenges often hinder the productivity of engineers, and result in less optimal solutions to their given tasks  This motivates us to develop an NLP toolkit for DNN models, which facilitates engineers to develop DNN approaches  Before designing this NLP toolkit, we conducted a survey among engineers and identified a spectrum of three typical personas 
To satisfy the requirements of all the above three personas, the NLP toolkit has to be generic enough to cover as many tasks as possible  At the same time, it also needs to be flexible enough to allow alternative network architectures as well as customized modules  Therefore, we analyzed the NLP jobs submitted to a commercial centralized GPU cluster  Table TABREF11 showed that about 87 5% NLP related jobs belong to a few common tasks, including sentence classification, text matching, sequence labeling, MRC, etc  It further suggested that more than 90% of the networks were composed of several common components, such as embedding, CNN/RNN, Transformer and so on 
Based on the above observations, we developed NeuronBlocks, a DNN toolkit for NLP tasks  The basic idea is to provide two layers of support to the engineers  The upper layer targets common NLP tasks  For each task, the toolkit contains several end-to-end network templates, which can be immediately instantiated with simple configuration  The bottom layer consists of a suite of reusable and standard components, which can be adopted as building blocks to construct networks with complex architecture  By following the interface guidelines, users can also contribute to this gallery of components with their own modules 
The technical contributions of NeuronBlocks are summarized into the following three aspects 
Related Work
There are several general-purpose deep learning frameworks, such as TensorFlow, PyTorch and Keras, which have gained popularity in NLP community  These frameworks offer huge flexibility in DNN model design and support various NLP tasks  However, building models under these frameworks requires a large overhead of mastering these framework details  Therefore, higher level abstraction to hide the framework details is favored by many engineers 
There are also several popular deep learning toolkits in NLP, including OpenNMT BIBREF0 , AllenNLP BIBREF1 etc  OpenNMT is an open-source toolkit mainly targeting neural machine translation or other natural language generation tasks  AllenNLP provides several pre-built models for NLP tasks, such as semantic role labeling, machine comprehension, textual entailment, etc  Although these toolkits reduce the development cost, they are limited to certain tasks, and thus not flexible enough to support new network architectures or new components 
Design
The Neuronblocks is built on PyTorch  The overall framework is illustrated in Figure FIGREF16   It consists of two layers: the Block Zoo and the Model Zoo  In Block Zoo, the most commonly used components of deep neural networks are categorized into several groups according to their functions  Within each category, several alternative components are encapsulated into standard and reusable blocks with a consistent interface  These blocks serve as basic and exchangeable units to construct complex network architectures for different NLP tasks  In Model Zoo, the most popular NLP tasks are identified  For each task, several end-to-end network templates are provided in the form of JSON configuration files  Users can simply browse these configurations and choose one or more to instantiate  The whole task can be completed without any coding efforts 
Block Zoo
We recognize the following major functional categories of neural network components  Each category covers as many commonly used modules as possible  The Block Zoo is an open framework, and more modules can be added in the future 
[itemsep= -0 4em,topsep = 0 3em, align=left, labelsep=-0 6em, leftmargin=1 2em]
Embedding Layer: Word/character embedding and extra handcrafted feature embedding such as pos-tagging are supported 
Neural Network Layers: Block zoo provides common layers like RNN, CNN, QRNN BIBREF2 , Transformer BIBREF3 , Highway network, Encoder Decoder architecture, etc  Furthermore, attention mechanisms are widely used in neural networks  Thus we also support multiple attention layers, such as Linear/Bi-linear Attention, Full Attention BIBREF4 , Bidirectional attention flow BIBREF5 , etc  Meanwhile, regularization layers such as Dropout, Layer Norm, Batch Norm, etc are also supported for improving generalization ability 
Loss Function: Besides of the loss functions built in PyTorch, we offer more options such as Focal Loss BIBREF6  
Metrics: For classification task, AUC, Accuracy, Precision/Recall, F1 metrics are supported  For sequence labeling task, F1/Accuracy are supported  For knowledge distillation task, MSE/RMSE are supported  For MRC task, ExactMatch/F1 are supported 
Model Zoo
In NeuronBlocks, we identify four types of most popular NLP tasks  For each task, we provide various end-to-end network templates 
[itemsep= -0 4em,topsep = 0 3em, align=left, labelsep=-0 6em, leftmargin=1 2em]
Text Classification and Matching  Tasks such as domain/intent classification, question answer matching are supported 
Sequence Labeling  Predict each token in a sequence into predefined types  Common tasks include NER, POS tagging, Slot tagging, etc 
Knowledge Distillation BIBREF7   Teacher-Student based knowledge distillation is one common approach for model compression  NeuronBlocks provides knowledge distillation template to improve the inference speed of heavy DNN models like BERT/GPT 
Extractive Machine Reading Comprehension  Given a pair of question and passage, predict the start and end positions of the answer spans in the passage 
User Interface
NeuronBlocks provides convenient user interface for users to build, train, and test DNN models  The details are described in the following 
[itemsep= -0 4em,topsep = 0 3em, align=left, labelsep=-0 6em, leftmargin=1 2em]
I/O interface  This part defines model input/output, such as training data, pre-trained models/embeddings, model saving path, etc 
Model Architecture interface  This is the key part of the configuration file, which defines the whole model architecture  Figure FIGREF19 shows an example of how to specify a model architecture using the blocks in NeuronBlocks  To be more specific, it consists of a list of layers/blocks to construct the architecture, where the blocks are supplied in the gallery of Block Zoo 
Training Parameters interface  In this part, the model optimizer as well as all other training hyper parameters are indicated 
Workflow
Figure FIGREF34 shows the workflow of building DNN models in NeuronBlocks  Users only need to write a JSON configuration file  They can either instantiate an existing template from Model Zoo, or construct a new architecture based on the blocks from Block Zoo  This configuration file is shared across training, test, and prediction  For model hyper-parameter tuning or architecture modification, users just need to change the JSON configuration file  Advanced users can also contribute novel customized blocks into Block Zoo, as long as they follow the same interface guidelines with the existing blocks  These new blocks can be further shared across all users for model architecture design  Moreover, NeuronBlocks has flexible platform support, such as GPU/CPU, GPU management platforms like PAI 
Experiments
To verify the performance of NeuronBlocks, we conducted extensive experiments for common NLP tasks on public data sets including CoNLL-2003 BIBREF14 , GLUE benchmark BIBREF13 , and WikiQA corpus BIBREF15   The experimental results showed that the models built with NeuronBlocks can achieve reliable and competitive results on various tasks, with productivity greatly improved 
Sequence Labeling
For sequence labeling task, we evaluated NeuronBlocks on CoNLL-2003 BIBREF14 English NER dataset, following most works on the same task  This dataset includes four types of named entities, namely, PERSON, LOCATION, ORGANIZATION, and MISC  We adopted the BIOES tagging scheme instead of IOB, as many previous works indicated meaningful improvement with BIOES scheme BIBREF16 , BIBREF17   Table TABREF28 shows the results on CoNLL-2003 Englist testb dataset, with 12 different combinations of network layers/blocks, such as word/character embedding, CNN/LSTM and CRF  The results suggest that the flexible combination of layers/blocks in NeuronBlocks can easily reproduce the performance of original models, with comparative or slightly better performance 
GLUE Benchmark
The General Language Understanding Evaluation (GLUE) benchmark BIBREF13 is a collection of natural language understanding tasks  We experimented on the GLUE benchmark tasks using BiLSTM and Attention based models  As shown in Table TABREF29 , the models built by NeuronBlocks can achieve competitive or even better results on GLUE tasks with minimal coding efforts 
Knowledge Distillation
We evaluated Knowledge Distillation task in NeuronBlocks on a dataset collected from one commercial search engine  We refer to this dataset as Domain Classification Dataset  Each sample in this dataset consists of two parts, i e , a question and a binary label indicating whether the question belongs to a specific domain  Table TABREF36 shows the results, where Area Under Curve (AUC) metric is used as the performance evaluation criteria and Queries per Second (QPS) is used to measure inference speed  By knowledge distillation training approach, the student model by NeuronBlocks managed to get 23-27 times inference speedup with only small performance regression compared with BERTbase fine-tuned classifier 
WikiQA
The WikiQA corpus BIBREF15 is a publicly available dataset for open-domain question answering  This dataset contains 3,047 questions from Bing query logs, each associated with some candidate answer sentences from Wikipedia  We conducted experiments on WikiQA dataset using CNN, BiLSTM, and Attention based models  The results are shown in Table TABREF41   The models built in NeuronBlocks achieved competitive or even better results with simple model configurations 
Conclusion and Future Work
In this paper, we introduce NeuronBlocks, a DNN toolkit for NLP tasks built on PyTorch  NeuronBlocks targets three types of engineers, and provides a two-layer solution to satisfy the requirements from all three types of users  To be more specific, the Model Zoo consists of various templates for the most common NLP tasks, while the Block Zoo supplies a gallery of alternative layers/modules for the networks  Such design achieves a balance between generality and flexibility  Extensive experiments have verified the effectiveness of this approach  NeuronBlocks has been widely used in a product team of a commercial search engine, and significantly improved the productivity for developing NLP DNN approaches 
As an open-source toolkit, we will further extend it in various directions  The following names a few examples 

ground_truth: error detection system by Rei2016

error detection system by Rei2016

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What neural network modules are included in NeuronBlocks?
Context 3: Introduction
Automatic classification of sentiment has mainly focused on categorizing tweets in either two (binary sentiment analysis) or three (ternary sentiment analysis) categories BIBREF0   In this work we study the problem of fine-grained sentiment classification where tweets are classified according to a five-point scale ranging from VeryNegative to VeryPositive  To illustrate this, Table TABREF3 presents examples of tweets associated with each of these categories  Five-point scales are widely adopted in review sites like Amazon and TripAdvisor, where a user's sentiment is ordered with respect to its intensity  From a sentiment analysis perspective, this defines a classification problem with five categories  In particular, Sebastiani et al  BIBREF1 defined such classification problems whose categories are explicitly ordered to be ordinal classification problems  To account for the ordering of the categories, learners are penalized according to how far from the true class their predictions are 
Although considering different scales, the various settings of sentiment classification are related  First, one may use the same feature extraction and engineering approaches to represent the text spans such as word membership in lexicons, morpho-syntactic statistics like punctuation or elongated word counts BIBREF2 , BIBREF3   Second, one would expect that knowledge from one task can be transfered to the others and this would benefit the performance  Knowing that a tweet is “Positive” in the ternary setting narrows the classification decision between the VeryPositive and Positive categories in the fine-grained setting  From a research perspective this raises the question of whether and how one may benefit when tackling such related tasks and how one can transfer knowledge from one task to another during the training phase 
Our focus in this work is to exploit the relation between the sentiment classification settings and demonstrate the benefits stemming from combining them  To this end, we propose to formulate the different classification problems as a multitask learning problem and jointly learn them  Multitask learning BIBREF4 has shown great potential in various domains and its benefits have been empirically validated BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 using different types of data and learning approaches  An important benefit of multitask learning is that it provides an elegant way to access resources developed for similar tasks  By jointly learning correlated tasks, the amount of usable data increases  For instance, while for ternary classification one can label data using distant supervision with emoticons BIBREF9 , there is no straightforward way to do so for the fine-grained problem  However, the latter can benefit indirectly, if the ternary and fine-grained tasks are learned jointly 
The research question that the paper attempts to answer is the following: Can twitter sentiment classification problems, and fine-grained sentiment classification in particular, benefit from multitask learning? To answer the question, the paper brings the following two main contributions: (i) we show how jointly learning the ternary and fine-grained sentiment classification problems in a multitask setting improves the state-of-the-art performance, and (ii) we demonstrate that recurrent neural networks outperform models previously proposed without access to huge corpora while being flexible to incorporate different sources of data 
Multitask Learning for Twitter Sentiment Classification
In his work, Caruana BIBREF4 proposed a multitask approach in which a learner takes advantage of the multiplicity of interdependent tasks while jointly learning them  The intuition is that if the tasks are correlated, the learner can learn a model jointly for them while taking into account the shared information which is expected to improve its generalization ability  People express their opinions online on various subjects (events, products  ), on several languages and in several styles (tweets, paragraph-sized reviews  ), and it is exactly this variety that motivates the multitask approaches  Specifically for Twitter for instance, the different settings of classification like binary, ternary and fine-grained are correlated since their difference lies in the sentiment granularity of the classes which increases while moving from binary to fine-grained problems 
There are two main decisions to be made in our approach: the learning algorithm, which learns a decision function, and the data representation  With respect to the former, neural networks are particularly suitable as one can design architectures with different properties and arbitrary complexity  Also, as training neural network usually relies on back-propagation of errors, one can have shared parts of the network trained by estimating errors on the joint tasks and others specialized for particular tasks  Concerning the data representation, it strongly depends on the data type available  For the task of sentiment classification of tweets with neural networks, distributed embeddings of words have shown great potential  Embeddings are defined as low-dimensional, dense representations of words that can be obtained in an unsupervised fashion by training on large quantities of text BIBREF10  
Concerning the neural network architecture, we focus on Recurrent Neural Networks (RNNs) that are capable of modeling short-range and long-range dependencies like those exhibited in sequence data of arbitrary length like text  While in the traditional information retrieval paradigm such dependencies are captured using INLINEFORM0 -grams and skip-grams, RNNs learn to capture them automatically BIBREF11   To circumvent the problems with capturing long-range dependencies and preventing gradients from vanishing, the long short-term memory network (LSTM) was proposed BIBREF12   In this work, we use an extended version of LSTM called bidirectional LSTM (biLSTM)  While standard LSTMs access information only from the past (previous words), biLSTMs capture both past and future information effectively BIBREF13 , BIBREF11   They consist of two LSTM networks, for propagating text forward and backwards with the goal being to capture the dependencies better  Indeed, previous work on multitask learning showed the effectiveness of biLSTMs in a variety of problems: BIBREF14 tackled sequence prediction, while BIBREF6 and BIBREF15 used biLSTMs for Named Entity Recognition and dependency parsing respectively 
Figure FIGREF2 presents the architecture we use for multitask learning  In the top-left of the figure a biLSTM network (enclosed by the dashed line) is fed with embeddings INLINEFORM0 that correspond to the INLINEFORM1 words of a tokenized tweet  Notice, as discussed above, the biLSTM consists of two LSTMs that are fed with the word sequence forward and backwards  On top of the biLSTM network one (or more) hidden layers INLINEFORM2 transform its output  The output of INLINEFORM3 is led to the softmax layers for the prediction step  There are INLINEFORM4 softmax layers and each is used for one of the INLINEFORM5 tasks of the multitask setting  In tasks such as sentiment classification, additional features like membership of words in sentiment lexicons or counts of elongated/capitalized words can be used to enrich the representation of tweets before the classification step BIBREF3   The lower part of the network illustrates how such sources of information can be incorporated to the process  A vector “Additional Features” for each tweet is transformed from the hidden layer(s) INLINEFORM6 and then is combined by concatenation with the transformed biLSTM output in the INLINEFORM7 layer 
Experimental setup
Our goal is to demonstrate how multitask learning can be successfully applied on the task of sentiment classification of tweets  The particularities of tweets are to be short and informal text spans  The common use of abbreviations, creative language etc , makes the sentiment classification problem challenging  To validate our hypothesis, that learning the tasks jointly can benefit the performance, we propose an experimental setting where there are data from two different twitter sentiment classification problems: a fine-grained and a ternary  We consider the fine-grained task to be our primary task as it is more challenging and obtaining bigger datasets, e g  by distant supervision, is not straightforward and, hence we report the performance achieved for this task 
Ternary and fine-grained sentiment classification were part of the SemEval-2016 “Sentiment Analysis in Twitter” task BIBREF16   We use the high-quality datasets the challenge organizers released  The dataset for fine-grained classification is split in training, development, development_test and test parts  In the rest, we refer to these splits as train, development and test, where train is composed by the training and the development instances  Table TABREF7 presents an overview of the data  As discussed in BIBREF16 and illustrated in the Table, the fine-grained dataset is highly unbalanced and skewed towards the positive sentiment: only INLINEFORM0 of the training examples are labeled with one of the negative classes 
Feature representation We report results using two different feature sets  The first one, dubbed nbow, is a neural bag-of-words that uses text embeddings to generate low-dimensional, dense representations of the tweets

ground_truth: Embedding Layer, Neural Network Layers, Loss Function, Metrics

Embedding Layer, Neural Network Layers, Loss Function, Metrics

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What dataset did they use?
Context 3:  To construct the nbow representation, given the word embeddings dictionary where each word is associated with a vector, we apply the average compositional function that averages the embeddings of the words that compose a tweet  Simple compositional functions like average were shown to be robust and efficient in previous work BIBREF17   Instead of training embeddings from scratch, we use the pre-trained on tweets GloVe embeddings of BIBREF10   In terms of resources required, using only nbow is efficient as it does not require any domain knowledge  However, previous research on sentiment analysis showed that using extra resources, like sentiment lexicons, can benefit significantly the performance BIBREF3 , BIBREF2   To validate this and examine at which extent neural networks and multitask learning benefit from such features we evaluate the models using an augmented version of nbow, dubbed nbow+  The feature space of the latter, is augmented using 1,368 extra features consisting mostly of counts of punctuation symbols ('!?#@'), emoticons, elongated words and word membership features in several sentiment lexicons  Due to space limitations, for a complete presentation of these features, we refer the interested reader to BIBREF2 , whose open implementation we used to extract them 
Evaluation measure To reproduce the setting of the SemEval challenges BIBREF16 , we optimize our systems using as primary measure the macro-averaged Mean Absolute Error ( INLINEFORM0 ) given by: INLINEFORM1
where INLINEFORM0 is the number of categories, INLINEFORM1 is the set of instances whose true class is INLINEFORM2 , INLINEFORM3 is the true label of the instance INLINEFORM4 and INLINEFORM5 the predicted label  The measure penalizes decisions far from the true ones and is macro-averaged to account for the fact that the data are unbalanced  Complementary to INLINEFORM6 , we report the performance achieved on the micro-averaged INLINEFORM7 measure, which is a commonly used measure for classification 
The models To evaluate the multitask learning approach, we compared it with several other models  Support Vector Machines (SVMs) are maximum margin classification algorithms that have been shown to achieve competitive performance in several text classification problems BIBREF16   SVM INLINEFORM0 stands for an SVM with linear kernel and an one-vs-rest approach for the multi-class problem  Also, SVM INLINEFORM1 is an SVM with linear kernel that employs the crammer-singer strategy BIBREF18 for the multi-class problem  Logistic regression (LR) is another type of linear classification method, with probabilistic motivation  Again, we use two types of Logistic Regression depending on the multi-class strategy: LR INLINEFORM2 that uses an one-vs-rest approach and multinomial Logistic Regression also known as the MaxEnt classifier that uses a multinomial criterion 
Both SVMs and LRs as discussed above treat the problem as a multi-class one, without considering the ordering of the classes  For these four models, we tuned the hyper-parameter INLINEFORM0 that controls the importance of the L INLINEFORM1 regularization part in the optimization problem with grid-search over INLINEFORM2 using 10-fold cross-validation in the union of the training and development data and then retrained the models with the selected values  Also, to account for the unbalanced classification problem we used class weights to penalize more the errors made on the rare classes  These weights were inversely proportional to the frequency of each class  For the four models we used the implementations of Scikit-learn BIBREF19  
For multitask learning we use the architecture shown in Figure FIGREF2 , which we implemented with Keras BIBREF20   The embeddings are initialized with the 50-dimensional GloVe embeddings while the output of the biLSTM network is set to dimension 50  The activation function of the hidden layers is the hyperbolic tangent  The weights of the layers were initialized from a uniform distribution, scaled as described in BIBREF21   We used the Root Mean Square Propagation optimization method  We used dropout for regularizing the network  We trained the network using batches of 128 examples as follows: before selecting the batch, we perform a Bernoulli trial with probability INLINEFORM0 to select the task to train for  With probability INLINEFORM1 we pick a batch for the fine-grained sentiment classification problem, while with probability INLINEFORM2 we pick a batch for the ternary problem  As shown in Figure FIGREF2 , the error is backpropagated until the embeddings, that we fine-tune during the learning process  Notice also that the weights of the network until the layer INLINEFORM3 are shared and therefore affected by both tasks 
To tune the neural network hyper-parameters we used 5-fold cross validation  We tuned the probability INLINEFORM0 of dropout after the hidden layers INLINEFORM1 and for the biLSTM for INLINEFORM2 , the size of the hidden layer INLINEFORM3 and the probability INLINEFORM4 of the Bernoulli trials from INLINEFORM5   During training, we monitor the network's performance on the development set and apply early stopping if the performance on the validation set does not improve for 5 consecutive epochs 
Experimental results Table TABREF9 illustrates the performance of the models for the different data representations  The upper part of the Table summarizes the performance of the baselines  The entry “Balikas et al ” stands for the winning system of the 2016 edition of the challenge BIBREF2 , which to the best of our knowledge holds the state-of-the-art  Due to the stochasticity of training the biLSTM models, we repeat the experiment 10 times and report the average and the standard deviation of the performance achieved 
Several observations can be made from the table  First notice that, overall, the best performance is achieved by the neural network architecture that uses multitask learning  This entails that the system makes use of the available resources efficiently and improves the state-of-the-art performance  In conjunction with the fact that we found the optimal probability INLINEFORM0 , this highlights the benefits of multitask learning over single task learning  Furthermore, as described above, the neural network-based models have only access to the training data as the development are hold for early stopping  On the other hand, the baseline systems were retrained on the union of the train and development sets  Hence, even with fewer resources available for training on the fine-grained problem, the neural networks outperform the baselines  We also highlight the positive effect of the additional features that previous research proposed  Adding the features both in the baselines and in the biLSTM-based architectures improves the INLINEFORM1 scores by several points 
Lastly, we compare the performance of the baseline systems with the performance of the state-of-the-art system of BIBREF2   While BIBREF2 uses n-grams (and character-grams) with INLINEFORM0 , the baseline systems (SVMs, LRs) used in this work use the nbow+ representation, that relies on unigrams  Although they perform on par, the competitive performance of nbow highlights the potential of distributed representations for short-text classification  Further, incorporating structure and distributed representations leads to the gains of the biLSTM network, in the multitask and single task setting 
Similar observations can be drawn from Figure FIGREF10 that presents the INLINEFORM0 scores  Again, the biLSTM network with multitask learning achieves the best performance  It is also to be noted that although the two evaluation measures are correlated in the sense that the ranking of the models is the same, small differences in the INLINEFORM1 have large effect on the scores of the INLINEFORM2 measure 
Conclusion
In this paper, we showed that by jointly learning the tasks of ternary and fine-grained classification with a multitask learning model, one can greatly improve the performance on the second  This opens several avenues for future research  Since sentiment is expressed in different textual types like tweets and paragraph-sized reviews, in different languages (English, German,   ) and in different granularity levels (binary, ternary,  ) one can imagine multitask approaches that could benefit from combining such resources  Also, while we opted for biLSTM networks here, one could use convolutional neural networks or even try to combine different types of networks and tasks to investigate the performance effect of multitask learning  Lastly, while our approach mainly relied on the foundations of BIBREF4 , the internal mechanisms and the theoretical guarantees of multitask learning remain to be better understood 
Acknowledgements
This work is partially supported by the CIFRE N 28/2015 

ground_truth:  high-quality datasets  from SemEval-2016 “Sentiment Analysis in Twitter” task

 SemEval-2016 “Sentiment Analysis in Twitter”