prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What is best performing model among author's submissions, what performance it had?
Context 1: Introduction
In the age of information dissemination without quality control, it has enabled malicious users to spread misinformation via social media and aim individual users with propaganda campaigns to achieve political and financial gains as well as advance a specific agenda  Often disinformation is complied in the two major forms: fake news and propaganda, where they differ in the sense that the propaganda is possibly built upon true information (e g , biased, loaded language, repetition, etc ) 
Prior works BIBREF0, BIBREF1, BIBREF2 in detecting propaganda have focused primarily at document level, typically labeling all articles from a propagandistic news outlet as propaganda and thus, often non-propagandistic articles from the outlet are mislabeled  To this end, EMNLP19DaSanMartino focuses on analyzing the use of propaganda and detecting specific propagandistic techniques in news articles at sentence and fragment level, respectively and thus, promotes explainable AI  For instance, the following text is a propaganda of type `slogan' 
Trump tweeted: $\underbrace{\text{`}`{\texttt {BUILD THE WALL!}"}}_{\text{slogan}}$
Shared Task: This work addresses the two tasks in propaganda detection BIBREF3 of different granularities: (1) Sentence-level Classification (SLC), a binary classification that predicts whether a sentence contains at least one propaganda technique, and (2) Fragment-level Classification (FLC), a token-level (multi-label) classification that identifies both the spans and the type of propaganda technique(s) 
Contributions: (1) To address SLC, we design an ensemble of different classifiers based on Logistic Regression, CNN and BERT, and leverage transfer learning benefits using the pre-trained embeddings/models from FastText and BERT  We also employed different features such as linguistic (sentiment, readability, emotion, part-of-speech and named entity tags, etc ), layout, topics, etc  (2) To address FLC, we design a multi-task neural sequence tagger based on LSTM-CRF and linguistic features to jointly detect propagandistic fragments and its type  Moreover, we investigate performing FLC and SLC jointly in a multi-granularity network based on LSTM-CRF and BERT  (3) Our system (MIC-CIS) is ranked 3rd (out of 12 participants) and 4th (out of 25 participants) in FLC and SLC tasks, respectively 
System Description ::: Linguistic, Layout and Topical Features
Some of the propaganda techniques BIBREF3 involve word and phrases that express strong emotional implications, exaggeration, minimization, doubt, national feeling, labeling , stereotyping, etc  This inspires us in extracting different features (Table TABREF1) including the complexity of text, sentiment, emotion, lexical (POS, NER, etc ), layout, etc  To further investigate, we use topical features (e g , document-topic proportion) BIBREF4, BIBREF5, BIBREF6 at sentence and document levels in order to determine irrelevant themes, if introduced to the issue being discussed (e g , Red Herring) 
For word and sentence representations, we use pre-trained vectors from FastText BIBREF7 and BERT BIBREF8 
System Description ::: Sentence-level Propaganda Detection
Figure FIGREF2 (left) describes the three components of our system for SLC task: features, classifiers and ensemble  The arrows from features-to-classifier indicate that we investigate linguistic, layout and topical features in the two binary classifiers: LogisticRegression and CNN  For CNN, we follow the architecture of DBLP:conf/emnlp/Kim14 for sentence-level classification, initializing the word vectors by FastText or BERT  We concatenate features in the last hidden layer before classification 
One of our strong classifiers includes BERT that has achieved state-of-the-art performance on multiple NLP benchmarks  Following DBLP:conf/naacl/DevlinCLT19, we fine-tune BERT for binary classification, initializing with a pre-trained model (i e , BERT-base, Cased)  Additionally, we apply a decision function such that a sentence is tagged as propaganda if prediction probability of the classifier is greater than a threshold ($\tau $)  We relax the binary decision boundary to boost recall, similar to pankajgupta:CrossRE2019 
Ensemble of Logistic Regression, CNN and BERT: In the final component, we collect predictions (i e , propaganda label) for each sentence from the three ($\mathcal {M}=3$) classifiers and thus, obtain $\mathcal {M}$ number of predictions for each sentence  We explore two ensemble strategies (Table TABREF1): majority-voting and relax-voting to boost precision and recall, respectively 
System Description ::: Fragment-level Propaganda Detection
Figure FIGREF2 (right) describes our system for FLC task, where we design sequence taggers BIBREF9, BIBREF10 in three modes: (1) LSTM-CRF BIBREF11 with word embeddings ($w\_e$) and character embeddings $c\_e$, token-level features ($t\_f$) such as polarity, POS, NER, etc  (2) LSTM-CRF+Multi-grain that jointly performs FLC and SLC with FastTextWordEmb and BERTSentEmb, respectively  Here, we add binary sentence classification loss to sequence tagging weighted by a factor of $\alpha $  (3) LSTM-CRF+Multi-task that performs propagandistic span/fragment detection (PFD) and FLC (fragment detection + 19-way classification) 
Ensemble of Multi-grain, Multi-task LSTM-CRF with BERT: Here, we build an ensemble by considering propagandistic fragments (and its type) from each of the sequence taggers  In doing so, we first perform majority voting at the fragment level for the fragment where their spans exactly overlap  In case of non-overlapping fragments, we consider all  However, when the spans overlap (though with the same label), we consider the fragment with the largest span 
Experiments and Evaluation
Data: While the SLC task is binary, the FLC consists of 18 propaganda techniques BIBREF3  We split (80-20%) the annotated corpus into 5-folds and 3-folds for SLC and FLC tasks, respectively  The development set of each the folds is represented by dev (internal); however, the un-annotated corpus used in leaderboard comparisons by dev (external)  We remove empty and single token sentences after tokenization  Experimental Setup: We use PyTorch framework for the pre-trained BERT model (Bert-base-cased), fine-tuned for SLC task  In the multi-granularity loss, we set $\alpha = 0 1$ for sentence classification based on dev (internal, fold1) scores  We use BIO tagging scheme of NER in FLC task  For CNN, we follow DBLP:conf/emnlp/Kim14 with filter-sizes of [2, 3, 4, 5, 6], 128 filters and 16 batch-size  We compute binary-F1and macro-F1 BIBREF12 in SLC and FLC, respectively on dev (internal) 
Experiments and Evaluation ::: Results: Sentence-Level Propaganda
Table TABREF10 shows the scores on dev (internal and external) for SLC task  Observe that the pre-trained embeddings (FastText or BERT) outperform TF-IDF vector representation  In row r2, we apply logistic regression classifier with BERTSentEmb that leads to improved scores over FastTextSentEmb  Subsequently, we augment the sentence vector with additional features that improves F1 on dev (external), however not dev (internal)  Next, we initialize CNN by FastTextWordEmb or BERTWordEmb and augment the last hidden layer (before classification) with BERTSentEmb and feature vectors, leading to gains in F1 for both the dev sets  Further, we fine-tune BERT and apply different thresholds in relaxing the decision boundary, where $\tau \ge 0 35$ is found optimal 
We choose the three different models in the ensemble: Logistic Regression, CNN and BERT on fold1 and subsequently an ensemble+ of r3, r6 and r12 from each fold1-5 (i e , 15 models) to obtain predictions for dev (external)  We investigate different ensemble schemes (r17-r19), where we observe that the relax-voting improves recall and therefore, the higher F1 (i e , 0 673)  In postprocess step, we check for repetition propaganda technique by computing cosine similarity between the current sentence and its preceding $w=10$ sentence vectors (i e , BERTSentEmb) in the document  If the cosine-similarity is greater than $\lambda \in \lbrace  99,  95\rbrace $, then the current sentence is labeled as propaganda due to repetition  Comparing r19 and r21, we observe a gain in recall, however an overall decrease in F1 applying postprocess 
Finally, we use the configuration of r19 on the test set  The ensemble+ of (r4, r7 r12) was analyzed after test submission  Table TABREF9 (SLC) shows that our submission is ranked at 4th position 
Experiments and Evaluation ::: Results: Fragment-Level Propaganda
Table TABREF11 shows the scores on dev (internal and external) for FLC task  Observe that the features (i e , polarity, POS and NER in row II) when introduced in LSTM-CRF improves F1  We run multi-grained LSTM-CRF without BERTSentEmb (i e , row III) and with it (i e , row IV), where the latter improves scores on dev (internal), however not on dev (external)  Finally, we perform multi-tasking with another auxiliary task of PFD  Given the scores on dev (internal and external) using different configurations (rows I-V), it is difficult to infer the optimal configuration  Thus, we choose the two best configurations (II and IV) on dev (internal) set and build an ensemble+ of predictions (discussed in section SECREF6), leading to a boost in recall and thus an improved F1 on dev (external) 
Finally, we use the ensemble+ of (II and IV) from each of the folds 1-3, i e , $|{\mathcal {M}}|=6$ models to obtain predictions on test  Table TABREF9 (FLC) shows that our submission is ranked at 3rd position 
Conclusion and Future Work
Our system (Team: MIC-CIS) explores different neural architectures (CNN, BERT and LSTM-CRF) with linguistic, layout and topical features to address the tasks of fine-grained propaganda detection  We have demonstrated gains in performance due to the features, ensemble schemes, multi-tasking and multi-granularity architectures  Compared to the other participating systems, our submissions are ranked 3rd and 4th in FLC and SLC tasks, respectively 
In future, we would like to enrich BERT models with linguistic, layout and topical features during their fine-tuning  Further, we would also be interested in understanding and analyzing the neural network learning, i e , extracting salient fragments (or key-phrases) in the sentence that generate propaganda, similar to pankajgupta:2018LISA in order to promote explainable AI 
Context 2: Introduction
Abusive language refers to any type of insult, vulgarity, or profanity that debases the target; it also can be anything that causes aggravation BIBREF0 , BIBREF1   Abusive language is often reframed as, but not limited to, offensive language BIBREF2 , cyberbullying BIBREF3 , othering language BIBREF4 , and hate speech BIBREF5  
Recently, an increasing number of users have been subjected to harassment, or have witnessed offensive behaviors online BIBREF6   Major social media companies (i e  Facebook, Twitter) have utilized multiple resources—artificial intelligence, human reviewers, user reporting processes, etc —in effort to censor offensive language, yet it seems nearly impossible to successfully resolve the issue BIBREF7 , BIBREF8  
The major reason of the failure in abusive language detection comes from its subjectivity and context-dependent characteristics BIBREF9   For instance, a message can be regarded as harmless on its own, but when taking previous threads into account it may be seen as abusive, and vice versa  This aspect makes detecting abusive language extremely laborious even for human annotators; therefore it is difficult to build a large and reliable dataset BIBREF10  
Previously, datasets openly available in abusive language detection research on Twitter ranged from 10K to 35K in size BIBREF9 , BIBREF11   This quantity is not sufficient to train the significant number of parameters in deep learning models  Due to this reason, these datasets have been mainly studied by traditional machine learning methods  Most recently, Founta et al  founta2018large introduced Hate and Abusive Speech on Twitter, a dataset containing 100K tweets with cross-validated labels  Although this corpus has great potential in training deep models with its significant size, there are no baseline reports to date 
This paper investigates the efficacy of different learning models in detecting abusive language  We compare accuracy using the most frequently studied machine learning classifiers as well as recent neural network models  Reliable baseline results are presented with the first comparative study on this dataset  Additionally, we demonstrate the effect of different features and variants, and describe the possibility for further improvements with the use of ensemble models 
Related Work
The research community introduced various approaches on abusive language detection  Razavi et al  razavi2010offensive applied Naïve Bayes, and Warner and Hirschberg warner2012detecting used Support Vector Machine (SVM), both with word-level features to classify offensive language  Xiang et al  xiang2012detecting generated topic distributions with Latent Dirichlet Allocation BIBREF12 , also using word-level features in order to classify offensive tweets 
More recently, distributed word representations and neural network models have been widely applied for abusive language detection  Djuric et al  djuric2015hate used the Continuous Bag Of Words model with paragraph2vec algorithm BIBREF13 to more accurately detect hate speech than that of the plain Bag Of Words models  Badjatiya et al  badjatiya2017deep implemented Gradient Boosted Decision Trees classifiers using word representations trained by deep learning models  Other researchers have investigated character-level representations and their effectiveness compared to word-level representations BIBREF14 , BIBREF15  
As traditional machine learning methods have relied on feature engineering, (i e  n-grams, POS tags, user information) BIBREF1 , researchers have proposed neural-based models with the advent of larger datasets  Convolutional Neural Networks and Recurrent Neural Networks have been applied to detect abusive language, and they have outperformed traditional machine learning classifiers such as Logistic Regression and SVM BIBREF15 , BIBREF16   However, there are no studies investigating the efficiency of neural models with large-scale datasets over 100K 
Methodology
This section illustrates our implementations on traditional machine learning classifiers and neural network based models in detail  Furthermore, we describe additional features and variant models investigated 
Traditional Machine Learning Models
We implement five feature engineering based machine learning classifiers that are most often used for abusive language detection  In data preprocessing, text sequences are converted into Bag Of Words (BOW) representations, and normalized with Term Frequency-Inverse Document Frequency (TF-IDF) values  We experiment with word-level features using n-grams ranging from 1 to 3, and character-level features from 3 to 8-grams  Each classifier is implemented with the following specifications:
Naïve Bayes (NB): Multinomial NB with additive smoothing constant 1
Logistic Regression (LR): Linear LR with L2 regularization constant 1 and limited-memory BFGS optimization
Support Vector Machine (SVM): Linear SVM with L2 regularization constant 1 and logistic loss function
Random Forests (RF): Averaging probabilistic predictions of 10 randomized decision trees
Gradient Boosted Trees (GBT): Tree boosting with learning rate 1 and logistic loss function
Neural Network based Models
Along with traditional machine learning approaches, we investigate neural network based models to evaluate their efficacy within a larger dataset  In particular, we explore Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), and their variant models  A pre-trained GloVe BIBREF17 representation is used for word-level features 
CNN: We adopt Kim's kim2014convolutional implementation as the baseline  The word-level CNN models have 3 convolutional filters of different sizes [1,2,3] with ReLU activation, and a max-pooling layer  For the character-level CNN, we use 6 convolutional filters of various sizes [3,4,5,6,7,8], then add max-pooling layers followed by 1 fully-connected layer with a dimension of 1024 
Park and Fung park2017one proposed a HybridCNN model which outperformed both word-level and character-level CNNs in abusive language detection  In order to evaluate the HybridCNN for this dataset, we concatenate the output of max-pooled layers from word-level and character-level CNN, and feed this vector to a fully-connected layer in order to predict the output 
All three CNN models (word-level, character-level, and hybrid) use cross entropy with softmax as their loss function and Adam BIBREF18 as the optimizer 
RNN: We use bidirectional RNN BIBREF19 as the baseline, implementing a GRU BIBREF20 cell for each recurrent unit  From extensive parameter-search experiments, we chose 1 encoding layer with 50 dimensional hidden states and an input dropout probability of 0 3  The RNN models use cross entropy with sigmoid as their loss function and Adam as the optimizer 
For a possible improvement, we apply a self-matching attention mechanism on RNN baseline models BIBREF21 so that they may better understand the data by retrieving text sequences twice  We also investigate a recently introduced method, Latent Topic Clustering (LTC) BIBREF22   The LTC method extracts latent topic information from the hidden states of RNN, and uses it for additional information in classifying the text data 
Feature Extension
While manually analyzing the raw dataset, we noticed that looking at the tweet one has replied to or has quoted, provides significant contextual information  We call these, “context tweets"

ground_truth: For SLC task, the "ltuorp" team  has the best performing  model (0.6323/0.6028/0.6649 for F1/P/R  respectively) and for FLC task the "newspeak" team  has the best performing  model (0.2488/0.2863/0.2201 for F1/P/R respectively).

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What additional features and context are proposed?
Context 2:  As humans can better understand a tweet with the reference of its context, our assumption is that computers also benefit from taking context tweets into account in detecting abusive language 
As shown in the examples below, (2) is labeled abusive due to the use of vulgar language  However, the intention of the user can be better understood with its context tweet (1) 
(1) I hate when I'm sitting in front of the bus and somebody with a wheelchair get on 
INLINEFORM0 (2) I hate it when I'm trying to board a bus and there's already an as**ole on it 
Similarly, context tweet (3) is important in understanding the abusive tweet (4), especially in identifying the target of the malice 
(3) Survivors of #Syria Gas Attack Recount `a Cruel Scene' 
INLINEFORM0 (4) Who the HELL is “LIKE" ING this post? Sick people    
Huang et al  huang2016modeling used several attributes of context tweets for sentiment analysis in order to improve the baseline LSTM model  However, their approach was limited because the meta-information they focused on—author information, conversation type, use of the same hashtags or emojis—are all highly dependent on data 
In order to avoid data dependency, text sequences of context tweets are directly used as an additional feature of neural network models  We use the same baseline model to convert context tweets to vectors, then concatenate these vectors with outputs of their corresponding labeled tweets  More specifically, we concatenate max-pooled layers of context and labeled tweets for the CNN baseline model  As for RNN, the last hidden states of context and labeled tweets are concatenated 
Dataset
Hate and Abusive Speech on Twitter BIBREF10 classifies tweets into 4 labels, “normal", “spam", “hateful" and “abusive"  We were only able to crawl 70,904 tweets out of 99,996 tweet IDs, mainly because the tweet was deleted or the user account had been suspended  Table shows the distribution of labels of the crawled data 
Data Preprocessing
In the data preprocessing steps, user IDs, URLs, and frequently used emojis are replaced as special tokens  Since hashtags tend to have a high correlation with the content of the tweet BIBREF23 , we use a segmentation library BIBREF24 for hashtags to extract more information 
For character-level representations, we apply the method Zhang et al  zhang2015character proposed  Tweets are transformed into one-hot encoded vectors using 70 character dimensions—26 lower-cased alphabets, 10 digits, and 34 special characters including whitespace 
Training and Evaluation
In training the feature engineering based machine learning classifiers, we truncate vector representations according to the TF-IDF values (the top 14,000 and 53,000 for word-level and character-level representations, respectively) to avoid overfitting  For neural network models, words that appear only once are replaced as unknown tokens 
Since the dataset used is not split into train, development, and test sets, we perform 10-fold cross validation, obtaining the average of 5 tries; we divide the dataset randomly by a ratio of 85:5:10, respectively  In order to evaluate the overall performance, we calculate the weighted average of precision, recall, and F1 scores of all four labels, “normal”, “spam”, “hateful”, and “abusive” 
Empirical Results
As shown in Table , neural network models are more accurate than feature engineering based models (i e  NB, SVM, etc ) except for the LR model—the best LR model has the same F1 score as the best CNN model 
Among traditional machine learning models, the most accurate in classifying abusive language is the LR model followed by ensemble models such as GBT and RF  Character-level representations improve F1 scores of SVM and RF classifiers, but they have no positive effect on other models 
For neural network models, RNN with LTC modules have the highest accuracy score, but there are no significant improvements from its baseline model and its attention-added model  Similarly, HybridCNN does not improve the baseline CNN model  For both CNN and RNN models, character-level features significantly decrease the accuracy of classification 
The use of context tweets generally have little effect on baseline models, however they noticeably improve the scores of several metrics  For instance, CNN with context tweets score the highest recall and F1 for “hateful" labels, and RNN models with context tweets have the highest recall for “abusive" tweets 
Discussion and Conclusion
While character-level features are known to improve the accuracy of neural network models BIBREF16 , they reduce classification accuracy for Hate and Abusive Speech on Twitter  We conclude this is because of the lack of labeled data as well as the significant imbalance among the different labels  Unlike neural network models, character-level features in traditional machine learning classifiers have positive results because we have trained the models only with the most significant character elements using TF-IDF values 
Variants of neural network models also suffer from data insufficiency  However, these models show positive performances on “spam" (14%) and “hateful" (4%) tweets—the lower distributed labels  The highest F1 score for “spam" is from the RNN-LTC model (0 551), and the highest for “hateful" is CNN with context tweets (0 309)  Since each variant model excels in different metrics, we expect to see additional improvements with the use of ensemble models of these variants in future works 
In this paper, we report the baseline accuracy of different learning models as well as their variants on the recently introduced dataset, Hate and Abusive Speech on Twitter  Experimental results show that bidirectional GRU networks with LTC provide the most accurate results in detecting abusive language  Additionally, we present the possibility of using ensemble models of variant models and features for further improvements 
Acknowledgments
K  Jung is with the Department of Electrical and Computer Engineering, ASRI, Seoul National University, Seoul, Korea  This work was supported by the National Research Foundation of Korea (NRF) funded by the Korea government (MSIT) (No  2016M3C4A7952632), the Technology Innovation Program (10073144) funded by the Ministry of Trade, Industry & Energy (MOTIE, Korea) 
We would also like to thank Yongkeun Hwang and Ji Ho Park for helpful discussions and their valuable insights 
Context 3: Introduction
Understanding the emotions expressed in a text or message is of high relevance nowadays  Companies are interested in this to get an understanding of the sentiment of their current customers regarding their products and the sentiment of their potential customers to attract new ones  Moreover, changes in a product or a company may also affect the sentiment of a customer  However, the intensity of an emotion is crucial in determining the urgency and importance of that sentiment  If someone is only slightly happy about a product, is a customer willing to buy it again? Conversely, if someone is very angry about customer service, his or her complaint might be given priority over somewhat milder complaints 
BIBREF0 present four tasks in which systems have to automatically determine the intensity of emotions (EI) or the intensity of the sentiment (Valence) of tweets in the languages English, Arabic, and Spanish  The goal is to either predict a continuous regression (reg) value or to do ordinal classification (oc) based on a number of predefined categories  The EI tasks have separate training sets for four different emotions: anger, fear, joy and sadness  Due to the large number of subtasks and the fact that this language does not have many resources readily available, we only focus on the Spanish subtasks  Our work makes the following contributions:
Our submissions ranked second (EI-Reg), second (EI-Oc), fourth (V-Reg) and fifth (V-Oc), demonstrating that the proposed method is accurate in automatically determining the intensity of emotions and sentiment of Spanish tweets  This paper will first focus on the datasets, the data generation procedure, and the techniques and tools used  Then we present the results in detail, after which we perform a small error analysis on the largest mistakes our model made  We conclude with some possible ideas for future work 
Data
For each task, the training data that was made available by the organizers is used, which is a selection of tweets with for each tweet a label describing the intensity of the emotion or sentiment BIBREF1   Links and usernames were replaced by the general tokens URL and @username, after which the tweets were tokenized by using TweetTokenizer  All text was lowercased  In a post-processing step, it was ensured that each emoji is tokenized as a single token 
Word Embeddings
To be able to train word embeddings, Spanish tweets were scraped between November 8, 2017 and January 12, 2018  We chose to create our own embeddings instead of using pre-trained embeddings, because this way the embeddings would resemble the provided data set: both are based on Twitter data  Added to this set was the Affect in Tweets Distant Supervision Corpus (DISC) made available by the organizers BIBREF0 and a set of 4 1 million tweets from 2015, obtained from BIBREF2   After removing duplicate tweets and tweets with fewer than ten tokens, this resulted in a set of 58 7 million tweets, containing 1 1 billion tokens  The tweets were preprocessed using the method described in Section SECREF6   The word embeddings were created using word2vec in the gensim library BIBREF3 , using CBOW, a window size of 40 and a minimum count of 5  The feature vectors for each tweet were then created by using the AffectiveTweets WEKA package BIBREF4  
Translating Lexicons
Most lexical resources for sentiment analysis are in English  To still be able to benefit from these sources, the lexicons in the AffectiveTweets package were translated to Spanish, using the machine translation platform Apertium BIBREF5  
All lexicons from the AffectiveTweets package were translated, except for SentiStrength  Instead of translating this lexicon, the English version was replaced by the Spanish variant made available by BIBREF6  
For each subtask, the optimal combination of lexicons was determined  This was done by first calculating the benefits of adding each lexicon individually, after which only beneficial lexicons were added until the score did not increase anymore (e g  after adding the best four lexicons the fifth one did not help anymore, so only four were added)  The tests were performed using a default SVM model, with the set of word embeddings described in the previous section  Each subtask thus uses a different set of lexicons (see Table TABREF1 for an overview of the lexicons used in our final ensemble)  For each subtask, this resulted in a (modest) increase on the development set, between 0 01 and 0 05 
Translating Data
The training set provided by BIBREF0 is not very large, so it was interesting to find a way to augment the training set  A possible method is to simply translate the datasets into other languages, leaving the labels intact  Since the present study focuses on Spanish tweets, all tweets from the English datasets were translated into Spanish  This new set of “Spanish” data was then added to our original training set  Again, the machine translation platform Apertium BIBREF5 was used for the translation of the datasets 
Algorithms Used
Three types of models were used in our system, a feed-forward neural network, an LSTM network and an SVM regressor  The neural nets were inspired by the work of Prayas BIBREF7 in the previous shared task  Different regression algorithms (e g  AdaBoost, XGBoost) were also tried due to the success of SeerNet BIBREF8 , but our study was not able to reproduce their results for Spanish 
For both the LSTM network and the feed-forward network, a parameter search was done for the number of layers, the number of nodes and dropout used  This was done for each subtask, i e  different tasks can have a different number of layers  All models were implemented using Keras BIBREF9   After the best parameter settings were found, the results of 10 system runs to produce our predictions were averaged (note that this is different from averaging our different type of models in Section SECREF16 )  For the SVM (implemented in scikit-learn BIBREF10 ), the RBF kernel was used and a parameter search was conducted for epsilon  Detailed parameter settings for each subtask are shown in Table TABREF12   Each parameter search was performed using 10-fold cross validation, as to not overfit on the development set 
Semi-supervised Learning
One of the aims of this study was to see if using semi-supervised learning is beneficial for emotion intensity tasks  For this purpose, the DISC BIBREF0 corpus was used  This corpus was created by querying certain emotion-related words, which makes it very suitable as a semi-supervised corpus  However, the specific emotion the tweet belonged to was not made public  Therefore, a method was applied to automatically assign the tweets to an emotion by comparing our scraped tweets to this new data set 
First, in an attempt to obtain the query-terms, we selected the 100 words which occurred most frequently in the DISC corpus, in comparison with their frequencies in our own scraped tweets corpus  Words that were clearly not indicators of emotion were removed  The rest was annotated per emotion or removed if it was unclear to which emotion the word belonged  This allowed us to create silver datasets per emotion, assigning tweets to an emotion if an annotated emotion-word occurred in the tweet 
Our semi-supervised approach is quite straightforward: first a model is trained on the training set and then this model is used to predict the labels of the silver data  This silver data is then simply added to our training set, after which the model is retrained  However, an extra step is applied to ensure that the silver data is of reasonable quality  Instead of training a single model initially, ten different models were trained which predict the labels of the silver instances  If the highest and lowest prediction do not differ more than a certain threshold the silver instance is maintained, otherwise it is discarded 
This results in two parameters that could be optimized: the threshold and the number of silver instances that would be added  This method can be applied to both the LSTM and feed-forward networks that were used  An overview of the characteristics of our data set with the final parameter settings is shown in Table TABREF14   Usually, only a small subset of data was added to our training set, meaning that most of the silver data is not used in the experiments  Note that since only the emotions were annotated, this method is only applicable to the EI tasks 
Ensembling
To boost performance, the SVM, LSTM, and feed-forward models were combined into an ensemble  For both the LSTM and feed-forward approach, three different models were trained  The first model was trained on the training data (regular), the second model was trained on both the training and translated training data (translated) and the third one was trained on both the training data and the semi-supervised data (silver)  Due to the nature of the SVM algorithm, semi-supervised learning does not help, so only the regular and translated model were trained in this case  This results in 8 different models per subtask  Note that for the valence tasks no silver training data was obtained, meaning that for those tasks the semi-supervised models could not be used 
Per task, the LSTM and feed-forward model's predictions were averaged over 10 prediction runs  Subsequently, the predictions of all individual models were combined into an average  Finally, models were removed from the ensemble in a stepwise manner if the removal increased the average score  This was done based on their original scores, i e  starting out by trying to remove the worst individual model and working our way up to the best model  We only consider it an increase in score if the difference is larger than 0 002 (i e  the difference between 0 716 and 0 718)  If at some point the score does not increase and we are therefore unable to remove a model, the process is stopped and our best ensemble of models has been found  This process uses the scores on the development set of different combinations of models  Note that this means that the ensembles for different subtasks can contain different sets of models  The final model selections can be found in Table TABREF17  
Results and Discussion
Table TABREF18 shows the results on the development set of all individuals models, distinguishing the three types of training: regular (r), translated (t) and semi-supervised (s)  In Tables TABREF17 and TABREF18 , the letter behind each model (e g  SVM-r, LSTM-r) corresponds to the type of training used  Comparing the regular and translated columns for the three algorithms, it shows that in 22 out of 30 cases, using translated instances as extra training data resulted in an improvement  For the semi-supervised learning approach, an improvement is found in 15 out of 16 cases  Moreover, our best individual model for each subtask (bolded scores in Table TABREF18 ) is always either a translated or semi-supervised model  Table TABREF18 also shows that, in general, our feed-forward network obtained the best results, having the highest F-score for 8 out of 10 subtasks 
However, Table TABREF19 shows that these scores can still be improved by averaging or ensembling the individual models  On the dev set, averaging our 8 individual models results in a better score for 8 out of 10 subtasks, while creating an ensemble beats all of the individual models as well as the average for each subtask  On the test set, however, only a small increase in score (if any) is found for stepwise ensembling, compared to averaging  Even though the results do not get worse, we cannot conclude that stepwise ensembling is a better method than simply averaging 
Our official scores (column Ens Test in Table TABREF19 ) have placed us second (EI-Reg, EI-Oc), fourth (V-Reg) and fifth (V-Oc) on the SemEval AIT-2018 leaderboard  However, it is evident that the results obtained on the test set are not always in line with those achieved on the development set  Especially on the anger subtask for both EI-Reg and EI-Oc, the scores are considerably lower on the test set in comparison with the results on the development set  Therefore, a small error analysis was performed on the instances where our final model made the largest errors 
Error Analysis
Due to some large differences between our results on the dev and test set of this task, we performed a small error analysis in order to see what caused these differences  For EI-Reg-anger, the gold labels were compared to our own predictions, and we manually checked 50 instances for which our system made the largest errors 
Some examples that were indicative of the shortcomings of our system are shown in Table TABREF20   First of all, our system did not take into account capitalization  The implications of this are shown in the first sentence, where capitalization intensifies the emotion used in the sentence  In the second sentence, the name Imperator Furiosa is not understood  Since our texts were lowercased, our system was unable to capture the named entity and thought the sentence was about an angry emperor instead  In the third sentence, our system fails to capture that when you are so angry that it makes you laugh, it results in a reduced intensity of the angriness  Finally, in the fourth sentence, it is the figurative language me infla la vena (it inflates my vein) that the system is not able to understand 
The first two error-categories might be solved by including smart features regarding capitalization and named entity recognition  However, the last two categories are problems of natural language understanding and will be very difficult to fix 
Conclusion
To conclude, the present study described our submission for the Semeval 2018 Shared Task on Affect in Tweets  We participated in four Spanish subtasks and our submissions ranked second, second, fourth and fifth place  Our study aimed to investigate whether the automatic generation of additional training data through translation and semi-supervised learning, as well as the creation of stepwise ensembles, increase the performance of our Spanish-language models  Strong support was found for the translation and semi-supervised learning approaches; our best models for all subtasks use either one of these approaches  These results suggest that both of these additional data resources are beneficial when determining emotion intensity (for Spanish)  However, the creation of a stepwise ensemble from the best models did not result in better performance compared to simply averaging the models  In addition, some signs of overfitting on the dev set were found  In future work, we would like to apply the methods (translation and semi-supervised learning) used on Spanish on other low-resource languages and potentially also on other tasks 

ground_truth: using tweets that one has replied or quoted to as contextual information

text sequences of context tweets

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What were the scores of their system?
Context 4:  Now instead of orthogonalizing INLINEFORM10 to INLINEFORM11 we could orthogonalize INLINEFORM12 to the hidden state of this GRU at time-step INLINEFORM13   In practice, we found this to work better as it accounts for all the field vectors in the history instead of only the previous field vector 
In summary, Equation provides a mechanism for remembering the current field vector when appropriate (thus capturing stay-on behavior) using a remember gate  On the other hand, Equation EQREF10 explicitly ensures that the field vector is very different (soft-orthogonal) from the previous field vectors once it is time to move on (thus capturing never look back behavior)  The value of INLINEFORM0 computed in Equation EQREF10 is then used in Equation   The INLINEFORM1 (macro) thus obtained is then concatenated with INLINEFORM2 (micro) and fed to the decoder (see Fig  FIGREF3 )
Experimental setup
We now describe our experimental setup:
Datasets
We use the WikiBio dataset introduced by lebret2016neural  It consists of INLINEFORM0 biography articles from English Wikipedia  A biography article corresponds to a person (sportsman, politician, historical figure, actor, etc )  Each Wikipedia article has an accompanying infobox which serves as the structured input and the task is to generate the first sentence of the article (which typically is a one-line description of the person)  We used the same train, valid and test sets which were made publicly available by lebret2016neural 
We also introduce two new biography datasets, one in French and one in German  These datasets were created and pre-processed using the same procedure as outlined in lebret2016neural  Specifically, we extracted the infoboxes and the first sentence from the corresponding Wikipedia article  As with the English dataset, we split the French and German datasets randomly into train (80%), test (10%) and valid (10%)  The French and German datasets extracted by us has been made publicly available  The number of examples was 170K and 50K and the vocabulary size was 297K and 143K for French and German respectively  Although in this work we focus only on generating descriptions in one language, we hope that this dataset will also be useful for developing models which jointly learn to generate descriptions from structured data in multiple languages 
Models compared
We compare with the following models:
1  BIBREF0 : This is a conditional language model which uses a feed-forward neural network to predict the next word in the description conditioned on local characteristics (i e , words within a field) and global characteristics (i e , overall structure of the infobox) 
2  BIBREF9 : This model was proposed in the context of the WeatherGov and RoboCup datasets which have a much smaller vocabulary  They use an improved attention model with additional regularizer terms which influence the weights assigned to the fields 
3  Basic Seq2Seq: This is the vanilla encode-attend-decode model BIBREF1   Further, to deal with the large vocabulary ( INLINEFORM0 400K words) we use a copying mechanism as a post-processing step  Specifically, we identify the time steps at which the decoder produces unknown words (denoted by the special symbol UNK)  For each such time step, we look at the attention weights on the input words and replace the UNK word by that input word which has received maximum attention at this timestep  This process is similar to the one described in BIBREF30   Even lebret2016neural have a copying mechanism tightly integrated with their model 
Hyperparameter tuning
We tuned the hyperparameters of all the models using a validation set  As mentioned earlier, we used a bidirectional GRU cell as the function INLINEFORM0 for computing the representation of the fields and the values (see Section SECREF4 )  For all the models, we experimented with GRU state sizes of 128, 256 and 512  The total number of unique words in the corpus is around 400K (this includes the words in the infobox and the descriptions)  Of these, we retained only the top 20K words in our vocabulary (same as BIBREF0 )  We initialized the embeddings of these words with 300 dimensional Glove embeddings BIBREF31   We used Adam BIBREF32 with a learning rate of INLINEFORM1 , INLINEFORM2 and INLINEFORM3   We trained the model for a maximum of 20 epochs and used early stopping with the patience set to 5 epochs 
Results and Discussions
We now discuss the results of our experiments 
Comparison of different models
Following lebret2016neural, we used BLEU-4, NIST-4 and ROUGE-4 as the evaluation metrics  We first make a few observations based on the results on the English dataset (Table TABREF15 )  The basic seq2seq model, as well as the model proposed by weather16, perform better than the model proposed by lebret2016neural  Our final model with bifocal attention and gated orthogonalization gives the best performance and does 10% (relative) better than the closest baseline (basic seq2seq) and 21% (relative) better than the current state of the art method BIBREF0   In Table TABREF16 , we show some qualitative examples of the output generated by different models 
Human Evaluations
To make a qualitative assessment of the generated sentences, we conducted a human study on a sample of 500 Infoboxes which were sampled from English dataset  The annotators for this task were undergraduate and graduate students  For each of these infoboxes, we generated summaries using the basic seq2seq model and our final model with bifocal attention and gated orthogonalization  For each description and for each model, we asked three annotators to rank the output of the systems based on i) adequacy (i e  does it capture relevant information from the infobox), (ii) fluency (i e  grammar) and (iii) relative preference (i e , which of the two outputs would be preferred)  Overall the average fluency/adequacy (on a scale of 5) for basic seq2seq model was INLINEFORM0 and INLINEFORM1 for our model respectively 
The results from Table TABREF17 suggest that in general gated orthogonalization model performs better than the basic seq2seq model  Additionally, annotators were asked to verify if the generated summaries look natural (i e, as if they were generated by humans)  In 423 out of 500 cases, the annotators said “Yes” suggesting that gated orthogonalization model indeed produces good descriptions 
Performance on different languages
The results on the French and German datasets are summarized in Tables TABREF20 and TABREF20 respectively  Note that the code of BIBREF0 is not publicly available, hence we could not report numbers for French and German using their model  We observe that our final model gives the best performance - though the bifocal attention model performs poorly as compared to the basic seq2seq model on French  However, the overall performance for French and German are much smaller than those for English  There could be multiple reasons for this  First, the amount of training data in these two languages is smaller than that in English  Specifically, the amount of training data available in French (German) is only INLINEFORM0 ( INLINEFORM1 )% of that available for English  Second, on average the descriptions in French and German are longer than that in English (EN: INLINEFORM2 words, FR: INLINEFORM3 words and DE: INLINEFORM4 words)  Finally, a manual inspection across the three languages suggests that the English descriptions have a more consistent structure than the French descriptions  For example, most English descriptions start with name followed by date of birth but this is not the case in French  However, this is only a qualitative observation and it is hard to quantify this characteristic of the French and German datasets 
Visualizing Attention Weights
If the proposed model indeed works well then we should see attention weights that are consistent with the stay on and never look back behavior  To verify this, we plotted the attention weights in cases where the model with gated orthogonalization does better than the model with only bifocal attention  Figure FIGREF21 shows the attention weights corresponding to infobox in Figure FIGREF25   Notice that the model without gated orthogonalization has attention on both name field and article title while rendering the name  The model with gated orthogonalization, on the other hand, stays on the name field for as long as it is required but then moves and never returns to it (as expected) 
Due to lack of space, we do not show similar plots for French and German but we would like to mention that, in general, the differences between the attention weights learned by the model with and without gated orthogonalization were more pronounced for the French/German dataset than the English dataset  This is in agreement with the results reported in Table TABREF20 and TABREF20 where the improvements given by gated orthogonalization are more for French/German than for English 
Out of domain results
What if the model sees a different INLINEFORM0 of person at test time? For example, what if the training data does not contain any sportspersons but at test time we encounter the infobox of a sportsperson  This is the same as seeing out-of-domain data at test time  Such a situation is quite expected in the products domain where new products with new features (fields) get frequently added to the catalog  We were interested in three questions here  First, we wanted to see if testing the model on out-of-domain data indeed leads to a drop in the performance  For this, we compared the performance of our best model in two scenarios (i) trained on data from all domains (including the target domain) and tested on the target domain (sports, arts) and (ii) trained on data from all domains except the target domain and tested on the target domain  Comparing rows 1 and 2 of Table TABREF32 we observed a significant drop in the performance  Note that the numbers for sports domain in row 1 are much better than the Arts domain because roughly 40% of the WikiBio training data contains sportspersons 
Next, we wanted to see if we can use a small amount of data from the target domain to fine tune a model trained on the out of domain data  We observe that even with very small amounts of target domain data the performance starts improving significantly (see rows 3 and 4 of Table TABREF32 )  Note that if we train a model from scratch with only limited data from the target domain instead of fine-tuning a model trained on a different source domain then the performance is very poor  In particular, training a model from scratch with 10K training instances we get a BLEU score of INLINEFORM0 and INLINEFORM1 for arts and sports respectively  Finally, even though the actual words used for describing a sportsperson (footballer, cricketer, etc ) would be very different from the words used to describe an artist (actor, musician, etc ) they might share many fields (for example, date of birth, occupation, etc )  As seen in Figure FIGREF28 (attention weights corresponding to the infobox in Figure FIGREF27 ), the model predicts the attention weights correctly for common fields (such as occupation) but it is unable to use the right vocabulary to describe the occupation (since it has not seen such words frequently in the training data)  However, once we fine tune the model with limited data from the target domain we see that it picks up the new vocabulary and produces a correct description of the occupation 
Conclusion
We present a model for generating natural language descriptions from structured data  To address specific characteristics of the problem we propose neural components for fused bifocal attention and gated orthogonalization to address stay on and never look back behavior while decoding  Our final model outperforms an existing state of the art model on a large scale WikiBio dataset by 21%  We also introduce datasets for French and German and demonstrate that our model gives state of the art results on these datasets  Finally, we perform experiments with an out-of-domain model and show that if such a model is fine-tuned with small amounts of in domain data then it can give an improved performance on the target domain 
Given the multilingual nature of the new datasets, as future work, we would like to build models which can jointly learn to generate natural language descriptions from structured data in multiple languages  One idea is to replace the concepts in the input infobox by Wikidata concept ids which are language agnostic  A large amount of input vocabulary could thus be shared across languages thereby facilitating joint learning 
Acknowledgements
We thank Google for supporting Preksha Nema through their Google India Ph D  Fellowship program  We also thank Microsoft Research India for supporting Shreyas Shetty through their generous travel grant for attending the conference 

ground_truth: column Ens Test in Table TABREF19