prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What additional features and context are proposed?
Context 1: Introduction
Abusive language refers to any type of insult, vulgarity, or profanity that debases the target; it also can be anything that causes aggravation BIBREF0 , BIBREF1   Abusive language is often reframed as, but not limited to, offensive language BIBREF2 , cyberbullying BIBREF3 , othering language BIBREF4 , and hate speech BIBREF5  
Recently, an increasing number of users have been subjected to harassment, or have witnessed offensive behaviors online BIBREF6   Major social media companies (i e  Facebook, Twitter) have utilized multiple resources—artificial intelligence, human reviewers, user reporting processes, etc —in effort to censor offensive language, yet it seems nearly impossible to successfully resolve the issue BIBREF7 , BIBREF8  
The major reason of the failure in abusive language detection comes from its subjectivity and context-dependent characteristics BIBREF9   For instance, a message can be regarded as harmless on its own, but when taking previous threads into account it may be seen as abusive, and vice versa  This aspect makes detecting abusive language extremely laborious even for human annotators; therefore it is difficult to build a large and reliable dataset BIBREF10  
Previously, datasets openly available in abusive language detection research on Twitter ranged from 10K to 35K in size BIBREF9 , BIBREF11   This quantity is not sufficient to train the significant number of parameters in deep learning models  Due to this reason, these datasets have been mainly studied by traditional machine learning methods  Most recently, Founta et al  founta2018large introduced Hate and Abusive Speech on Twitter, a dataset containing 100K tweets with cross-validated labels  Although this corpus has great potential in training deep models with its significant size, there are no baseline reports to date 
This paper investigates the efficacy of different learning models in detecting abusive language  We compare accuracy using the most frequently studied machine learning classifiers as well as recent neural network models  Reliable baseline results are presented with the first comparative study on this dataset  Additionally, we demonstrate the effect of different features and variants, and describe the possibility for further improvements with the use of ensemble models 
Related Work
The research community introduced various approaches on abusive language detection  Razavi et al  razavi2010offensive applied Naïve Bayes, and Warner and Hirschberg warner2012detecting used Support Vector Machine (SVM), both with word-level features to classify offensive language  Xiang et al  xiang2012detecting generated topic distributions with Latent Dirichlet Allocation BIBREF12 , also using word-level features in order to classify offensive tweets 
More recently, distributed word representations and neural network models have been widely applied for abusive language detection  Djuric et al  djuric2015hate used the Continuous Bag Of Words model with paragraph2vec algorithm BIBREF13 to more accurately detect hate speech than that of the plain Bag Of Words models  Badjatiya et al  badjatiya2017deep implemented Gradient Boosted Decision Trees classifiers using word representations trained by deep learning models  Other researchers have investigated character-level representations and their effectiveness compared to word-level representations BIBREF14 , BIBREF15  
As traditional machine learning methods have relied on feature engineering, (i e  n-grams, POS tags, user information) BIBREF1 , researchers have proposed neural-based models with the advent of larger datasets  Convolutional Neural Networks and Recurrent Neural Networks have been applied to detect abusive language, and they have outperformed traditional machine learning classifiers such as Logistic Regression and SVM BIBREF15 , BIBREF16   However, there are no studies investigating the efficiency of neural models with large-scale datasets over 100K 
Methodology
This section illustrates our implementations on traditional machine learning classifiers and neural network based models in detail  Furthermore, we describe additional features and variant models investigated 
Traditional Machine Learning Models
We implement five feature engineering based machine learning classifiers that are most often used for abusive language detection  In data preprocessing, text sequences are converted into Bag Of Words (BOW) representations, and normalized with Term Frequency-Inverse Document Frequency (TF-IDF) values  We experiment with word-level features using n-grams ranging from 1 to 3, and character-level features from 3 to 8-grams  Each classifier is implemented with the following specifications:
Naïve Bayes (NB): Multinomial NB with additive smoothing constant 1
Logistic Regression (LR): Linear LR with L2 regularization constant 1 and limited-memory BFGS optimization
Support Vector Machine (SVM): Linear SVM with L2 regularization constant 1 and logistic loss function
Random Forests (RF): Averaging probabilistic predictions of 10 randomized decision trees
Gradient Boosted Trees (GBT): Tree boosting with learning rate 1 and logistic loss function
Neural Network based Models
Along with traditional machine learning approaches, we investigate neural network based models to evaluate their efficacy within a larger dataset  In particular, we explore Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), and their variant models  A pre-trained GloVe BIBREF17 representation is used for word-level features 
CNN: We adopt Kim's kim2014convolutional implementation as the baseline  The word-level CNN models have 3 convolutional filters of different sizes [1,2,3] with ReLU activation, and a max-pooling layer  For the character-level CNN, we use 6 convolutional filters of various sizes [3,4,5,6,7,8], then add max-pooling layers followed by 1 fully-connected layer with a dimension of 1024 
Park and Fung park2017one proposed a HybridCNN model which outperformed both word-level and character-level CNNs in abusive language detection  In order to evaluate the HybridCNN for this dataset, we concatenate the output of max-pooled layers from word-level and character-level CNN, and feed this vector to a fully-connected layer in order to predict the output 
All three CNN models (word-level, character-level, and hybrid) use cross entropy with softmax as their loss function and Adam BIBREF18 as the optimizer 
RNN: We use bidirectional RNN BIBREF19 as the baseline, implementing a GRU BIBREF20 cell for each recurrent unit  From extensive parameter-search experiments, we chose 1 encoding layer with 50 dimensional hidden states and an input dropout probability of 0 3  The RNN models use cross entropy with sigmoid as their loss function and Adam as the optimizer 
For a possible improvement, we apply a self-matching attention mechanism on RNN baseline models BIBREF21 so that they may better understand the data by retrieving text sequences twice  We also investigate a recently introduced method, Latent Topic Clustering (LTC) BIBREF22   The LTC method extracts latent topic information from the hidden states of RNN, and uses it for additional information in classifying the text data 
Feature Extension
While manually analyzing the raw dataset, we noticed that looking at the tweet one has replied to or has quoted, provides significant contextual information  We call these, “context tweets"  As humans can better understand a tweet with the reference of its context, our assumption is that computers also benefit from taking context tweets into account in detecting abusive language 
As shown in the examples below, (2) is labeled abusive due to the use of vulgar language  However, the intention of the user can be better understood with its context tweet (1) 
(1) I hate when I'm sitting in front of the bus and somebody with a wheelchair get on 
INLINEFORM0 (2) I hate it when I'm trying to board a bus and there's already an as**ole on it 
Similarly, context tweet (3) is important in understanding the abusive tweet (4), especially in identifying the target of the malice 
(3) Survivors of #Syria Gas Attack Recount `a Cruel Scene' 
INLINEFORM0 (4) Who the HELL is “LIKE" ING this post? Sick people    
Huang et al  huang2016modeling used several attributes of context tweets for sentiment analysis in order to improve the baseline LSTM model  However, their approach was limited because the meta-information they focused on—author information, conversation type, use of the same hashtags or emojis—are all highly dependent on data 
In order to avoid data dependency, text sequences of context tweets are directly used as an additional feature of neural network models  We use the same baseline model to convert context tweets to vectors, then concatenate these vectors with outputs of their corresponding labeled tweets  More specifically, we concatenate max-pooled layers of context and labeled tweets for the CNN baseline model  As for RNN, the last hidden states of context and labeled tweets are concatenated 
Dataset
Hate and Abusive Speech on Twitter BIBREF10 classifies tweets into 4 labels, “normal", “spam", “hateful" and “abusive"  We were only able to crawl 70,904 tweets out of 99,996 tweet IDs, mainly because the tweet was deleted or the user account had been suspended  Table shows the distribution of labels of the crawled data 
Data Preprocessing
In the data preprocessing steps, user IDs, URLs, and frequently used emojis are replaced as special tokens  Since hashtags tend to have a high correlation with the content of the tweet BIBREF23 , we use a segmentation library BIBREF24 for hashtags to extract more information 
For character-level representations, we apply the method Zhang et al  zhang2015character proposed  Tweets are transformed into one-hot encoded vectors using 70 character dimensions—26 lower-cased alphabets, 10 digits, and 34 special characters including whitespace 
Training and Evaluation
In training the feature engineering based machine learning classifiers, we truncate vector representations according to the TF-IDF values (the top 14,000 and 53,000 for word-level and character-level representations, respectively) to avoid overfitting  For neural network models, words that appear only once are replaced as unknown tokens 
Since the dataset used is not split into train, development, and test sets, we perform 10-fold cross validation, obtaining the average of 5 tries; we divide the dataset randomly by a ratio of 85:5:10, respectively  In order to evaluate the overall performance, we calculate the weighted average of precision, recall, and F1 scores of all four labels, “normal”, “spam”, “hateful”, and “abusive” 
Empirical Results
As shown in Table , neural network models are more accurate than feature engineering based models (i e  NB, SVM, etc ) except for the LR model—the best LR model has the same F1 score as the best CNN model 
Among traditional machine learning models, the most accurate in classifying abusive language is the LR model followed by ensemble models such as GBT and RF  Character-level representations improve F1 scores of SVM and RF classifiers, but they have no positive effect on other models 
For neural network models, RNN with LTC modules have the highest accuracy score, but there are no significant improvements from its baseline model and its attention-added model  Similarly, HybridCNN does not improve the baseline CNN model  For both CNN and RNN models, character-level features significantly decrease the accuracy of classification 
The use of context tweets generally have little effect on baseline models, however they noticeably improve the scores of several metrics  For instance, CNN with context tweets score the highest recall and F1 for “hateful" labels, and RNN models with context tweets have the highest recall for “abusive" tweets 
Discussion and Conclusion
While character-level features are known to improve the accuracy of neural network models BIBREF16 , they reduce classification accuracy for Hate and Abusive Speech on Twitter  We conclude this is because of the lack of labeled data as well as the significant imbalance among the different labels  Unlike neural network models, character-level features in traditional machine learning classifiers have positive results because we have trained the models only with the most significant character elements using TF-IDF values 
Variants of neural network models also suffer from data insufficiency  However, these models show positive performances on “spam" (14%) and “hateful" (4%) tweets—the lower distributed labels  The highest F1 score for “spam" is from the RNN-LTC model (0 551), and the highest for “hateful" is CNN with context tweets (0 309)  Since each variant model excels in different metrics, we expect to see additional improvements with the use of ensemble models of these variants in future works 
In this paper, we report the baseline accuracy of different learning models as well as their variants on the recently introduced dataset, Hate and Abusive Speech on Twitter  Experimental results show that bidirectional GRU networks with LTC provide the most accurate results in detecting abusive language  Additionally, we present the possibility of using ensemble models of variant models and features for further improvements 
Acknowledgments
K  Jung is with the Department of Electrical and Computer Engineering, ASRI, Seoul National University, Seoul, Korea  This work was supported by the National Research Foundation of Korea (NRF) funded by the Korea government (MSIT) (No  2016M3C4A7952632), the Technology Innovation Program (10073144) funded by the Ministry of Trade, Industry & Energy (MOTIE, Korea) 
We would also like to thank Yongkeun Hwang and Ji Ho Park for helpful discussions and their valuable insights 
Context 3: Introduction
Event detection on microblogging platforms such as Twitter aims to detect events preemptively  A main task in event detection is detecting events of predetermined types BIBREF0, such as concerts or controversial events based on microposts matching specific event descriptions  This task has extensive applications ranging from cyber security BIBREF1, BIBREF2 to political elections BIBREF3 or public health BIBREF4, BIBREF5  Due to the high ambiguity and inconsistency of the terms used in microposts, event detection is generally performed though statistical machine learning models, which require a labeled dataset for model training  Data labeling is, however, a long, laborious, and usually costly process  For the case of micropost classification, though positive labels can be collected (e g , using specific hashtags, or event-related date-time information), there is no straightforward way to generate negative labels useful for model training  To tackle this lack of negative labels and the significant manual efforts in data labeling, BIBREF1 (BIBREF1, BIBREF3) introduced a weak supervision based learning approach, which uses only positively labeled data, accompanied by unlabeled examples by filtering microposts that contain a certain keyword indicative of the event type under consideration (e g , `hack' for cyber security)  Another key technique in this context is expectation regularization BIBREF6, BIBREF7, BIBREF1  Here, the estimated proportion of relevant microposts in an unlabeled dataset containing a keyword is given as a keyword-specific expectation  This expectation is used in the regularization term of the model's objective function to constrain the posterior distribution of the model predictions  By doing so, the model is trained with an expectation on its prediction for microposts that contain the keyword  Such a method, however, suffers from two key problems:
Due to the unpredictability of event occurrences and the constantly changing dynamics of users' posting frequency BIBREF8, estimating the expectation associated with a keyword is a challenging task, even for domain experts;
The performance of the event detection model is constrained by the informativeness of the keyword used for model training  As of now, we lack a principled method for discovering new keywords and improve the model performance 
To address the above issues, we advocate a human-AI loop approach for discovering informative keywords and estimating their expectations reliably  Our approach iteratively leverages 1) crowd workers for estimating keyword-specific expectations, and 2) the disagreement between the model and the crowd for discovering new informative keywords  More specifically, at each iteration after we obtain a keyword-specific expectation from the crowd, we train the model using expectation regularization and select those keyword-related microposts for which the model's prediction disagrees the most with the crowd's expectation; such microposts are then presented to the crowd to identify new keywords that best explain the disagreement  By doing so, our approach identifies new keywords which convey more relevant information with respect to existing ones, thus effectively boosting model performance  By exploiting the disagreement between the model and the crowd, our approach can make efficient use of the crowd, which is of critical importance in a human-in-the-loop context BIBREF9, BIBREF10  An additional advantage of our approach is that by obtaining new keywords that improve model performance over time, we are able to gain insight into how the model learns for specific event detection tasks  Such an advantage is particularly useful for event detection using complex models, e g , deep neural networks, which are intrinsically hard to understand BIBREF11, BIBREF12  An additional challenge in involving crowd workers is that their contributions are not fully reliable BIBREF13  In the crowdsourcing literature, this problem is usually tackled with probabilistic latent variable models BIBREF14, BIBREF15, BIBREF16, which are used to perform truth inference by aggregating a redundant set of crowd contributions  Our human-AI loop approach improves the inference of keyword expectation by aggregating contributions not only from the crowd but also from the model  This, however, comes with its own challenge as the model's predictions are further dependent on the results of expectation inference, which is used for model training  To address this problem, we introduce a unified probabilistic model that seamlessly integrates expectation inference and model training, thereby allowing the former to benefit from the latter while resolving the inter-dependency between the two 
To the best of our knowledge, we are the first to propose a human-AI loop approach that iteratively improves machine learning models for event detection  In summary, our work makes the following key contributions:
A novel human-AI loop approach for micropost event detection that jointly discovers informative keywords and estimates their expectation;
A unified probabilistic model that infers keyword expectation and simultaneously performs model training;
An extensive empirical evaluation of our approach on multiple real-world datasets demonstrating that our approach significantly improves the state of the art by an average of 24 3% AUC 
The rest of this paper is organized as follows  First, we present our human-AI loop approach in Section SECREF2  Subsequently, we introduce our proposed probabilistic model in Section SECREF3  The experimental setup and results are presented in Section SECREF4  Finally, we briefly cover related work in Section SECREF5 before concluding our work in Section SECREF6 
The Human-AI Loop Approach
Given a set of labeled and unlabeled microposts, our goal is to extract informative keywords and estimate their expectations in order to train a machine learning model  To achieve this goal, our proposed human-AI loop approach comprises two crowdsourcing tasks, i e , micropost classification followed by keyword discovery, and a unified probabilistic model for expectation inference and model training  Figure FIGREF6 presents an overview of our approach  Next, we describe our approach from a process-centric perspective 
Following previous studies BIBREF1, BIBREF17, BIBREF2, we collect a set of unlabeled microposts $\mathcal {U}$ from a microblogging platform and post-filter, using an initial (set of) keyword(s), those microposts that are potentially relevant to an event category  Then, we collect a set of event-related microposts (i e , positively labeled microposts) $\mathcal {L}$, post-filtering with a list of seed events  $\mathcal {U}$ and $\mathcal {L}$ are used together to train a discriminative model (e g , a deep neural network) for classifying the relevance of microposts to an event  We denote the target model as $p_\theta (y|x)$, where $\theta $ is the model parameter to be learned and $y$ is the label of an arbitrary micropost, represented by a bag-of-words vector $x$  Our approach iterates several times $t=\lbrace 1, 2, \ldots \rbrace $ until the performance of the target model converges  Each iteration starts from the initial keyword(s) or the new keyword(s) discovered in the previous iteration  Given such a keyword, denoted by $w^{(t)}$, the iteration starts by sampling microposts containing the keyword from $\mathcal {U}$, followed by dynamically creating micropost classification tasks and publishing them on a crowdsourcing platform 
Micropost Classification  The micropost classification task requires crowd workers to label the selected microposts into two classes: event-related and non event-related  In particular, workers are given instructions and examples to differentiate event-instance related microposts and general event-category related microposts  Consider, for example, the following microposts in the context of Cyber attack events, both containing the keyword `hack':
Credit firm Equifax says 143m Americans' social security numbers exposed in hack
This micropost describes an instance of a cyber attack event that the target model should identify  This is, therefore, an event-instance related micropost and should be considered as a positive example  Contrast this with the following example:
Companies need to step their cyber security up
This micropost, though related to cyber security in general, does not mention an instance of a cyber attack event, and is of no interest to us for event detection  This is an example of a general event-category related micropost and should be considered as a negative example 
In this task, each selected micropost is labeled by multiple crowd workers  The annotations are passed to our probabilistic model for expectation inference and model training 
Expectation Inference & Model Training  Our probabilistic model takes crowd-contributed labels and the model trained in the previous iteration as input  As output, it generates a keyword-specific expectation, denoted as $e^{(t)}$, and an improved version of the micropost classification model, denoted as $p_{\theta ^{(t)}}(y|x)$  The details of our probabilistic model are given in Section SECREF3 
Keyword Discovery  The keyword discovery task aims at discovering a new keyword (or a set of keywords) that is most informative for model training with respect to existing keywords  To this end, we first apply the current model $p_{\theta ^{(t)}}(y|x)$ on the unlabeled microposts $\mathcal {U}$  For those that contain the keyword $w^{(t)}$, we calculate the disagreement between the model predictions and the keyword-specific expectation $e^{(t)}$:
and select the ones with the highest disagreement for keyword discovery  These selected microposts are supposed to contain information that can explain the disagreement between the model prediction and keyword-specific expectation, and can thus provide information that is most different from the existing set of keywords for model training 
For instance, our study shows that the expectation for the keyword `hack' is 0 20, which means only 20% of the initial set of microposts retrieved with the keyword are event-related  A micropost selected with the highest disagreement (Eq  DISPLAY_FORM7), whose likelihood of being event-related as predicted by the model is $99 9\%$, is shown as an example below:
RT @xxx: Hong Kong securities brokers hit by cyber attacks, may face more: regulator #cyber #security #hacking https://t co/rC1s9CB
This micropost contains keywords that can better indicate the relevance to a cyber security event than the initial keyword `hack', e g , `securities', `hit', and `attack' 
Note that when the keyword-specific expectation $e^{(t)}$ in Equation DISPLAY_FORM7 is high, the selected microposts will be the ones that contain keywords indicating the irrelevance of the microposts to an event category  Such keywords are also useful for model training as they help improve the model's ability to identify irrelevant microposts 
To identify new keywords in the selected microposts, we again leverage crowdsourcing, as humans are typically better than machines at providing specific explanations BIBREF18, BIBREF19  In the crowdsourcing task, workers are first asked to find those microposts where the model predictions are deemed correct  Then, from those microposts, workers are asked to find the keyword that best indicates the class of the microposts as predicted by the model  The keyword most frequently identified by the workers is then used as the initial keyword for the following iteration  In case multiple keywords are selected, e g , the top-$N$ frequent ones, workers will be asked to perform $N$ micropost classification tasks for each keyword in the next iteration, and the model training will be performed on multiple keyword-specific expectations 
Unified Probabilistic Model
This section introduces our probabilistic model that infers keyword expectation and trains the target model simultaneously  We start by formalizing the problem and introducing our model, before describing the model learning method 
Problem Formalization  We consider the problem at iteration $t$ where the corresponding keyword is $w^{(t)}$  In the current iteration, let $\mathcal {U}^{(t)} \subset \mathcal {U}$ denote the set of all microposts containing the keyword and $\mathcal {M}^{(t)}= \lbrace x_{m}\rbrace _{m=1}^M\subset \mathcal {U}^{(t)}$ be the randomly selected subset of $M$ microposts labeled by $N$ crowd workers $\mathcal {C} = \lbrace c_n\rbrace _{n=1}^N$  The annotations form a matrix $\mathbf {A}\in \mathbb {R}^{M\times N}$ where $\mathbf {A}_{mn}$ is the label for the micropost $x_m$ contributed by crowd worker $c_n$  Our goal is to infer the keyword-specific expectation $e^{(t)}$ and train the target model by learning the model parameter $\theta ^{(t)}$  An additional parameter of our probabilistic model is the reliability of crowd workers, which is essential when involving crowdsourcing  Following Dawid and Skene BIBREF14, BIBREF16, we represent the annotation reliability of worker $c_n$ by a latent confusion matrix $\pi ^{(n)}$, where the $rs$-th element $\pi _{rs}^{(n)}$ denotes the probability of $c_n$ labeling a micropost as class $r$ given the true class $s$ 
Unified Probabilistic Model ::: Expectation as Model Posterior
First, we introduce an expectation regularization technique for the weakly supervised learning of the target model $p_{\theta ^{(t)}}(y|x)$  In this setting, the objective function of the target model is composed of two parts, corresponding to the labeled microposts $\mathcal {L}$ and the unlabeled ones $\mathcal {U}$ 
The former part aims at maximizing the likelihood of the labeled microposts:
where we assume that $\theta $ is generated from a prior distribution (e g , Laplacian or Gaussian) parameterized by $\sigma $ 
To leverage unlabeled data for model training, we make use of the expectations of existing keywords, i e , {($w^{(1)}$, $e^{(1)}$),    , ($w^{(t-1)}$, $e^{(t-1)}$), ($w^{(t)}$, $e^{(t)}$)} (Note that $e^{(t)}$ is inferred), as a regularization term to constrain model training  To do so, we first give the model's expectation for each keyword $w^{(k)}$ ($1\le k\le t$) as follows:
which denotes the empirical expectation of the model’s posterior predictions on the unlabeled microposts $\mathcal {U}^{(k)}$ containing keyword $w^{(k)}$  Expectation regularization can then be formulated as the regularization of the distance between the Bernoulli distribution parameterized by the model's expectation and the expectation of the existing keyword:
where $D_{KL}[\cdot \Vert \cdot ]$ denotes the KL-divergence between the Bernoulli distributions $Ber(e^{(k)})$ and $Ber(\mathbb {E}_{x\sim \mathcal {U}^{(k)}}(y))$, and $\lambda $ controls the strength of expectation regularization 
Unified Probabilistic Model ::: Expectation as Class Prior
To learn the keyword-specific expectation $e^{(t)}$ and the crowd worker reliability $\pi ^{(n)}$ ($1\le n\le N$), we model the likelihood of the crowd-contributed labels $\mathbf {A}$ as a function of these parameters  In this context, we view the expectation as the class prior, thus performing expectation inference as the learning of the class prior  By doing so, we connect expectation inference with model training 
Specifically, we model the likelihood of an arbitrary crowd-contributed label $\mathbf {A}_{mn}$ as a mixture of multinomials where the prior is the keyword-specific expectation $e^{(t)}$:
where $e_s^{(t)}$ is the probability of the ground truth label being $s$ given the keyword-specific expectation as the class prior; $K$ is the set of possible ground truth labels (binary in our context); and $r=\mathbf {A}_{mn}$ is the crowd-contributed label  Then, for an individual micropost $x_m$, the likelihood of crowd-contributed labels $\mathbf {A}_{m:}$ is given by:
Therefore, the objective function for maximizing the likelihood of the entire annotation matrix $\mathbf {A}$ can be described as:
Unified Probabilistic Model ::: Unified Probabilistic Model
Integrating model training with expectation inference, the overall objective function of our proposed model is given by:
Figure FIGREF18 depicts a graphical representation of our model, which combines the target model for training (on the left) with the generative model for crowd-contributed labels (on the right) through a keyword-specific expectation 
Model Learning  Due to the unknown ground truth labels of crowd-annotated microposts ($y_m$ in Figure FIGREF18), we resort to expectation maximization for model learning  The learning algorithm iteratively takes two steps: the E-step and the M-step

ground_truth: using tweets that one has replied or quoted to as contextual information

text sequences of context tweets

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. what was the baseline?
Context 2: Introduction
Neural machine translation (NMT) BIBREF0 , BIBREF1 , BIBREF2 has enabled end-to-end training of a translation system without needing to deal with word alignments, translation rules, and complicated decoding algorithms, which are the characteristics of phrase-based statistical machine translation (PBSMT) BIBREF3   Although NMT can be significantly better than PBSMT in resource-rich scenarios, PBSMT performs better in low-resource scenarios BIBREF4   Only by exploiting cross-lingual transfer learning techniques BIBREF5 , BIBREF6 , BIBREF7 , can the NMT performance approach PBSMT performance in low-resource scenarios 
However, such methods usually require an NMT model trained on a resource-rich language pair like French INLINEFORM0 English (parent), which is to be fine-tuned for a low-resource language pair like Uzbek INLINEFORM1 English (child)  On the other hand, multilingual approaches BIBREF8 propose to train a single model to translate multiple language pairs  However, these approaches are effective only when the parent target or source language is relatively resource-rich like English (En)  Furthermore, the parents and children models should be trained on similar domains; otherwise, one has to take into account an additional problem of domain adaptation BIBREF9  
In this paper, we work on a linguistically distant and thus challenging language pair Japanese INLINEFORM0 Russian (Ja INLINEFORM1 Ru) which has only 12k lines of news domain parallel corpus and hence is extremely resource-poor  Furthermore, the amount of indirect in-domain parallel corpora, i e , Ja INLINEFORM2 En and Ru INLINEFORM3 En, are also small  As we demonstrate in Section SECREF4 , this severely limits the performance of prominent low-resource techniques, such as multilingual modeling, back-translation, and pivot-based PBSMT  To remedy this, we propose a novel multistage fine-tuning method for NMT that combines multilingual modeling BIBREF8 and domain adaptation BIBREF9  
We have addressed two important research questions (RQs) in the context of extremely low-resource machine translation (MT) and our explorations have derived rational contributions (CTs) as follows:
To the best of our knowledge, we are the first to perform such an extensive evaluation of extremely low-resource MT problem and propose a novel multilingual multistage fine-tuning approach involving multilingual modeling and domain adaptation to address it 
Our Japanese–Russian Setting
In this paper, we deal with Ja INLINEFORM0 Ru news translation  This language pair is very challenging because the languages involved have completely different writing system, phonology, morphology, grammar, and syntax  Among various domains, we experimented with translations in the news domain, considering the importance of sharing news between different language speakers  Moreover, news domain is one of the most challenging tasks, due to large presence of out-of-vocabulary (OOV) tokens and long sentences  To establish and evaluate existing methods, we also involved English as the third language  As direct parallel corpora are scarce, involving a language such as English for pivoting is quite common BIBREF10  
There has been no clean held-out parallel data for Ja INLINEFORM0 Ru and Ja INLINEFORM1 En news translation  Therefore, we manually compiled development and test sets using News Commentary data as a source  Since the given Ja INLINEFORM2 Ru and Ja INLINEFORM3 En data share many lines in the Japanese side, we first compiled tri-text data  Then, from each line, corresponding parts across languages were manually identified, and unaligned parts were split off into a new line  Note that we have never merged two or more lines  As a result, we obtained 1,654 lines of data comprising trilingual, bilingual, and monolingual segments (mainly sentences) as summarized in Table TABREF8   Finally, for the sake of comparability, we randomly chose 600 trilingual sentences to create a test set, and concatenated the rest of them and bilingual sentences to form development sets 
Our manually aligned development and test sets are publicly available 
Related Work
koehn-knowles:2017:NMT showed that NMT is unable to handle low-resource language pairs as opposed to PBSMT  Transfer learning approaches BIBREF5 , BIBREF6 , BIBREF7 work well when a large helping parallel corpus is available  This restricts one of the source or the target languages to be English which, in our case, is not possible  Approaches involving bi-directional NMT modeling is shown to drastically improve low-resource translation BIBREF11   However, like most other, this work focuses on translation from and into English 
Remaining options include (a) unsupervised MT BIBREF12 , BIBREF13 , BIBREF14 , (b) parallel sentence mining from non-parallel or comparable corpora BIBREF15 , BIBREF16 , (c) generating pseudo-parallel data BIBREF17 , and (d) MT based on pivot languages BIBREF10   The linguistic distance between Japanese and Russian makes it extremely difficult to learn bilingual knowledge, such as bilingual lexicons and bilingual word embeddings  Unsupervised MT is thus not promising yet, due to its heavy reliance on accurate bilingual word embeddings  Neither does parallel sentence mining, due to the difficulty of obtaining accurate bilingual lexicons  Pseudo-parallel data can be used to augment existing parallel corpora for training, and previous work has reported that such data generated by so-called back-translation can substantially improve the quality of NMT  However, this approach requires base MT systems that can generate somewhat accurate translations  It is thus infeasible in our scenario, because we can obtain only a weak system which is the consequence of an extremely low-resource situation  MT based on pivot languages requires large in-domain parallel corpora involving the pivot languages  This technique is thus infeasible, because the in-domain parallel corpora for Ja INLINEFORM0 En and Ru INLINEFORM1 En pairs are also extremely limited, whereas there are large parallel corpora in other domains  Section SECREF4 empirically confirms the limit of these existing approaches 
Fortunately, there are two useful transfer learning solutions using NMT: (e) multilingual modeling to incorporate multiple language pairs into a single model BIBREF8 and (f) domain adaptation to incorporate out-of-domain data BIBREF9   In this paper, we explore a novel method involving step-wise fine-tuning to combine these two methods  By improving the translation quality in this way, we can also increase the likelihood of pseudo-parallel data being useful to further improve translation quality 
Limit of Using only In-domain Data
This section answers our first research question, [RQ1], about the translation quality that we can achieve using existing methods and in-domain parallel and monolingual data  We then use the strongest model to conduct experiments on generating and utilizing back-translated pseudo-parallel data for augmenting NMT  Our intention is to empirically identify the most effective practices as well as recognize the limitations of relying only on in-domain parallel corpora 
Data
To train MT systems among the three languages, i e , Japanese, Russian, and English, we used all the parallel data provided by Global Voices, more specifically those available at OPUS  Table TABREF9 summarizes the size of train/development/test splits used in our experiments  The number of parallel sentences for Ja INLINEFORM0 Ru is 12k, for Ja INLINEFORM1 En is 47k, and for Ru INLINEFORM2 En is 82k  Note that the three corpora are not mutually exclusive: 9k out of 12k sentences in the Ja INLINEFORM3 Ru corpus were also included in the other two parallel corpora, associated with identical English translations  This puts a limit on the positive impact that the helping corpora can have on the translation quality 
Even when one focuses on low-resource language pairs, we often have access to larger quantities of in-domain monolingual data of each language  Such monolingual data are useful to improve quality of MT, for example, as the source of pseudo-parallel data for augmenting training data for NMT BIBREF17 and as the training data for large and smoothed language models for PBSMT BIBREF4   Table TABREF13 summarizes the statistics on our monolingual corpora for several domains including the news domain  Note that we removed from the Global Voices monolingual corpora those sentences that are already present in the parallel corpus 
https://dumps wikimedia org/backup-index html (20180501) http://www statmt org/wmt18/translation-task html https://www yomiuri co jp/database/glossary/ http://www cs jhu edu/~kevinduh/a/multitarget-tedtalks/ http://opus nlpl eu/Tatoeba-v2 php
We tokenized English and Russian sentences using tokenizer perl of Moses BIBREF3   To tokenize Japanese sentences, we used MeCab with the IPA dictionary  After tokenization, we eliminated duplicated sentence pairs and sentences with more than 100 tokens for all the languages 
MT Methods Examined
We began with evaluating standard MT paradigms, i e , PBSMT BIBREF3 and NMT BIBREF1   As for PBSMT, we also examined two advanced methods: pivot-based translation relying on a helping language BIBREF10 and induction of phrase tables from monolingual data BIBREF14  
As for NMT, we compared two types of encoder-decoder architectures: attentional RNN-based model (RNMT) BIBREF2 and the Transformer model BIBREF18   In addition to standard uni-directional modeling, to cope with the low-resource problem, we examined two multi-directional models: bi-directional model BIBREF11 and multi-to-multi (M2M) model BIBREF8  
After identifying the best model, we also examined the usefulness of a data augmentation method based on back-translation BIBREF17  
First, we built a PBSMT system for each of the six translation directions  We obtained phrase tables from parallel corpus using SyMGIZA++ with the grow-diag-final heuristics for word alignment, and Moses for phrase pair extraction  Then, we trained a bi-directional MSD (monotone, swap, and discontinuous) lexicalized reordering model  We also trained three 5-gram language models, using KenLM on the following monolingual data: (1) the target side of the parallel data, (2) the concatenation of (1) and the monolingual data from Global Voices, and (3) the concatenation of (1) and all monolingual data in the news domain in Table TABREF13  
Subsequently, using English as the pivot language, we examined the following three types of pivot-based PBSMT systems BIBREF10 , BIBREF19 for each of Ja INLINEFORM0 Ru and Ru INLINEFORM1 Ja 
2-step decoding using the source-to-English and English-to-target systems 
Obtain a new phrase table from synthetic parallel data generated by translating English side of the target–English training parallel data to the source language with the English-to-source system 
Compile a new phrase table combining those for the source-to-English and English-to-target systems 
Among these three, triangulation is the most computationally expensive method  Although we had filtered the component phrase tables using the statistical significance pruning method BIBREF20 , triangulation can generate an enormous number of phrase pairs  To reduce the computational cost during decoding and the negative effects of potentially noisy phrase pairs, we retained for each source phrase INLINEFORM0 only the INLINEFORM1 -best translations INLINEFORM2 according to the forward translation probability INLINEFORM3 calculated from the conditional probabilities in the component models as defined in utiyama:07  For each of the retained phrase pairs, we also calculated the backward translation probability, INLINEFORM4 , and lexical translation probabilities, INLINEFORM5 and INLINEFORM6 , in the same manner as INLINEFORM7  
We also investigated the utility of recent advances in unsupervised MT  Even though we began with a publicly available implementation of unsupervised PBSMT BIBREF13 , it crashed due to unknown reasons  We therefore followed another method described in marie:usmt-unmt  Instead of short INLINEFORM0 -grams BIBREF12 , BIBREF13 , we collected a set of phrases in Japanese and Russian from respective monolingual data using the word2phrase algorithm BIBREF21 , as in marie:usmt-unmt  To reduce the complexity, we used randomly selected 10M monolingual sentences, and 300k most frequent phrases made of words among the 300k most frequent words  For each source phrase INLINEFORM1 , we selected 300-best target phrases INLINEFORM2 according to the translation probability as in D18-1549: INLINEFORM3 where INLINEFORM4 stands for a bilingual embedding of a given phrase, obtained through averaging bilingual embeddings of constituent words learned from the two monolingual data using fastText and vecmap  For each of the retained phrase pair, INLINEFORM5 was computed analogously  We also computed lexical translation probabilities relying on those learned from the given small parallel corpus 
Up to four phrase tables were jointly exploited by the multiple decoding path ability of Moses  Weights for the features were tuned using KB-MIRA BIBREF22 on the development set; we took the best weights after 15 iterations  Two hyper-parameters, namely, INLINEFORM0 for the number of pivot-based phrase pairs per source phrase and INLINEFORM1 for distortion limit, were determined by a grid search on INLINEFORM2 and INLINEFORM3   In contrast, we used predetermined hyper-parameters for phrase table induction from monolingual data, following the convention: 200 for the dimension of word and phrase embeddings and INLINEFORM4  
We used the open-source implementation of the RNMT and the Transformer models in tensor2tensor  A uni-directional model for each of the six translation directions was trained on the corresponding parallel corpus  Bi-directional and M2M models were realized by adding an artificial token that specifies the target language to the beginning of each source sentence and shuffling the entire training data BIBREF8  
Table TABREF22 contains some specific hyper-parameters for our baseline NMT models  The hyper-parameters not mentioned in this table used the default values in tensor2tensor  For M2M systems, we over-sampled Ja INLINEFORM0 Ru and Ja INLINEFORM1 En training data so that their sizes match the largest Ru INLINEFORM2 En data  To reduce the number of unknown words, we used tensor2tensor's internal sub-word segmentation mechanism  Since we work in a low-resource setting, we used shared sub-word vocabularies of size 16k for the uni- and bi-directional models and 32k for the M2M models  The number of training iterations was determined by early-stopping: we evaluated our models on the development set every 1,000 updates, and stopped training if BLEU score for the development set was not improved for 10,000 updates (10 check-points)  Note that the development set was created by concatenating those for the individual translation directions without any over-sampling 
Having trained the models, we averaged the last 10 check-points and decoded the test sets with a beam size of 4 and a length penalty which was tuned by a linear search on the BLEU score for the development set 
Similarly to PBSMT, we also evaluated “Cascade” and “Synthesize” methods with uni-directional NMT models 
Results
We evaluated MT models using case-sensitive and tokenized BLEU BIBREF23 on test sets, using Moses's multi-bleu perl  Statistical significance ( INLINEFORM0 ) on the difference of BLEU scores was tested by Moses's bootstrap-hypothesis-difference-significance pl 
Tables TABREF27 and TABREF31 show BLEU scores of all the models, except the NMT systems augmented with back-translations  Whereas some models achieved reasonable BLEU scores for Ja INLINEFORM0 En and Ru INLINEFORM1 En translation, all the results for Ja INLINEFORM2 Ru, which is our main concern, were abysmal 
Among the NMT models, Transformer models (b INLINEFORM0 ) were proven to be better than RNMT models (a INLINEFORM1 )  RNMT models could not even outperform the uni-directional PBSMT models (c1)  M2M models (a3) and (b3) outperformed their corresponding uni- and bi-directional models in most cases  It is worth noting that in this extremely low-resource scenario, BLEU scores of the M2M RNMT model for the largest language pair, i e , Ru INLINEFORM2 En, were lower than those of the uni- and bi-directional RNMT models as in TACL1081  In contrast, with the M2M Transformer model, Ru INLINEFORM3 En also benefited from multilingualism 
Standard PBSMT models (c1) achieved higher BLEU scores than uni-directional NMT models (a1) and (b1), as reported by koehn-knowles:2017:NMT, whereas they underperform the M2M Transformer NMT model (b3)  As shown in Table TABREF31 , pivot-based PBSMT systems always achieved higher BLEU scores than (c1)  The best model with three phrase tables, labeled “Synthesize / Triangulate / Gold,” brought visible BLEU gains with substantial reduction of OOV tokens (3047 INLINEFORM0 1180 for Ja INLINEFORM1 Ru, 4463 INLINEFORM2 1812 for Ru INLINEFORM3 Ja)  However, further extension with phrase tables induced from monolingual data did not push the limit, despite their high coverage; only 336 and 677 OOV tokens were left for the two translation directions, respectively  This is due to the poor quality of the bilingual word embeddings used to extract the phrase table, as envisaged in Section SECREF3  
None of pivot-based approaches with uni-directional NMT models could even remotely rival the M2M Transformer NMT model (b3) 
Table TABREF46 shows the results of our multistage fine-tuning, where the IDs of each row refer to those described in Section SECREF41   First of all, the final models of our multistage fine-tuning, i e , V and VII, achieved significantly higher BLEU scores than (b3) in Table TABREF27 , a weak baseline without using any monolingual data, and #10 in Table TABREF33 , a strong baseline established with monolingual data 
The performance of the initial model (I) depends on the language pair  For Ja INLINEFORM0 Ru pair, it cannot achieve minimum level of quality since the model has never seen parallel data for this pair  The performance on Ja INLINEFORM1 En pair was much lower than the two baseline models, reflecting the crucial mismatch between training and testing domains  In contrast, Ru INLINEFORM2 En pair benefited the most and achieved surprisingly high BLEU scores  The reason might be due to the proximity of out-of-domain training data and in-domain test data 
The first fine-tuning stage significantly pushed up the translation quality for Ja INLINEFORM0 En and Ru INLINEFORM1 En pairs, in both cases with fine-tuning (II) and mixed fine-tuning (III)  At this stage, both models performed only poorly for Ja INLINEFORM2 Ru pair as they have not yet seen Ja INLINEFORM3 Ru parallel data  Either model had a consistent advantage to the other 
When these models were further fine-tuned only on the in-domain Ja INLINEFORM0 Ru parallel data (IV and VI), we obtained translations of better quality than the two baselines for Ja INLINEFORM1 Ru pair  However, as a result of complete ignorance of Ja INLINEFORM2 En and Ru INLINEFORM3 En pairs, the models only produced translations of poor quality for these language pairs  In contrast, mixed fine-tuning for the second fine-tuning stage (V and VII) resulted in consistently better models than conventional fine-tuning (IV and VI), irrespective of the choice at the first stage, thanks to the gradual shift of parameters realized by in-domain Ja INLINEFORM4 En and Ru INLINEFORM5 En parallel data  Unfortunately, the translation quality for Ja INLINEFORM6 En and Ru INLINEFORM7 En pairs sometimes degraded from II and III  Nevertheless, the BLEU scores still retain the large margin against two baselines 
The last three rows in Table TABREF46 present BLEU scores obtained by the methods with fewer fine-tuning steps  The most naive model I', trained on the balanced mixture of whole five types of corpora from scratch, and the model II', obtained through a single-step conventional fine-tuning of I on all the in-domain data, achieved only BLEU scores consistently worse than VII  In contrast, when we merged our two fine-tuning steps into a single mixed fine-tuning on I, we obtained a model III' which is better for the Ja INLINEFORM0 Ru pair than VII  Nevertheless, they are still comparable to those of VII and the BLEU scores for the other two language pairs are much lower than VII  As such, we conclude that our multistage fine-tuning leads to a more robust in-domain multilingual model 
Augmentation with Back-translation
Given that the M2M Transformer NMT model (b3) achieved best results for most of the translation directions and competitive results for the rest, we further explored it through back-translation 
We examined the utility of pseudo-parallel data for all the six translation directions, unlike the work of lakew2017improving and lakew2018comparison, which concentrate only on the zero-shot language pair, and the work of W18-2710, which compares only uni- or bi-directional models  We investigated whether each translation direction in M2M models will benefit from pseudo-parallel data and if so, what kind of improvement takes place 
First, we selected sentences to be back-translated from in-domain monolingual data (Table TABREF13 ), relying on the score proposed by moore:intelligent via the following procedure 
For each language, train two 4-gram language models, using KenLM: an in-domain one on all the Global Voices data, i e , both parallel and monolingual data, and a general-domain one on the concatenation of Global Voices, IWSLT, and Tatoeba data 
For each language, discard sentences containing OOVs according to the in-domain language model 
For each translation direction, select the INLINEFORM0 -best monolingual sentences in the news domain, according to the difference between cross-entropy scores given by the in-domain and general-domain language models 
Whereas W18-2710 exploited monolingual data much larger than parallel data, we maintained a 1:1 ratio between them BIBREF8 , setting INLINEFORM0 to the number of lines of parallel data of given language pair 
Selected monolingual sentences were then translated using the M2M Transformer NMT model (b3) to compose pseudo-parallel data  Then, the pseudo-parallel data were enlarged by over-sampling as summarized in Table TABREF32   Finally, new NMT models were trained on the concatenation of the original parallel and pseudo-parallel data from scratch in the same manner as the previous NMT models with the same hyper-parameters 
Table TABREF33 shows the BLEU scores achieved by several reasonable combinations of six-way pseudo-parallel data  We observed that the use of all six-way pseudo-parallel data (#10) significantly improved the base model for all the translation directions, except En INLINEFORM0 Ru  A translation direction often benefited when the pseudo-parallel data for that specific direction was used 
Summary
We have evaluated an extensive variation of MT models that rely only on in-domain parallel and monolingual data  However, the resulting BLEU scores for Ja INLINEFORM2 Ru and Ru INLINEFORM3 Ja tasks do not exceed 10 BLEU points, implying the inherent limitation of the in-domain data as well as the difficulty of these translation directions 
Exploiting Large Out-of-Domain Data Involving a Helping Language
The limitation of relying only on in-domain data demonstrated in Section SECREF4 motivates us to explore other types of parallel data  As raised in our second research question, [RQ2], we considered the effective ways to exploit out-of-domain data 
According to language pair and domain, parallel data can be classified into four categories in Table TABREF40   Among all the categories, out-of-domain data for the language pair of interest have been exploited in the domain adaptation scenarios (C INLINEFORM0 A) BIBREF9   However, for Ja INLINEFORM1 Ru, no out-of-domain data is available  To exploit out-of-domain parallel data for Ja INLINEFORM2 En and Ru INLINEFORM3 En pairs instead, we propose a multistage fine-tuning method, which combines two types of transfer learning, i e , domain adaptation for Ja INLINEFORM4 En and Ru INLINEFORM5 En (D INLINEFORM6 B) and multilingual transfer (B INLINEFORM7 A), relying on the M2M model examined in Section SECREF4   We also examined the utility of fine-tuning for iteratively generating and using pseudo-parallel data 
Multistage Fine-tuning
Simply using NMT systems trained on out-of-domain data for in-domain translation is known to perform badly  In order to effectively use large-scale out-of-domain data for our extremely low-resource task, we propose to perform domain adaptation through either (a) conventional fine-tuning, where an NMT system trained on out-of-domain data is fine-tuned only on in-domain data, or (b) mixed fine-tuning BIBREF9 , where pre-trained out-of-domain NMT system is fine-tuned using a mixture of in-domain and out-of-domain data  The same options are available for transferring from Ja INLINEFORM0 En and Ru INLINEFORM1 En to Ja INLINEFORM2 Ru 
We inevitably involve two types of transfer learning, i e , domain adaptation for Ja INLINEFORM0 En and Ru INLINEFORM1 En and multilingual transfer for Ja INLINEFORM2 Ru pair  Among several conceivable options for managing these two problems, we examined the following multistage fine-tuning 
Pre-train a multilingual model only on the Ja INLINEFORM0 En and Ru INLINEFORM1 En out-of-domain parallel data (I), where the vocabulary of the model is determined on the basis of the in-domain parallel data in the same manner as the M2M NMT models examined in Section SECREF4  
Fine-tune the pre-trained model (I) on the in-domain Ja INLINEFORM0 En and Ru INLINEFORM1 En parallel data (fine-tuning, II) or on the mixture of in-domain and out-of-domain Ja INLINEFORM2 En and Ru INLINEFORM3 En parallel data (mixed fine-tuning, III) 
Further fine-tune the models (each of II and III) for Ja INLINEFORM0 Ru on in-domain parallel data for this language pair only (fine-tuning, IV and VI) or on all the in-domain parallel data (mixed fine-tuning, V and VII) 
We chose this way due to the following two reasons  First, we need to take a balance between several different parallel corpora sizes  The other reason is division of labor; we assume that solving each sub-problem one by one should enable gradual shift of parameters 
Data Selection
As an additional large-scale out-of-domain parallel data for Ja INLINEFORM0 En, we used the cleanest 1 5M sentences from the Asian Scientific Paper Excerpt Corpus (ASPEC) BIBREF24   As for Ru INLINEFORM1 En, we used the UN and Yandex corpora released for the WMT 2018 News Translation Task  We retained Ru INLINEFORM2 En sentence pairs that contain at least one OOV token in both sides, according to the in-domain language model trained in Section SECREF34   Table TABREF45 summarizes the statistics on the remaining out-of-domain parallel data 
Further Augmentation with Back-translation
Having obtained a better model, we examined again the utility of back-translation  More precisely, we investigated (a) whether the pseudo-parallel data generated by an improved NMT model leads to a further improvement, and (b) whether one more stage of fine-tuning on the mixture of original parallel and pseudo-parallel data will result in a model better than training a new model from scratch as examined in Section SECREF34  
Given an NMT model, we first generated six-way pseudo-parallel data by translating monolingual data  For the sake of comparability, we used the identical monolingual sentences sampled in Section SECREF34   Then, we further fine-tuned the given model on the mixture of the generated pseudo-parallel data and the original parallel data, following the same over-sampling procedure in Section SECREF34   We repeated these steps five times 
Table TABREF51 shows the results  “new #10” in the second row indicates an M2M Transformer model trained from scratch on the mixture of six-way pseudo-parallel data generated by VII and the original parallel data  It achieved higher BLEU scores than #10 in Table TABREF33 thanks to the pseudo-parallel data of better quality, but underperformed the base NMT model VII  In contrast, our fine-tuned model VIII successfully surpassed VII, and one more iteration (IX) further improved BLEU scores for all translation directions, except Ru INLINEFORM0 En  Although further iterations did not necessarily gain BLEU scores, we came to a much higher plateau compared to the results in Section SECREF4  
Conclusion
In this paper, we challenged the difficult task of Ja INLINEFORM0 Ru news domain translation in an extremely low-resource setting  We empirically confirmed the limited success of well-established solutions when restricted to in-domain data  Then, to incorporate out-of-domain data, we proposed a multilingual multistage fine-tuning approach and observed that it substantially improves Ja INLINEFORM1 Ru translation by over 3 7 BLEU points compared to a strong baseline, as summarized in Table TABREF53   This paper contains an empirical comparison of several existing approaches and hence we hope that our paper can act as a guideline to researchers attempting to tackle extremely low-resource translation 
In the future, we plan to confirm further fine-tuning for each of specific translation directions  We will also explore the way to exploit out-of-domain pseudo-parallel data, better domain-adaptation approaches, and additional challenging language pairs 
Acknowledgments
This work was carried out when Aizhan Imankulova was taking up an internship at NICT, Japan  We would like to thank the reviewers for their insightful comments  A part of this work was conducted under the program “Promotion of Global Communications Plan: Research, Development, and Social Demonstration of Multilingual Speech Translation Technology” of the Ministry of Internal Affairs and Communications (MIC), Japan 
Context 3:  The E-step infers the ground truth labels given the current model parameters  The M-step updates the model parameters, including the crowd reliability parameters $\pi ^{(n)}$ ($1\le n\le N$), the keyword-specific expectation $e^{(t)}$, and the parameter of the target model $\theta ^{(t)}$  The E-step and the crowd parameter update in the M-step are similar to the Dawid-Skene model BIBREF14  The keyword expectation is inferred by taking into account both the crowd-contributed labels and the model prediction:
The parameter of the target model is updated by gradient descent  For example, when the target model to be trained is a deep neural network, we use back-propagation with gradient descent to update the weight matrices 
Experiments and Results
This section presents our experimental setup and results for evaluating our approach  We aim at answering the following questions:
[noitemsep,leftmargin=*]
Q1: How effectively does our proposed human-AI loop approach enhance the state-of-the-art machine learning models for event detection?
Q2: How well does our keyword discovery method work compare to existing keyword expansion methods?
Q3: How effective is our approach using crowdsourcing at obtaining new keywords compared with an approach labelling microposts for model training under the same cost?
Q4: How much benefit does our unified probabilistic model bring compared to methods that do not take crowd reliability into account?
Experiments and Results ::: Experimental Setup
Datasets  We perform our experiments with two predetermined event categories: cyber security (CyberAttack) and death of politicians (PoliticianDeath)  These event categories are chosen as they are representative of important event types that are of interest to many governments and companies  The need to create our own dataset was motivated by the lack of public datasets for event detection on microposts  The few available datasets do not suit our requirements  For example, the publicly available Events-2012 Twitter dataset BIBREF20 contains generic event descriptions such as Politics, Sports, Culture etc  Our work targets more specific event categories BIBREF21  Following previous studies BIBREF1, we collect event-related microposts from Twitter using 11 and 8 seed events (see Section SECREF2) for CyberAttack and PoliticianDeath, respectively  Unlabeled microposts are collected by using the keyword `hack' for CyberAttack, while for PoliticianDeath, we use a set of keywords related to `politician' and `death' (such as `bureaucrat', `dead' etc ) For each dataset, we randomly select 500 tweets from the unlabeled subset and manually label them for evaluation  Table TABREF25 shows key statistics from our two datasets 
Comparison Methods  To demonstrate the generality of our approach on different event detection models, we consider Logistic Regression (LR) BIBREF1 and Multilayer Perceptron (MLP) BIBREF2 as the target models  As the goal of our experiments is to demonstrate the effectiveness of our approach as a new model training technique, we use these widely used models  Also, we note that in our case other neural network models with more complex network architectures for event detection, such as the bi-directional LSTM BIBREF17, turn out to be less effective than a simple feedforward network  For both LR and MLP, we evaluate our proposed human-AI loop approach for keyword discovery and expectation estimation by comparing against the weakly supervised learning method proposed by BIBREF1 (BIBREF1) and BIBREF17 (BIBREF17) where only one initial keyword is used with an expectation estimated by an individual expert 
Parameter Settings  We empirically set optimal parameters based on a held-out validation set that contains 20% of the test data  These include the hyperparamters of the target model, those of our proposed probabilistic model, and the parameters used for training the target model  We explore MLP with 1, 2 and 3 hidden layers and apply a grid search in 32, 64, 128, 256, 512 for the dimension of the embeddings and that of the hidden layers  For the coefficient of expectation regularization, we follow BIBREF6 (BIBREF6) and set it to $\lambda =10 \times $ #labeled examples  For model training, we use the Adam BIBREF22 optimization algorithm for both models 
Evaluation  Following BIBREF1 (BIBREF1) and BIBREF3 (BIBREF3), we use accuracy and area under the precision-recall curve (AUC) metrics to measure the performance of our proposed approach  We note that due to the imbalance in our datasets (20% positive microposts in CyberAttack and 27% in PoliticianDeath), accuracy is dominated by negative examples; AUC, in comparison, better characterizes the discriminative power of the model 
Crowdsourcing  We chose Level 3 workers on the Figure-Eight crowdsourcing platform for our experiments  The inter-annotator agreement in micropost classification is taken into account through the EM algorithm  For keyword discovery, we filter keywords based on the frequency of the keyword being selected by the crowd  In terms of cost-effectiveness, our approach is motivated from the fact that crowdsourced data annotation can be expensive, and is thus designed with minimal crowd involvement  For each iteration, we selected 50 tweets for keyword discovery and 50 tweets for micropost classification per keyword  For a dataset with 80k tweets (e g , CyberAttack), our approach only requires to manually inspect 800 tweets (for 8 keywords), which is only 1% of the entire dataset 
Experiments and Results ::: Results of our Human-AI Loop (Q1)
Table TABREF26 reports the evaluation of our approach on both the CyberAttack and PoliticianDeath event categories  Our approach is configured such that each iteration starts with 1 new keyword discovered in the previous iteration 
Our approach improves LR by 5 17% (Accuracy) and 18 38% (AUC), and MLP by 10 71% (Accuracy) and 30 27% (AUC) on average  Such significant improvements clearly demonstrate that our approach is effective at improving model performance  We observe that the target models generally converge between the 7th and 9th iteration on both datasets when performance is measured by AUC  The performance can slightly degrade when the models are further trained for more iterations on both datasets  This is likely due to the fact that over time, the newly discovered keywords entail lower novel information for model training  For instance, for the CyberAttack dataset the new keyword in the 9th iteration `election' frequently co-occurs with the keyword `russia' in the 5th iteration (in microposts that connect Russian hackers with US elections), thus bringing limited new information for improving the model performance  As a side remark, we note that the models converge faster when performance is measured by accuracy  Such a comparison result confirms the difference between the metrics and shows the necessity for more keywords to discriminate event-related microposts from non event-related ones 
Experiments and Results ::: Comparative Results on Keyword Discovery (Q2)
Figure FIGREF31 shows the evaluation of our approach when discovering new informative keywords for model training (see Section SECREF2: Keyword Discovery)  We compare our human-AI collaborative way of discovering new keywords against a query expansion (QE) approach BIBREF23, BIBREF24 that leverages word embeddings to find similar words in the latent semantic space  Specifically, we use pre-trained word embeddings based on a large Google News dataset for query expansion  For instance, the top keywords resulting from QE for `politician' are, `deputy',`ministry',`secretary', and `minister'  For each of these keywords, we use the crowd to label a set of tweets and obtain a corresponding expectation 
We observe that our approach consistently outperforms QE by an average of $4 62\%$ and $52 58\%$ AUC on CyberAttack and PoliticianDeath, respectively  The large gap between the performance improvements for the two datasets is mainly due to the fact that microposts that are relevant for PoliticianDeath are semantically more complex than those for CyberAttack, as they encode noun-verb relationship (e g , “the king of     died    ”) rather than a simple verb (e g , “    hacked ”) for the CyberAttack microposts  QE only finds synonyms of existing keywords related to either `politician' or `death', however cannot find a meaningful keyword that fully characterizes the death of a politician  For instance, QE finds the keywords `kill' and `murder', which are semantically close to `death' but are not specifically relevant to the death of a politician  Unlike QE, our approach identifies keywords that go beyond mere synonyms and that are more directly related to the end task, i e , discriminating event-related microposts from non related ones  Examples are `demise' and `condolence'  As a remark, we note that in Figure FIGREF31(b), the increase in QE performance on PoliticianDeath is due to the keywords `deputy' and `minister', which happen to be highly indicative of the death of a politician in our dataset; these keywords are also identified by our approach 
Experiments and Results ::: Cost-Effectiveness Results (Q3)
To demonstrate the cost-effectiveness of using crowdsourcing for obtaining new keywords and consequently, their expectations, we compare the performance of our approach with an approach using crowdsourcing to only label microposts for model training at the same cost  Specifically, we conducted an additional crowdsourcing experiment where the same cost used for keyword discovery in our approach is used to label additional microposts for model training  These newly labeled microposts are used with the microposts labeled in the micropost classification task of our approach (see Section SECREF2: Micropost Classification) and the expectation of the initial keyword to train the model for comparison  The model trained in this way increases AUC by 0 87% for CyberAttack, and by 1 06% for PoliticianDeath; in comparison, our proposed approach increases AUC by 33 42% for PoliticianDeath and by 15 23% for CyberAttack over the baseline presented by BIBREF1)  These results show that using crowdsourcing for keyword discovery is significantly more cost-effective than simply using crowdsourcing to get additional labels when training the model 
Experiments and Results ::: Expectation Inference Results (Q4)
To investigate the effectiveness of our expectation inference method, we compare it against a majority voting approach, a strong baseline in truth inference BIBREF16  Figure FIGREF36 shows the result of this evaluation  We observe that our approach results in better models for both CyberAttack and PoliticianDeath  Our manual investigation reveals that workers' annotations are of high reliability, which explains the relatively good performance of majority voting  Despite limited margin for improvement, our method of expectation inference improves the performance of majority voting by $0 4\%$ and $1 19\%$ AUC on CyberAttack and PoliticianDeath, respectively 
Related Work
Event Detection  The techniques for event extraction from microblogging platforms can be classified according to their domain specificity and their detection method BIBREF0  Early works mainly focus on open domain event detection BIBREF25, BIBREF26, BIBREF27  Our work falls into the category of domain-specific event detection BIBREF21, which has drawn increasing attention due to its relevance for various applications such as cyber security BIBREF1, BIBREF2 and public health BIBREF4, BIBREF5  In terms of technique, our proposed detection method is related to the recently proposed weakly supervised learning methods BIBREF1, BIBREF17, BIBREF3  This comes in contrast with fully-supervised learning methods, which are often limited by the size of the training data (e g , a few hundred examples) BIBREF28, BIBREF29 
Human-in-the-Loop Approaches  Our work extends weakly supervised learning methods by involving humans in the loop BIBREF13  Existing human-in-the-loop approaches mainly leverage crowds to label individual data instances BIBREF9, BIBREF10 or to debug the training data BIBREF30, BIBREF31 or components BIBREF32, BIBREF33, BIBREF34 of a machine learning system  Unlike these works, we leverage crowd workers to label sampled microposts in order to obtain keyword-specific expectations, which can then be generalized to help classify microposts containing the same keyword, thus amplifying the utility of the crowd  Our work is further connected to the topic of interpretability and transparency of machine learning models BIBREF11, BIBREF35, BIBREF12, for which humans are increasingly involved, for instance for post-hoc evaluations of the model's interpretability  In contrast, our approach directly solicits informative keywords from the crowd for model training, thereby providing human-understandable explanations for the improved model 
Conclusion
In this paper, we presented a new human-AI loop approach for keyword discovery and expectation estimation to better train event detection models  Our approach takes advantage of the disagreement between the crowd and the model to discover informative keywords and leverages the joint power of the crowd and the model in expectation inference  We evaluated our approach on real-world datasets and showed that it significantly outperforms the state of the art and that it is particularly useful for detecting events where relevant microposts are semantically complex, e g , the death of a politician  As future work, we plan to parallelize the crowdsourcing tasks and optimize our pipeline in order to use our event detection approach in real-time 
Acknowledgements
This project has received funding from the Swiss National Science Foundation (grant #407540_167320 Tighten-it-All) and from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement 683253/GraphInt) 

ground_truth: pivot-based translation relying on a helping language BIBREF10, nduction of phrase tables from monolingual data BIBREF14 , attentional RNN-based model (RNMT) BIBREF2, Transformer model BIBREF18, bi-directional model BIBREF11, multi-to-multi (M2M) model BIBREF8, back-translation BIBREF17

M2M Transformer

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What type of classifiers are used?
Context 4:  If detected, the action is executed  At this step, many substeps may be performed, like searching for an information, computing maths, or generating information to create the answer  All of this may require a search in the Context and also may activate the Error Detector component to check if the dialog did not run into a wrong state  After the answer is generated, the Filtering phase is activated again to check if the reply should be really sent  If so, it is sent to the Hub which, again may check if it can be broadcasted before actually doing it 
The topic classifier is domain-dependent and is not mandatory  However, the chatbot can better react when the intent or action is not detected, which means that it does not know how to answer  Many reasons might explain this situation: the set of intents might be incomplete, the action might not have produced the proper behavior, misunderstanding might happen, or the chatbot was not designed to reply to a particular topic  In all cases, it must be able to produce a proper reply, if needed  Because this might happen throughout the workflow, the sooner that information is available, the better the chatbot reacts  Therefore it is one of the first executions of the flow 
Dependency is the notion that linguistic units, e g  words, are connected to each other by directed links  The (finite) verb is taken to be the structural center of clause structure  All other syntactic units (words) are either directly or indirectly connected to the verb in terms of the directed links, which are called dependencies  It is a one-to-one correspondence: for every element (e g  word or morph) in the sentence, there is exactly one node in the structure of that sentence that corresponds to that element  The result of this one-to-one correspondence is that dependency grammars are word (or morph) grammars  All that exist are the elements and the dependencies that connect the elements into a structure  Dependency grammar (DG) is a class of modern syntactic theories that are all based on the dependency relation 
Semantic dependencies are understood in terms of predicates and their arguments  Morphological dependencies obtain between words or parts of words  To facilitate future research in unsupervised induction of syntactic structure and to standardize best-practices, a tagset that consists of twelve universal part-of-speech categories was proposed BIBREF41  
Dependency parsers have to cope with a high degree of ambiguity and nondeterminism which let to different techniques than the ones used for parsing well-defined formal languages  Currently the mainstream approach uses algorithms that derive a potentially very large set of analyses in parallel and when disambiguation is required, this approach can be coupled with a statistical model for parse selection that ranks competing analyses with respect to plausibility BIBREF42  
Below we present an example of a dependency tree for the utterance:
"I want to invest 10 thousands":
[s]""blue[l]:black
"tree": {
"want VERB ROOT": {
"I PRON nsubj": {},
"to ADP mark": {},
"invest VERB nmod": {
"thousands NOUN nmod": {
"10 NUM nummod": {}
}
}
}
The coarse-grained part-of-speech tags, or morphological dependencies (VERB, PRON, ADP, NOUN and NUM) encode basic grammatical categories and the grammatical relationships (nsubjs, nmod, nummod) are defined in the Universal Dependencies project BIBREF41  
In this module, the dependency tree generated is used together with a set of rules to extract information that is saved in the context using the Frame-based approach  This approach fills the slots of the frame with the extracted values from the dialogue  Frames are like forms and slots are like fields  Using the knowledge's conceptual model, the fields are represented by the elements Entities and Features  In the dependency tree example, the entity would be the implicit concept: the investment option, and the feature is the implicit concept: initial amount – 10 thousands  Since the goal is to invest, and there are more entities needed for that (i e , fields to be filled), the next node in the Intention Flow tree would return an utterance which asks the user the time of investment, if he/she has not provided yet 
This module could be implemented using different approaches according to the domain, but tree search algorithms will be necessary for doing the tree parsing 
The Intent Classifier component aims at recognizing not only the Intent but the goal of the utterance sent by a Member, so it can properly react  The development of an intent classifier needs to deal with the following steps:
i) the creation of dataset of intents, to train the classification algorithm;
ii) the design of a classification algorithm that provides a reasonable level of accuracy;
iii) the creation of dataset of trees of intents, the same as defined in i) and which maps the goals;
iv) the design of a plan-graph search algorithm that maps the goal's state to a node in the graph;
There are several approaches to create training sets for dialogues: from an incremental approach to crowdsourcing  In the incremental approach, the Wizard of Oz method can be applied to a set of potential users of the system, and from this study, a set of questions that the users asked posted to the `fake' system can be collected  These questions have to be manually classified into a set of intent classes, and used to train the first version of the system  Next, this set has to be increased both in terms of number of classes and samples per class 
The Speech Act Classifier can be implemented with many speech act classes as needed by the application  The more classes, the more flexible the chatbot is  It can be built based on dictionaries, or a machine learning-based classifier can be trained  In the table below we present the main and more general speech act classes BIBREF43 used in the Chatbots with examples to differentiate one from another:
There are at least as many Action classes as Speech Act classes, since the action is the realization of a speech act  The domain specific classes, like "Inform_News" or "Inform_Factoids", enhance the capabilities of answering of a chatbot 
The Action Classifier can be defined as a multi-class classifier with the tuple DISPLAYFORM0
where INLINEFORM0 is the intent of the answer defined in ( EQREF46 ), INLINEFORM1 is the speech act of the answer, INLINEFORM2 and INLINEFORM3 are the sets of entities and features needed to produce the answer, if needed, respectively 
This component is responsible for implementing the behavior of the Action class  Basic behaviors may exist and be shared among different chatbots, like the ones that implement the greetings, thanks or not understood  Although they can be generic, they can also be personalized to differentiate the bot from one another and also to make it more "real"  Other cases like to inform, to send a query, to send a proposal, they are all domain-dependent and may require specific implementations 
Anyway, figure FIGREF59 shows at the high level the generic workflow  If action class detected is task-oriented, the system will implement the execution of the task, say to guide a car, to move a robot's arm, or to compute the return of investments  The execution might need to access an external service in the Internet in order to complete the task, like getting the inflation rate, or the interest rate, or to get information about the environment, or any external factor  During the execution or after it is finished, the utterance is generated as a reply and, if no more tasks are needed, the action execution is finished 
In the case of coordination of chatbots, one or more chatbots with the role of mediator may exist in the chat group and, at this step, it is able to invite one or more chatbots to the chat group and it is also able to redirect the utterances, if the case 
The proposed architecture addresses the challenges as the following:
What is the message/utterance about? solved by the Parsing phase;
Who should reply to the utterance? solved by the Filtering phase and may be enforced by the Hub;
How the reply should be built/generated? solved by the Acting phase;
When should the reply be sent? may be solved by the Acting phase or the Filtering phase, and may be enforced by the Hub;
And Context and Logging module is used throughout all phases 
Architecture Implementation and Evaluation
This section presents one implementation of the conceptual architecture presented in last section  After many refactorings, a framework called SABIA (Speech-Act-Based Intelligent Agents Framework) has been developed and CognIA (Cognitive Investment Advisor) application has been developed as an instantiation of SABIA framework  We present then the accuracy and some automated tests of this implementation 
Speech-Act-based Intelligent Agents Framework
SABIA was developed on top of Akka middleware  Akka is a toolkit and runtime that implements the Actor Model on the JVM  Akka's features, like concurrency, distributed computing, resilience, and message-passing were inspired by Erlang's actor model BIBREF44 BIBREF45   The actor model is a mathematical model of concurrent computation that treats "actors" as the universal primitives of concurrent computation  In response to a message that it receives, an actor can: make local decisions, create more actors, send more messages, and determine how to respond to the next received message  Actors may modify private state, but can only affect each other through messages (avoiding the need for any locks)  Akka middleware manages the actors life cycle and actors look up by theirs name, locally or remotely 
We implemented each Member of the Chat Group as an Actor by extending the UntypedActor class of Akka middleware  Yet, we created and implemented the SabiaActorSystem as a singleton (i e , a single instance of it exists in the system) BIBREF46 that has a reference to Akka's ActorSystem  During SabiaActorSystem's initialization, all parsers that consume too much memory during their initialization to load models are instantiated as singletons  In this way, we save time on their calls during the runtime  Moreover, all chat group management, like to join or leave the group, or to broadcast or filter a message at the Hub level is implemented in SABIA through the Chat Group behavior 
This is implemented in SABIA as a singleton that is initialized during the SabiaActorSystem initialization with the URL of the service that implements the dependency parsing and is used on each utterance's arrival through the execution of the tagUtterance method  The service must retrieve a JSON Object with the dependency tree which is then parsed using depth-first search 
SABIA does not support invariants for frame parsing  We are leaving this task to the instantiated application 
There are two intent classifiers that can be loaded with trained models in order to be ready to be used at runtime: the 1-nearest-neighbor (1NN) and the SVM-based classifier 
SABIA implements the Action Classifier assuming that the application uses a relational database with a data schema that implements the conceptual model presented in Figure FIGREF44   Then the invariants parts that use SQL are already present and the application only needs to implement the database connection and follow the required data schema 
SABIA provides partial implemented behavior for the Action through the Template method design pattern BIBREF46 , which implements the invariants parts of the action execution and leaves placeholders for customization 
CognIA: A Cognitive Investment Advisor
We developed CognIA, which is an instantiation of Sabia framework  A conversation is composed of a group chat that can contain multiple users and multiple chatbots  This example, in particular, has a mediator that can help users on financial matters, more specifically on investment options  For example, consider the following dialogue in the table below:
The Table TABREF71 shows an example that uses the mixed-initiative dialogue strategy, and a dialogue mediator to provide coordination control  In this example of an application, there are many types of intentions that should be answered: Q&A (question and answer) about definitions, investment options, and about the current finance indexes, simulation of investments, which is task-oriented and requires computation, and opinions, which can be highly subjective 
In Table SECREF72 , we present the interaction norms that were needed in Cognia  The Trigger column describes the event that triggers the Behavior specified in the third column  The Pre-Conditions column specifies what must happen in order to start the behavior execution  So, for instance, line 2, when the user sends an utterance in the chat group, an event is triggered and, if the utterance's topic is CDB (Certificate of Deposit which is a fixed rate investment) or if it is about the Savings Account investment option and the speech act is not Query_Calculation and the CDB and Savings Account members are not in the chat, then the behavior is activated  The bot members that implement these behaviors are called cdbguru and poupancaguru  Therefore these names are used when there is a mention 
Note that these interactions norms are not explicitly defined as obligations, permissions, and prohibitions  They are implict from the behavior described  During this implementation, we did not worry about explicitly defining the norms, because the goal was to evaluate the overall architecture, not to enhance the state of the art on norms specification for conversational systems  In addition, CognIA has only the presented interaction norms defined in Table SECREF72 , which is a very small set that that does not required model checking or verification of conflicts 
|p2cm|p5 0cm|p5 0cm|Cognia Interaction NormsCognia Interaction Norms
Trigger Pre-Conditions Behavior
On group chat creation Cognia chatbot is available Cognia chatbot joins the chat with the mediator role and user joins the chat with the owner_user role
On utterance sent by user Utterance's topic is CDB (cdbguru) or Savings Account (poupancaguru) and speech act is not Query_Calculation and they are not in the chat Cognia invites experts to the chat and repeats the utterance to them
On utterance sent by user Utterance's topic is CDB (cdbguru) or Savings Account (poupancaguru) and speech act is not Query_Calculation and they are in the chat Cognia waits for while and cdbguru or poupancaguru respectively handles the utterance  If they don't understand, they don't reply
On utterance sent by the experts If Cognia is waiting for them and has received both replies Cognia does not wait anymore
On utterance sent Utterance mentions cdbguru or poupancaguru cdbguru or poupancaguru respectively handles the utterance
On utterance sent Utterance mentions cdbguru or poupancaguru and they don't reply after a while and speech act is Query_Calculation Cognia sends I can only chat about investments   
On utterance sent Utterance mentions cdbguru or poupancaguru and they don't reply after while and speech act is not Query_Calculation Cognia sends I didn't understand
On utterance sent Utterance's speech act is Query_Calculation and period or initial amount of investment were not specified Cognia asks the user the missing information
On utterance sent Utterance's speech act is Query_Calculation and period and initial amount of investment were specified and the experts are not in the chat Cognia invites experts to the chat and repeats the utterance to them
On utterance sent Utterance's speech act is Query_Calculation and period and initial amount of investment were specified and the experts are in the chat Cognia repeats the utterance to experts
On utterance sent Utterance's speech act is Query_Calculation Cognia extracts variables and saves the context
On utterance sent Utterance's speech act is Query_Calculation and the experts are in the chat and the experts are mentioned Experts extract information, save in the context, compute calculation and send information
On utterance sent Utterance's speech act is Inform_Calculation and Cognia received all replies Cognia compares the results and inform comparison
On utterance sent Utterance mentions a chatbot but has no other text The chatbot replies How can I help you?
On utterance sent Utterance is not understood and speech act is Question The chatbot replies I don't know    I can only talk about topic X
On utterance sent Utterance is not understood and speech act is not Question The chatbot replies I didn't understand
On utterance sent Utterance's speech act is one of { Greetings, Thank, Bye } All chatbots reply to utterance
On group chat end All chatbots leave the chat, and the date and time of the end of chat is registered
We instantiated SABIA to develop CognIA as follows: the Mediator, Savings Account, CDB and User Actors are the Members of the Chat Group  The Hub was implemented using two servers: Socket io and Node JS which is a socket client of the Socket io server  The CognIA system has also one Socket Client for receiving the broadcast and forwarding to the Group Chat Manager  The former will actually do the broadcast to every member after enforcing the norms that applies specified in Table SECREF72   Each Member will behave according to this table too  For each user of the chat group, on a mobile or a desktop, there is its corresponding actor represented by the User Actor in the figure  Its main job is to receive Akka's broadcast and forward to the Socket io server, so it can be finally propagated to the users 
All the intents, actions, factual answers, context and logging data are saved in DashDB (a relational Database-as-a-Service system)  When an answer is not retrieved, a service which executes the module Search Finance on Social Media on a separate server is called  This service was implemented with the assumption that finance experts post relevant questions and answers on social media  Further details are explained in the Action execution sub-section 
We built a small dictionary-based topic classifier to identify if an utterance refers to finance or not, and if it refers to the two investment options (CDB or Savings Account) or not 
The dependency parsing is extremely important for computing the return of investment when the user sends an utterance with this intention  Our first implementation used regular expressions which led to a very fragile approach  Then we used a TensorFlow implementation BIBREF47 of a SyntaxNet model for Portuguese and used it to generate the dependency parse trees of the utterances  The SyntaxNet model is a feed-forward neural network that operates on a task-specific transition system and achieves the state-of-the-art on part-of-speech tagging, dependency parsing and sentence compression results BIBREF48   Below we present output of the service for the utterance:
"I want to invest 10 thousands in 40 months":
[s]""blue[l]:black
{ "original": "I would like to invest 10 thousands in 40 months",
"start_pos": [
23,
32],
"end_pos": [
27,
33],
"digits": [
10000,
40],
"converted": "I would like to invest 10000 in 40 months",
"tree": {
"like VERB ROOT": {
"I PRON nsubj": {},
"would MD aux":{
"invest VERB xcomp":{
"to TO aux": {},
"10000 NUM dobj": {},
"in IN prep": {
"months NOUN pobj":{
"40 NUM num": {}}}}}}}
The service returns a JSON Object containing six fields: original, start_pos, end_pos, digits, converted and tree  The original field contains the original utterance sent to the service  The converted field contains the utterance replaced with decimal numbers, if the case (for instance, "10 thousands" was converted to "10000" and replaced in the utterance)  The start_pos and end_pos are arrays that contain the start and end char positions of the numbers in the converted utterance  While the tree contains the dependency parse tree for the converted utterance 
Given the dependency tree, we implemented the frame parsing which first extracts the entities and features from the utterance and saves them in the context  Then, it replaces the extracted entities and features for reserved characters 
extract_period_of_investment (utteranceTree) [1] [t] numbersNodes INLINEFORM0 utteranceTree getNumbersNodes(); [t] foreach(numberNode in numbersNodes) do [t] parentsOfNumbersNode INLINEFORM1 numbersNode getParents() [t] foreach(parent in parentsOfNumbersNodes) do [t] if ( parent name contains { "day", "month", "year"} ) then [t] parentOfParent INLINEFORM2 parent getParent() [t] if ( parentOfParent is not null and
parentOfParent getPosTag==Verb and
parentOfParent name in investmentVerbsSet ) then [t] return numberNode
Therefore an utterance like "I would like to invest 10 thousands in 3 years" becomes "I would like to invest #v in #dt years"  Or "10 in 3 years" becomes "#v in #dt years", and both intents have the same intent class 
For doing that we implemented a few rules using a depth-first search algorithm combined with the rules as described in Algorithm UID79 , Algorithm UID79 and Algorithm UID79   Note that our parser works only for short texts on which the user's utterance mentions only one period of time and/ or initial amount of investment in the same utterance 
extract_initial_amount_of_investment (utteranceTree) [1] [t] numbersNodes INLINEFORM0 utteranceTree getNumbersNodes(); [t] foreach(numberNode in numbersNodes) do [t] parentsOfNumbersNode INLINEFORM1 numbersNode getParents() [t] foreach(parent in parentsOfNumbersNodes) do [t] if ( parent name does not contain { "day", "month", "year"} ) then [t] return numberNode
frame_parsing(utterance, utteranceTree) [1] [t] period INLINEFORM0 extract_period_of_investment (utteranceTree) [t] save_period_of_investment(period) [t] value INLINEFORM1 extract_initial_amount_of_investment (utteranceTree) [t] save_initial_amount_of_investment(value) [t] new_intent INLINEFORM2 replace(new_intent, period, "#dt") [t] new_intent INLINEFORM3 replace(new_intent, value, "#v")
In CognIA we have complemented the speech act classes with the ones related to the execution of specific actions  Therefore, if the chatbot needed to compute the return of investment, then, once it is computed, the speech act of the reply will be Inform_Calculation and the one that represents the query for that is Query_Calculation  In table TABREF81 we list the specific ones 
Given that there is no public dataset available with financial intents in Portuguese, we have employed the incremental approach to create our own training set for the Intent Classifier  First, we applied the Wizard of Oz method and from this study, we have collected a set of 124 questions that the users asked  Next, after these questions have been manually classified into a set of intent classes, and used to train the first version of the system, this set has been increased both in terms of number of classes and samples per class, resulting in a training set with 37 classes of intents, and a total 415 samples, with samples per class ranging from 3 to 37 
We have defined our classification method based on features extracted from word vectors  Word vectors consist of a way to encode the semantic meaning of the words, based on their frequency of co-occurrence  To create domain-specific word vectors, a set of thousand documents are needed related to desired domain  Then each intent from the training set needs to be encoded with its corresponding mean word vector  The mean word vector is then used as feature vector for standard classifiers 
We have created domain-specific word vectors by considering a set 246,945 documents, corresponding to of 184,001 Twitter posts and and 62,949 news articles, all related to finance  
The set of tweets has been crawled from the feeds of blog users who are considered experts in the finance domain  The news article have been extracted from links included in these tweets  This set contained a total of 63,270,124 word occurrences, with a vocabulary of 97,616 distinct words  With the aforementioned word vectors, each intent from the training set has been encoded with its corresponding mean word vector  The mean word vector has been then used as feature vector for standard classifiers 
As the base classifier, we have pursued with a two-step approach  In the first step, the main goal was to make use of a classifier that could be easily retrained to include new classes and intents  For this reason, the first implementation of the system considered an 1-nearest-neighbor (1NN) classifier, which is simply a K-nearest-neighbor classifier with K set to 1  With 1NN, the developer of the system could simply add new intents and classes to the classifier, by means of inserting new lines into the database storing the training set  Once we have considered that the training set was stable enough for the system, we moved the focus to an approach that would be able to provide higher accuracy rates than 1NN  For this, we have employed Support Vector Machines (SVM) with a Gaussian kernel, the parameters of which are optimized by means of a grid search 
We manually mapped the intent classes used to train the intent classifier to action classes and the dependent entities and features, when the case  Table TABREF85 summarizes the number of intent classes per action class that we used in CognIA 
For the majority of action classes we used SABIA's default behavior  For instance, Greet and Bye actions classes are implemented using rapport, which means that if the user says "Hi" the chatbot will reply "Hi" 
The Search News, Compute and Ask More classes are the ones that require specific implemention for CognIA as following:
Search News: search finance on social media service BIBREF49 , BIBREF50 receives the utterance as input, searches on previously indexed Twitter data for finance for Portuguese and return to the one which has the highest score, if found 
Ask More: If the user sends an utterance that has the intention class of simulating the return of investment, while not all variables to compute the return of investment are extracted from the dialogue, the mediator keeps asking the user these information before it actually redirects the query to the experts  This action then checks the state of the context given the specified intent flow as described in ( EQREF46 ) and ( EQREF57 ) in section SECREF4 to decide which variables are missing  For CognIA we manually added these dependencies on the database 
Compute: Each expert Chatbot implements this action according to its expertise  The savings account chatbot computes the formula ( EQREF90 ) and the certificate of deposit computes the formula ( EQREF92 )  Both are currently formulas for estimating in Brazil  DISPLAYFORM0
where INLINEFORM0 is the return of investment for the savings account, INLINEFORM1 is the initial value of investment, INLINEFORM2 is the savings account interest rate and INLINEFORM3 is the savings account rate base  DISPLAYFORM0
where INLINEFORM0 is the return of investment for certificate of deposit, INLINEFORM1 is the initial value of investment, INLINEFORM2 is the Interbank Deposit rate (DI in Portuguese), INLINEFORM3 is the ID's percentual payed by the bank (varies from 90% to 120%), INLINEFORM4 is the number of days the money is invested, and finally INLINEFORM5 is the income tax on the earnings 
Intention Classifier Accuracy
In Table TABREF95 we present the comparison of some distinct classification on the first version of the training set, i e  the set used to deploy the first classifier into the system  Roughly speaking, the 1NN classifier has been able to achieve a level of accuracy that is higher than other well-known classifiers, such as Logistic Regression and Naïve Bayes, showing that 1NN is suitable as a development classifier  Nevertheless, a SVM can perform considerable better than 1NN, reaching accuracies of about 12 percentage points higher, which demonstrates that this type of base classifier is a better choice to be deployed once the system is stable enough  It is worth mentioning that these results consider the leave-one-out validation procedure, given the very low number of samples in some classes 
As we mentioned, the use of an 1NN classifier has allowed the developer of the system to easily add new intent classes and samples whenever they judged it necessary, so that the system could present new actions, or the understanding of the intents could be improved  As a consequence, the initial training set grew from 37 to 63 classes, and from 415 to 659 samples, with the number of samples per class varying from 2 to 63  For visualizing the impact on the accuracy of the system, in Table TABREF96 we present the accuracy of the same classifiers used in the previous evaluation, in the new set  In this case, we observe some drop in accuracy for 1NN, showing that this classifier suffers in dealing with scalability  On the other hand, SVM has shown to scale very well to more classes and samples, since its accuracy kept at a very similar level than that with the other set, with a difference of about only 1 percentage point 
Testing SABIA
In this section, we describe the validation framework that we created for integration tests  For this, we developed it as a new component of SABIA's system architecture and it provides a high level language which is able to specify interaction scenarios that simulate users interacting with the deployed chatbots  The system testers provide a set of utterances and their corresponding expected responses, and the framework automatically simulates users interacting with the bots and collect metrics, such as time taken to answer an utterance and other resource consumption metrics (e g , memory, CPU, network bandwidth)  Our goal was to: (i) provide a tool for integration tests, (ii) to validate CognIA's implementation, and (iii) to support the system developers in understanding the behavior of the system and which aspects can be improved  Thus, whenever developers modify the system's source code, the modifications must first pass the automatic test before actual deployment 
The test framework works as follows  The system testers provide a set INLINEFORM0 of dialogues as input  Each dialogue INLINEFORM1 INLINEFORM2 INLINEFORM3 is an ordered set whose elements are represented by INLINEFORM4 , where INLINEFORM5 is the user utterance and INLINEFORM6 is an ordered set of pairs INLINEFORM7 that lists each response INLINEFORM8 each chatbot INLINEFORM9 should respond when the user says INLINEFORM10   For instance, Table UID98 shows a typical dialogue ( INLINEFORM11 ) between a user and the CognIA system  Note that we are omitting part of the expected answer with "   " just to better visualize the content of the table 
|p3 6cmp0 4cmp4 5cmp3 2cm|Content of dialogue INLINEFORM0 (example of dialogue in CognIA)Content of dialogue INLINEFORM1 (example of dialogue in CognIA
User utterance INLINEFORM0 rId Expected response INLINEFORM1 Chatbot INLINEFORM2
gray!25 hello 1 Hello Mediator
white what is cdb? 2 @CDBExpert what is cdb? Mediator
white 3 CDB is a type of investment that    CDB Expert
gray!25 which is better: cdb or savings account? 4 I found a post in the social media for     Mediator
white i would like to invest R$ 50 in six months 5 @SavingsAccountExpert and @CDBExpert, could you do a simulation    Mediator
white 6 If you invest in Savings Account,     Savings Account Exp 
white 7 If you invest in CDB,    CDB Expert
white 8 Thanks Mediator
white 9 @User, there is no significant difference   Mediator
gray!25 so i want to invest R$ 10000 in 2 years 10 @SavingsAccountExpert and @CDBExpert, could you do a simulation    Mediator
gray!25 11 If you invest in Savings Account,    Savings Account Exp 
gray!25 12 If you invest in CDB,    CDB Expert
gray!25 13 Thanks Mediator
gray!25 14 @User, in that case, it is better    Mediator
white what if i invest R$10,000 in 5 years? 15 @SavingsAccountExpert and @CDBExpert, could you do a simulation    Mediator
white 16 If you invest in Saving Account,    Savings Account Exp 
white 17 If you invest in CDB,    CDB Expert
white 18 Thanks Mediator
white 19 @User, in that case, it is better    Mediator
gray!25 how about 15 years? 20 @SavingsAccountExpert and @CDBExpert, could you do a simulation    Mediator
gray!25 21 If you invest in Savings Account,    Savings Account Exp
gray!25 22 If you invest in CDB,    CDB Expert
gray!25 23 Thanks Mediator
gray!25 24 @User, in that case, it is better    Mediator
white and 50,0000? 25 @SavingsAccountExpert and @CDBExpert, could you do a simulation    Mediator
white 26 If you invest in Savings Account,    Savings Account Exp 
white 27 If you invest in CDB,    CDB Expert
white 28 Thanks Mediator
white 29 @User, in that case, it is better   Mediator
gray!25 I want to invest in 50,000 for 15 years in CDB 30 Sure, follow this link to your bank    Mediator
white thanks 31 You are welcome  Mediator
The testers may also inform the number of simulated users that will concurrently use the platform  Then, for each simulated user, the test framework iterates over the dialogues in INLINEFORM0 and iterates over the elements in each dialogue to check if each utterance INLINEFORM1 was correctly responded with INLINEFORM2 by the chatbot INLINEFORM3   There is a maximum time to wait  If a bot does not respond with the expected response in the maximum time (defined by the system developers), an error is raised and the test is stopped to inform the developers about the error  Otherwise, for each correct bot response, the test framework collects the time taken to respond that specific utterance by the bot for that specific user and continues for the next user utterance  Other consumption resource metrics (memory, CPU, network, disk)  The framework is divided into two parts  One part is responsible to gather resource consumption metrics and it resides inside SABIA  The other part works as clients (users) interacting with the server  It collects information about time taken to answer utterances and checks if the utterances are answered correctly 
By doing this, we not only provide a sanity test for the domain application (CognIA) developed in SABIA framework, but also a performance analysis of the platform  That is, we can: validate if the bots are answering correctly given a pre-defined set of known dialogues, check if they are answering in a reasonable time, and verify the amount of computing resources that were consumed to answer a specific utterance  Given the complexity of CognIA, these tests enable debugging of specific features like: understanding the amount of network bandwidth to use external services, or analyzing CPU and memory consumption when responding a specific utterance  The later may happen when the system is performing more complex calculations to indicate the investment return, for instance 
CognIA was deployed on IBM Bluemix, a platform as a service, on a Liberty for Java Cloud Foundry app with 3 GB RAM memory and 1 GB disk  Each of the modules shown in Figure FIGREF74 are deployed on separate Bluemix servers  Node JS and Socket IO servers are both deployed as Node Cloud Foundry apps, with 256 MB RAM memory and 512 MB disk each  Search Finance on Social Media is on a Go build pack Cloud Foundry app with 128 MB RAM memory and 128 GB disk  For the framework part that simulates clients, we instantiated a virtual machine with 8 cores on IBM's SoftLayer that is able to communicate with Bluemix  Then, the system testers built two dialogues, i e , INLINEFORM0   The example shown in Table UID98 is the dialogue test INLINEFORM1   For the dialogue INLINEFORM2 , although it also has 10 utterances, the testers varied some of them to check if other utterances in the finance domain (different from the ones in dialogue INLINEFORM3 ) are being responded as expected by the bots  Then, two tests are performed and the results are analyzed next  All tests were repeated until the standard deviation of the values was less than 1%  The results presented next are the average of these values within the 1% margin 
Test 1: The first test consists of running both dialogues INLINEFORM0 and INLINEFORM1 for only one user for sanity check  We set 30 seconds as the maximum time a simulated user should wait for a bot correct response before raising an error  The result is that all chatbots (Mediator, CDBExpert, and SavingsAccountExpert) responded all expected responses before the maximum time  Additionally, the framework collected how long each chatbot took to respond an expected answer 
In Figure FIGREF101 , we show the results for those time measurements for dialogue INLINEFORM0 , as for the dialogue INLINEFORM1 the results are approximately the same  The x-axis (Response Identifier) corresponds to the second column (Resp  Id) in Table UID98   We can see, for example, that when the bot CDBExpert responds with the message 3 to the user utterance "what is cdb?", it is the only bot that takes time different than zero to answer, which is the expected behavior  We can also see that the Mediator bot is the one that takes the longest, as it is responsible to coordinate the other bots and the entire dialogue with the user  Moreover, when the expert bots (CDBExpert and SavingsAccountExpert) are called by the Mediator to respond to the simulation calculations (this happens in responses 6, 7, 11, 12, 16, 17, 21, 22, 26, 27), they take approximately the same to respond  Finally, we see that when the concluding responses to the simulation calculations are given by the Mediator (this happens in responses 9, 14, 19, 24, 29), the response times reaches the greatest values, being 20 seconds the greatest value in response 19  These results support the system developers to understand the behavior of the system when simulated users interact with it and then focus on specific messages that are taking longer 
Test 2: This test consists of running dialogue INLINEFORM0 , but now using eight concurrent simulated users  We set the maximum time to wait to 240 seconds, i e , eight times the maximum set up for the single user in Test 1  The results are illustrated in Figure FIGREF102 , where we show the median time for the eight users  The maximum and minimum values are also presented with horizontal markers  Note that differently than what has been shown in Figure FIGREF101 , where each series represents one specific chatbot, in Figure FIGREF102 , the series represents the median response time for the responses in the order (x-axis) they are responded, regardless the chatbot 
Comparing the results in Figure FIGREF102 with the ones in Figure FIGREF101 , we can see that the bots take longer to respond when eight users are concurrently using the platform than when a single user uses it, as expected  For example, CDBExpert takes approximately 5 times longer to respond response 3 to eight users than to respond to one user  On average, the concluding responses to the simulation questions (i e , responses 9, 14, 19, 24, 29) take approximately 7 3 times more to be responded with eight users than with one user, being the response 9 the one that presented greatest difference (11 4 times longer with eight users than with one)  These results help the system developers to diagnose the scalability of the system architecture and to plan sizing and improvements 
Conclusions and Future Work
In this article, we explored the challenges of engineering MPCS and we have presented a hybrid conceptual architecture and its implementation with a finance advisory system 
We are currently evolving this architecture to be able to support decoupled interaction norms specification, and we are also developing a multi-party governance service that uses that specification to enforce exchange of compliant utterances 
In addition, we are exploring a micro-service implementation of SABIA in order to increase its scalability and performance, so thousands of members can join the system within thousands of conversations 
Acknowledgments
The authors would like to thank Maximilien de Bayser, Ana Paula Appel, Flavio Figueiredo and Marisa Vasconcellos, who contributed with discussions during SABIA and CognIA's implementation 

ground_truth: probabilistic model

Logistic Regression, Multilayer Perceptron