prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How big is imbalance in analyzed corpora?
Context 1: Introduction
In recent years, gender has become a hot topic within the political, societal and research spheres  Numerous studies have been conducted in order to evaluate the presence of women in media, often revealing their under-representation, such as the Global Media Monitoring Project BIBREF0  In the French context, the CSA BIBREF1 produces a report on gender representation in media on a yearly basis  The 2017 report shows that women represent 40% of French media speakers, with a significant drop during high-audience hours (6:00-8:00pm) reaching a value of only 29%  Another large scale study confirmed this trend with an automatic analysis of gender in French audiovisuals streams, highlighting a huge variation across type of shows BIBREF2 
Besides the social impact of gender representation, broadcast recordings are also a valuable source of data for the speech processing community  Indeed, automatic speech recognition (ASR) systems require large amount of annotated speech data to be efficiently trained, which leaves us facing the emerging concern about the fact that "AI artifacts tend to reflect the goals, knowledge and experience of their creators" BIBREF3  Since we know that women are under-represented in media and that the AI discipline has retained a male-oriented focus BIBREF4, we can legitimately wonder about the impact of using such data as a training set for ASR technologies  This concern is strengthened by the recent works uncovering gender bias in several natural language processing (NLP) tools such as BIBREF5, BIBREF6, BIBREF7, BIBREF8 
In this paper, we first highlight the importance of TV and radio broadcast as a source of data for ASR, and the potential impact it can have  We then perform a statistical analysis of gender representation in a data set composed of four state-of-the-art corpora of French broadcast, widely used within the speech community  Finally we question the impact of such a representation on the systems developed on this data, through the perspective of an ASR system 
From gender representation in data to gender bias in AI ::: On the importance of data
The ever growing use of machine learning in science has been enabled by several progresses among which the exponential growth of data available  The quality of a system now depends mostly on the quality and quantity of the data it has been trained on  If it does not discard the importance of an appropriate architecture, it reaffirms the fact that rich and large corpora are a valuable resource  Corpora are research contributions which do not only allow to save and observe certain phenomena or validate a hypothesis or model, but are also a mandatory part of the technology development  This trend is notably observable within the NLP field, where industrial technologies, such as Apple, Amazon or Google vocal assistants now reach high performance level partly due to the amount of data possessed by these companies BIBREF9 
Surprisingly, as data is said to be “the new oil", few data sets are available for ASR systems  The best known are corpora like TIMIT BIBREF10, Switchboard BIBREF11 or Fisher BIBREF12 which date back to the early 1990s  The scarceness of available corpora is justified by the fact that gathering and annotating audio data is costly both in terms of money and time  Telephone conversations and broadcast recordings have been the primary source of spontaneous speech used  Out of all the 130 audio resources proposed by LDC to train automatic speech recognition systems in English, approximately 14% of them are based on broadcast news and conversation  For French speech technologies, four corpora containing radio and TV broadcast are the most widely used: ESTER1 BIBREF13, ESTER2 BIBREF14, ETAPE BIBREF15 and REPERE BIBREF16  These four corpora have been built alongside evaluation campaigns and are still, to our knowledge, the largest French ones of their type available to date 
From gender representation in data to gender bias in AI ::: From data to bias
The gender issue has returned to the forefront of the media scene in recent years and with the emergence of AI technologies in our daily lives, gender bias has become a scientific topic that researchers are just beginning to address  Several studies revealed the existence of gender bias in AI technologies such as face recognition (GenderShades BIBREF17), NLP (word embeddings BIBREF5 and semantics BIBREF6) and machine translation (BIBREF18, BIBREF7)  The impact of the training data used within these deep-learning algorithms is therefore questioned 
Bias can be found at different levels as pointed out by BIBREF19  BIBREF20 defines bias as a skew that produces a type of harm  She distinguishes two types of harms that are allocation harm and representation harm  The allocation harm occurs when a system is performing better or worse for a certain group while representational harm contributes to the perpetuation of stereotypes  Both types of harm are the results of bias in machine learning that often comes from the data systems are trained on  Disparities in representation in our social structures is captured and reflected by the training data, through statistical patterns  The GenderShades study is a striking example of what data disparity and lack of representation can produce: the authors tested several gender recognition modules used by facial recognition tools and found difference in error-rate as high as 34 percentage points between recognition of white male and black female faces  The scarce presence of women and colored people in training set resulted in bias in performance towards these two categories, with a strong intersectional bias  As written by BIBREF21 "A data set may have many millions of pieces of data, but this does not mean it is random or representative  To make statistical claims about a data set, we need to know where data is coming from; it is similarly important to know and account for the weaknesses in that data " (p 668) 
Regarding ASR technology, little work has explored the presence of gender bias within the systems and no consensus has been reached  BIBREF22 found that speech recognizers perform better on female voice on a broadcast news and telephone corpus  They proposed several explanations to this observation, such as the larger presence of non-professional male speech in the broadcast data, implying a less prepared speech for these speakers or a more normative language and standard pronunciation for women linked to the traditional role of women in language acquisition and education  The same trend was observed by BIBREF23  More recently, BIBREF24 discovered a gender bias within YouTube's automatic captioning system but this bias was not observed in a second study evaluating Bing Speech system and YouTube Automatic Captions on a larger data set BIBREF8  However race and dialect bias were found  General American speakers and white speakers had the lowest error rate for both systems  If the better performance on General American speakers could be explained by the fact that they are all voice professionals, producing clear and articulated speech, but no explanation is provided for biases towards non-white speakers 
Gender bias in ASR technology is still an open research question as no clear answer has been reached so far  It seems that many parameters are to take into account to achieve a general agreement  As we established the importance of TV and radio broadcast as a source of data for ASR, and the potential impact it can have, the following content of this paper is structured as this: we first describe statistically the gender representation of a data set composed of four state-of-the-art corpora of French broadcast, widely used within the speech community, introducing the notion of speaker's role to refine our analysis in terms of voice professionalism  We then question the impact of such a representation on a ASR system trained on these data  BIBREF25
Methodology
This section is organized as follows: we first present the data we are working on  In a second time we explain how we proceed to describe the gender representation in our corpus and introduce the notion of speaker's role  The third subsection introduces the ASR system and metrics used to evaluate gender bias in performance 
Methodology ::: Data presentation
Our data consists of two sets used to train and evaluate our automatic speech recognition system  Four major evaluation campaigns have enabled the creation of wide corpora of French broadcast speech: ESTER1 BIBREF13, ESTER2 BIBREF14, ETAPE BIBREF15 and REPERE BIBREF16  These four collections contain radio and/or TV broadcasts aired between 1998 and 2013 which are used by most academic researchers in ASR  Show duration varies between 10min and an hour  As years went by and speech processing research was progressing, the difficulty of the tasks augmented and the content of these evaluation corpora changed  ESTER1 and ESTER2 mainly contain prepared speech such as broadcast news, whereas ETAPE and REPERE consists also of debates and entertainment shows, spontaneous speech introducing more difficulty in its recognition 
Our training set contains 27,085 speech utterances produced by 2,506 speakers, accounting for approximately 100 hours of speech  Our evaluation set contains 74,064 speech utterances produced by 1,268 speakers for a total of 70 hours of speech  Training data by show, medium and speech type is summarized in Table and evaluation data in Table   Evaluation data has a higher variety of shows with both prepared (P) and spontaneous (S) speech type (accented speech from African radio broadcast is also included in the evaluation set) 
Methodology ::: Methodology for descriptive analysis of gender representation in training data
We first describe the gender representation in training data  Gender representation is measured in terms of number of speakers, number of utterances (or speech turns), and turn lengths (descriptive statistics are given in Section SECREF16)  Each speech turn was mapped to its speaker in order to associate it with a gender 
As pointed out by the CSA report BIBREF1, women presence tends to be marginal within the high-audience hours, showing that women are represented but less than men and within certain given conditions  It is clear that a small number of speakers is responsible for a large number of speech turns  Most of these speakers are journalists, politicians, presenters and such, who are representative of a show  Therefore, we introduce the notion of speaker's role to refine our exploration of gender disparity, following studies which quantified women's presence in terms of role  Within our work, we define the notion of speaker role by two criteria specifying the speaker's on-air presence, namely the number of speech turns and the cumulative duration of his or her speaking time in a show  Based on the available speech transcriptions and meta-data, we compute for each speaker the number of speech turns uttered as well as their total length  We then use the following criteria to define speaker's role: a speaker is considered as speaking often (respectively seldom) if he/she accumulates a total of turns higher (respectively lower) than 1% of the total number of speech turns in a given show  The same process is applied to identify speakers talking for a long period from those who do not  We end up with two salient roles called Anchors and Punctual speakers:
the Anchor speakers (A) are above the threshold of 1% for both criteria, meaning they are intervening often and for a long time thus holding an important place in interaction;
the Punctual speakers (PS) on the contrary are below the threshold of 1% for both the total number of turns and the total speech time 
These roles are defined at the show level  They could be roughly assimilated to the categorization “host/guest” in radio and TV shows  Anchors could be described as professional speakers, producing mostly prepared speech, whereas Punctual speakers are more likely to be “everyday people"  The concept of speaker's role makes sense at both sociological and technical levels  An Anchor speaker is more likely to be known from the audience (society), but he or she will also likely have a professional (clear) way of speaking (as mentioned by BIBREF22 and BIBREF8), as well as a high number of utterances, augmenting the amount of data available for a given gender category 
Methodology ::: Gender bias evaluation procedure of an ASR system performance ::: ASR system
The ASR system used in this work is described in BIBREF25  It uses the KALDI toolkit BIBREF26, following a standard Kaldi recipe  The acoustic model is based on a hybrid HMM-DNN architecture and trained on the data summarized in Table   Acoustic training data correspond to 100h of non-spontaneous speech type (mostly broadcast news) coming from both radio and TV shows  A 5-gram language model is trained from several French corpora (3,323M words in total) using SRILM toolkit BIBREF27  The pronunciation model is developed using the lexical resource BDLEX BIBREF28 as well as automatic grapheme-to-phoneme (G2P) transcription to find pronunciation variants of our vocabulary (limited to 80K)  It is important to re-specify here, for further analysis, that our Kaldi pipeline follows speaker adaptive training (SAT) where we train and decode using speaker adapted features (fMLLR-adapted features) in per-speaker mode  It is well known that speaker adaptation acts as an effective procedure to reduce mismatch between training and evaluation conditions BIBREF29, BIBREF26 
Methodology ::: Gender bias evaluation procedure of an ASR system performance ::: Evaluation
Word Error Rate (WER) is a common metric to evaluate ASR performance  It is measured as the sum of errors (insertions, deletions and substitutions) divided by the total number of words in the reference transcription  As we are investigating the impact on performance of speaker's gender and role, we computed the WER for each speaker at the episode (show occurrence) level  Analyzing at such granularity allows us to avoid large WER variation that could be observed at utterance level (especially for short speech turns) but also makes possible to get several WER values for a given speaker, one for each occurrence of a show in which he/she appears on  Speaker's gender was provided by the meta-data and role was obtained using the criteria from Section SECREF6 computed for each show  This enables us to analyze our results across gender and role categories which was done using Wilcoxon rank sum tests also called Mann-Whitney U test (with $\alpha $= 0 001) BIBREF30  The choice of a Wilcoxon rank sum test and not the commonly used t-test is motivated by the non-normality of our data 
Results ::: Descriptive analysis of gender representation in training data ::: Gender representation
As expected, we observe a disparity in terms of gender representation in our data (see Table )  Women represent 33 16% of the speakers, confirming the figures given by the GMMP report BIBREF0  However, it is worth noticing that women account for only 22 57% of the total speech time, which leads us to conclude that women also speak less than men 
Results ::: Descriptive analysis of gender representation in training data ::: Speaker's role representation
Table presents roles' representation in training data and shows that despite the small number of Anchor speakers in our data (3 79%), they nevertheless concentrate 35 71 % of the total speech time 
Results ::: Descriptive analysis of gender representation in training data ::: Role and gender interaction
When crossing both parameters, we can observe that the gender distribution is not constant throughout roles  Women represent 29 47% of the speakers within the Anchor category, even less than among the Punctual speakers  Their percentage of speech is also smaller  When calculating the average speech time uttered by a female Anchor, we obtain a value of 15 9 min against 25 2 min for a male Anchor, which suggests that even within the Anchor category men tend to speak more  This confirms the existence of gender disparities within French media  It corroborates with the analysis of the CSA BIBREF1, which shows that women were less present during high-audience hours  Our study shows that they are also less present in important roles  These results legitimate our initial questioning on the impact of gender balance on ASR performance trained on broadcast recordings 
Results ::: Performance (WER) analysis on evaluation data ::: Impact of gender on WER
As explained in Section SECREF13, WER is the sum of errors divided by the number of words in the transcription reference  The higher the WER, the poorer the system performance  Our 70h evaluation data contains a large amount of spontaneous speech and is very challenging for the ASR system trained on prepared speech: we observe an overall average WER of 42 9% for women and 34 3% for men  This difference of WER between men and women is statistically significant (med(M) = 25%; med(F) = 29%; U = 709040; p-value < 0 001) 
However, when observing gender differences across shows, no clear trend can be identified, as shown in Figure FIGREF21  For shows like Africa1 Infos or La Place du Village, we find an average WER lower for women than for men, while the trend is reversed for shows such as Un Temps de Pauchon or Le Masque et la Plume  The disparity of the results depending on the show leads us to believe that other factors may be entangled within the observed phenomenon 
Results ::: Performance (WER) analysis on evaluation data ::: Impact of role on WER
Speaker's role seems to have an impact on WER: we obtain an average WER of 30 8% for the Anchor speakers and 42 23% for the Punctual speakers  This difference is statistically significant with a p-value smaller than $10^{-14}$ (med(A) = 21%; med(P) = 31%; U = 540,430; p-value < 0 001)  
Results ::: Performance (WER) analysis on evaluation data ::: Role and gender interaction
Figure FIGREF25 presents the WER distribution (WER being obtained for each speaker in a show occurrence) according to the speaker's role and gender  It is worth noticing that the gender difference is only significant within the Punctual speakers group  The average WER is of 49 04% for the women and 38 56% for the men with a p-value smaller than $10^{-6}$ (med(F) = 39%; med(M) = 29%; U = 251,450; p-value < 0 001), whereas it is just a trend between male and female Anchors (med(F) = 21%; med(M) = 21%; U = 116,230; p-value = 0 173)  This could be explained by the quantity of data available per speaker 
Results ::: Performance (WER) analysis on evaluation data ::: Speech type as a third entangled factor?
In order to try to explain the observed variation in our results depending on shows and gender (Figure FIGREF21), we add the notion of speech type to shed some light on our results  BIBREF22 and BIBREF24 suggested that the speaker professionalism, associated with clear and hyper-articulated speech could be an explaining factor for better performance 
Based on our categorization in prepared speech (mostly news reports) and spontaneous speech (mostly debates and entertainment shows), we cross this parameter in our performance analysis  As shown on Figure FIGREF26, these results confirm the inherent challenge of spontaneous speech compared to prepared speech  WER scores are similar between men and women when considering prepared speech (med(F) = 18%; med(M) = 21%; U = 217,160; p-value = 0 005) whereas they are worse for women (61 29%) than for men (46 51%) with p-value smaller than $10^{-14}$ for the spontaneous speech type (med(F) = 61%; med(M) = 37%; U = 153,580; p-value < 0 001) 
Discussion
We find a clear disparity in terms of women presence and speech quantity in French media  Our data being recorded between 1998 and 2013, we can expect this disparity to be smaller on more recent broadcast recordings, especially since the French government displays efforts toward parity in media representation  One can also argue that even if our analysis was conducted on a large amount of data it does not reach the exhaustiveness of large-scale studies such as the one of BIBREF2  Nonetheless it does not affect the relevance of our findings, because if real-world gender representation might be more balanced today, these corpora are still used as training data for AI systems 
The performance difference across gender we observed corroborates (on a larger quantity and variety of language data produced by more than 2400 speakers) the results obtained by BIBREF24 on isolated words recognition  However the following study on read speech does not replicate these results  Yet a performance degradation is observed across dialect and race BIBREF8  BIBREF22 found lower WER for women than men on broadcast news and conversational telephone speech for both English and French  The authors suggest that gender stereotypes associated with women role in education and language acquisition induce a more normative elocution  We observed that the higher the degree of normativity of speech the smaller the gender difference  No significant gender bias is observed for prepared speech nor within the Anchor category  Even if we do not find similar results with lower WER for women than men, we obtained a median WER smaller for women on prepared speech and equal to the male median WER for the Anchor speakers 
Another explanation could be the use of adaptation within the pipeline  Most broadcast programs transcription systems have a speaker adaptation step within their decoding pipeline, which is the case for our system  An Anchor speaker intervening more often would have a larger quantity of data to realize such adaptation of the acoustic model  On the contrary, Punctual speakers who appear scarcely in the data are not provided with the same amount of adaptation data  Hence we can hypothesize that gender performance difference observed for Punctual speakers is due to the fact that female speech is further from the (initial non-adapted) acoustic model as it was trained on unbalanced data (as shown in Table )  Considering that Punctual speakers represent 92 78% of the speakers, this explains why gender difference is significant over our entire data set  A way to confirm our hypothesis would be to reproduce our analysis on WER values obtained without using speaker adapted features at the decoding step 
When decoding prepared speech (hence similar to the training data), no significant difference is found in WER between men and women, revealing that the speaker adaptation step could be sufficient to reach same performance for both genders  But when decoding more spontaneous speech, there is a mismatch with the initial acoustic model (trained on prepared speech)  Consequently, the speaker adaptation step might not be enough to recover good ASR performance, especially for women for whom less adaptation data is available (see Section 4 2 3) 
Conclusion
This paper has investigated gender bias in ASR performance through the following research questions: i) what is the proportion of men and women in French radio and TV media data ? ii) what is the impact of the observed disparity on ASR performance ? iii) is this as simple as a problem of gender proportion in the training data or are other factors entangled ? Our contributions are the following:
Descriptive analysis of the broadcast data used to train our ASR system confirms the already known disparity, where 65% of the speakers are men, speaking more than 75% of the time 
When investigating WER scores according to gender, speaker's role and speech type, huge variations are observed  We conclude that gender is clearly a factor of variation in ASR performance, with a WER increase of 24% for women compared to men, exhibiting a clear gender bias 
Gender bias varies across speaker's role and speech spontaneity level  Performance for Punctual speakers respectively spontaneous speech seems to reinforce this gender bias with a WER increase of 27 2% respectively 31 8% between male and female speakers 
We found that an ASR system trained on unbalanced data regarding gender produces gender bias performance  Therefore, in order to create fair systems it is necessary to take into account the representation problems in society that are going to be encapsulated in the data  Understanding how women under-representation in broadcast data can lead to bias in ASR performances is the key to prevent re-implementing and reinforcing discrimination already existing in our societies  This is in line with the concept of “Fairness by Design" proposed by BIBREF31 
Gender, race, religion, nationality are all characteristics that we deem unfair to classify on, and these ethical standpoints needs to be taken into account in systems' design  Characteristics that are not considered as relevant in a given task can be encapsulated in data nonetheless, and lead to bias performance  Being aware of the demographic skews our data set might contain is a first step to track the life cycle of a training data set and a necessary step to control the tools we develop 
Context 2: Introduction
Back to 42 BC, the philosopher Cicero has raised the issue that although there were many Oratory classes, there were none for Conversational skills BIBREF0   He highlighted how important they were not only for politics, but also for educational purpose  Among other conversational norms, he claimed that people should be able to know when to talk in a conversation, what to talk depending on the subject of the conversation, and that they should not talk about themselves 
Norms such as these may become social conventions and are not learnt at home or at school  Social conventions are dynamic and may change according to context, culture and language  In online communication, new commonsense practices are evolved faster and accepted as a norm BIBREF1 , BIBREF2   There is not a discipline for that on elementary or high schools and there are few linguistics researchers doing research on this field 
On the other hand, within the Artificial Intelligence area, some Conversational Systems have been created in the past decades since the test proposed by Alan Turing in 1950  The test consists of a machine's ability to exhibit intelligent behavior equivalent to, or indistinguishable from that of a human BIBREF3   Turing proposed that a human evaluator would judge natural language conversations between a human and a machine that is designed to generate human-like responses  Since then, many systems have been created to pass the Turing's test  Some of them have won prizes, some not BIBREF4   Although in this paper we do not focus on creating a solution that is able to build conversational systems that pass the Turing's test, we focus on NDS  From BIBREF5 , "NDS are systems that try to improve usability and user satisfaction by imitating human behavior"  We refer to Conversational Systems as NDS, where the dialogues are expressed as natural language texts, either from artificial intelligent agents (a k a  bots) or from humans 
That said, the current popular name to systems that have the ability to make a conversation with humans using natural language is Chatbot  Chatbots are typically used in conversational systems for various practical purposes, including customer service or information acquisition  Chatbots are becoming more widely used by social media software vendors  For example, Facebook recently announced that it would make Facebook Messenger (its 900-million-user messaging app by 2016), into a full-fledged platform that allows businesses to communicate with users via chatbots  Google is also building a new mobile-messaging service that uses artificial intelligence know-how and chatbot technology  In addition, according to the Wall Street Journal, there are more than 2 billion users of mobile apps  Still, people can be reluctant to install apps  So it is believed that social messaging can be a platform and chatbots may provide a new conversational interface for interacting with online services, as chatbots are easier to build and deploy than apps BIBREF6  
China seems to be the place where chatbots adoption and use is most advanced today  For example, China's popular WeChat messaging platform can take payments, scan QR codes, and integrate chatbot systems  WeChat integrates e-mail, chat, videocalls and sharing of large multimedia files  Users can book flights or hotels using a mixed, multimedia interaction with active bots  WeChat was first released in 2011 by Tecent, a Chinese online-gaming and social-media firm, and today more than 700 million people use it, being one of the most popular messaging apps in the world (The Economist 2016)  WeChat has a mixture of real-live customer service agents and automated replies (Olson 2016) 
Still, current existing chatbot engines do not properly handle a group chat with many users and many chatbots  This makes the chatbots considerably less social, which is a problem since there is a strong demand of having social chatbots that are able to provide different kinds of services, from traveling packages to finance advisors  This happens because there is a lack of methods and tools to design and engineer the coordination and mediation among chatbots and humans, as we present in Sections 2 and 3  In this paper, we refer to conversational systems that are able to interact with one or more people or chatbots in a multi-party chat as MPCS  Altogether, this paper is not meant to advance the state of the art on the norms for MPCS  Instead, the main contributions of this paper are threefold:
We then present some discussion and future work in the last section 
Challenges on Chattering
There are plenty of challenges in conversation contexts, and even bigger ones when people and machines participate in those contexts  Conversation is a specialized form of interaction, which follows social conventions  Social interaction makes it possible to inform, context, create, ratify, refute, and ascribe, among other things, power, class, gender, ethnicity, and culture BIBREF2   Social structures are the norms that emerge from the contact people have with others BIBREF7 , for example, the communicative norms of a negotiation, taking turns in a group, the cultural identity of a person, or power relationships in a work context 
Conventions, norms and patterns from everyday real conversations are applied when designing those systems to result in adoption and match user's expectations  BIBREF8 describes implicit interactions in a framework of interactions between humans and machines  The framework is based on the theory of implicit interactions which posits that people rely on conventions of interaction to communicate queries, offers, responses, and feedback to one another  Conventions and patterns drive our expectations about interactive behaviors  This framework helps designers and developers create interactions that are more socially appropriate  According to the author, we have interfaces which are based on explicit interaction and implicit ones  The explicit are the interactions or interfaces where people rely on explicit input and output, whereas implicit interactions are the ones that occur without user awareness of the computer behavior 
Social practices and actions are essential for a conversation to take place during the turn-by-turn moments of communication  BIBREF9 highlights that a distinguishing feature of ordinary conversation is "the local, moment-by-moment management of the distribution of turns, of their size, and what gets done in them, those things being accomplished in the course of each current speaker's turn " Management of turns and subject change in each course is a situation that occurs in real life conversations based on circumstances (internal and external) to speakers in a dialogue  Nowadays, machines are not prepared to fully understand context and change the course of conversations as humans  Managing dialogues with machines is challenging, which increases even more when more than one conversational agent is part of the same conversation  Some of those challenges in the dialogue flow were addressed by BIBREF10   According to them, we have system-initiative, user-initiative, and mixed-initiative systems 
In the first case, system-initiative systems restrict user options, asking direct questions, such as (Table TABREF5 ): "What is the initial amount of investment?" Doing so, those types of systems are more successful and easier to answer to  On the other hand, user-initiative systems are the ones where users have freedom to ask what they wish  In this context, users may feel uncertain of the capabilities of the system and starting asking questions or requesting information or services which might be quite far from the system domain and understanding capacity, leading to user frustration  There is also a mixed-initiative approach, that is, a goal-oriented dialogue which users and computers participate interactively using a conversational paradigm  Challenges of this last classification are to understand interruptions, human utterances, and unclear sentences that were not always goal-oriented 
The dialog in Table TABREF5 has the system initiative in a question and answer mode, while the one in Table TABREF7 is a natural dialogue system where both the user and the system take the initiative  If we add another user in the chat, then we face other challenges 
In Table TABREF12 , line 4, the user U1 invites another person to the chat and the system does not reply to this utterance, nor to utterances on lines 6, 7 and 8 which are the ones when only the users (wife and husband) should reply to  On the other hand, when the couple agrees on the period and initial value of the investment (line 9), then the system S1 (at the time the only system in the chat) replies indicating that it will invite more systems (chatbots) that are experts on this kind of pair INLINEFORM0 period, initial value INLINEFORM1   They then join the chat and start interacting with each other  At the end, on line 17, the user U2 interacts with U1 and they agree with the certificate option  Then, the chatbot responsible for that, S3, is the only one that replies indicating how to invest 
Table TABREF12 is one example of interactions on which the chatbots require knowledge of when to reply given the context of the dialog  In general, we acknowledge that exist four dimensions of understanding and replying to an utterance in MPCS which a chatbot that interacts in a multi-party chat group should fulfill:
In the next section we present the state of the art and how they fullfil some of these dimensions 
Conversational Systems
In this section we discuss the state of the art on conversational systems in three perspectives: types of interactions, types of architecture, and types of context reasoning  Then we present a table that consolidates and compares all of them 
ELIZA BIBREF11 was one of the first softwares created to understand natural language processing  Joseph Weizenbaum created it at the MIT in 1966 and it is well known for acting like a psychotherapist and it had only to reflect back onto patient's statements  ELIZA was created to tackle five "fundamental technical problems": the identification of critical words, the discovery of a minimal context, the choice of appropriate transformations, the generation of appropriate responses to the transformation or in the absence of critical words, and the provision of an ending capacity for ELIZA scripts 
Right after ELIZA came PARRY, developed by Kenneth Colby, who is psychiatrist at Stanford University in the early 1970s  The program was written using the MLISP language (meta-lisp) on the WAITS operating system running on a DEC PDP-10 and the code is non-portable  Parts of it were written in PDP-10 assembly code and others in MLISP  There may be other parts that require other language translators  PARRY was the first system to pass the Turing test - the psychiatrists were able to make the correct identification only 48 percent of the time, which is the same as a random guessing 
A L I C E  (Artificial Linguistic Internet Computer Entity) BIBREF12 appeared in 1995 but current version utilizes AIML, an XML language designed for creating stimulus-response chat robots BIBREF13   A L I C E  bot has, at present, more than 40,000 categories of knowledge, whereas the original ELIZA had only about 200  The program is unable to pass the Turing test, as even the casual user will often expose its mechanistic aspects in short conversations 
Cleverbot (1997-2014) is a chatbot developed by the British AI scientist Rollo Carpenter  It passed the 2011 Turing Test at the Technique Techno-Management Festival held by the Indian Institute of Technology Guwahati  Volunteers participate in four-minute typed conversations with either Cleverbot or humans, with Cleverbot voted 59 3 per cent human, while the humans themselves were rated just 63 3 per cent human BIBREF14  
Types of Interactions
Although most part of the research literature focuses on the dialogue of two persons, the reality of everyday life interactions shows a substantial part of multi-user conversations, such as in meetings, classes, family dinners, chats in bars and restaurants, and in almost every collaborative or competitive environment such as hospitals, schools, offices, sports teams, etc  The ability of human beings to organize, manage, and (mostly) make productive such complex interactive structures which are multi-user conversations is nothing less than remarkable  The advent of social media platforms and messaging systems such as WhatsApp in the first 15 years of the 21st century expanded our ability as a society to have asynchronous conversations in text form, from family and friends chatgroups to whole nations conversing in a highly distributed form in social media BIBREF15  
In this context, many technological advances in the early 2010s in natural language processing (spearheaded by the IBM Watson's victory in Jeopardy BIBREF16 ) spurred the availability in the early 2010s of text-based chatbots in websites and apps (notably in China BIBREF17 ) and spoken speech interfaces such as Siri by Apple, Cortana by Microsoft, Alexa by Amazon, and Allo by Google  However, the absolute majority of those chatbot deployments were in contexts of dyadic dialog, that is, a conversation between a single chatbot with a single user  Most of the first toolkits for chatbot design and development of this initial period implicit assume that an utterance from the user is followed by an utterance of the chatbot, which greatly simplifies the management of the conversation as discussed in more details later  Therefore, from the interaction point of view, there are two types: 1) one in which the chatbot was designed to chat with one person or chatbot, and 2) other in which the chatbot can interact with more than two members in the chat 
Dyadic Chatbot
A Dyadic Chatbot is a chatbot that does know when to talk  If it receives an utterance, it will always handle and try to reply to the received utterance  For this chatbot to behave properly, either there are only two members in the chat, and the chatbot is one of them, or there are more, but the chatbot replies only when its name or nickname is mentioned  This means that a dyadic chatbot does not know how to coordinate with many members in a chat group  It lacks the social ability of knowing when it is more suitable to answer or not  Also, note that we are not considering here the ones that would use this social ability as an advantage in the conversation, because if the chatbot is doing with this intention, it means that the chatbot was designed to be aware of the social issues regarding a chat with multiple members, which is not the case of a dyadic chatbot  Most existing chatbots, from the first system, ELIZA BIBREF11 , until modern state-of-the-art ones fall into this category 
Multiparty Conversations
In multiparty conversations between people and computer systems, natural language becomes the communication protocol exchanged not only by the human users, but also among the bots themselves  When every actor, computer or user, understands human language and is able to engage effectively in a conversation, a new, universal computer protocol of communication is feasible, and one for which people are extremely good at 
There are many differences between dyadic and multiparty conversations, but chiefly among them is turn-taking, that is, how a participant determines when it is appropriate to make an utterance and how that is accomplished  There are many social settings, such as assemblies, debates, one-channel radio communications, and some formal meetings, where there are clear and explicit norms of who, when, and for long a participant can speak 
The state of the art for the creation of chatbots that can participate on multiparty conversations currently is a combination of the research on the creation of chatbots and research on the coordination or governance of multi-agents systems  A definition that mixes both concepts herein present is: A chatbot is an agent that interacts through natural language  Although these areas complement each other, there is a lack of solutions for creating multiparty-aware chatbots or governed chatbots, which can lead to higher degree of system trust 
Multi-Dyadic Chatbots
Turn-taking in generic, multiparty spoken conversation has been studied by, for example, Sacks et al  BIBREF18   In broad terms, it was found that participants in general do not overlap their utterances and that the structure of the language and the norms of conversation create specific moments, called transition-relevance places, where turns can occur  In many cases, the last utterances make clear to the participants who should be the next speaker (selected-next-speaker), and he or she can take that moment to start to talk  Otherwise, any other participant can start speaking, with preference for the first starter to get the turn; or the current speaker can continue BIBREF18  
A key part of the challenge is to determine whether the context of the conversation so far have or have not determined the next speaker  In its simplest form, a vocative such as the name of the next speaker is uttered  Also, there is a strong bias towards the speaker before the current being the most likely candidate to be the next speaker 
In general the detection of transition-relevance places and of the selected-next-speaker is still a challenge for speech-based machine conversational systems  However, in the case of text message chats, transition-relevance places are often determined by the acting of posting a message, so the main problem facing multiparty-enabled textual chatbots is in fact determining whether there is and who is the selected-next-speaker  In other words, chatbots have to know when to shut up  Bohus and Horowitz BIBREF19 have proposed a computational probabilistic model for speech-based systems, but we are not aware of any work dealing with modeling turn-taking in textual chats 
Coordination of Multi-Agent Systems
A multi-agent system (MAS) can be defined as a computational environment in which individual software agents interact with each other, in a cooperative manner, or in a competitive manner, and sometimes autonomously pursuing their individual goals  During this process, they access the environment's resources and services and occasionally produce results for the entities that initiated these software agents  As the agents interact in a concurrent, asynchronous and decentralized manner, this kind of system can be categorized as a complex system BIBREF20  
Research in the coordination of multi-agent systems area does not address coordination using natural dialogue, as usually all messages are structured and formalized so the agents can reason and coordinate themselves  On the other hand, chatbots coordination have some relations with general coordination mechanisms of multi-agent systems in that they specify and control interactions between agents  However, chatbots coordination mechanisms is meant to regulate interactions and actions from a social perspective, whereas general coordination languages and mechanisms focus on means for expressing synchronization and coordination of activities and exchange of information, at a lower computational level 
In open multi-agent systems the development takes place without a centralized control, thus it is necessary to ensure the reliability of these systems in a way that all the interactions between agents will occur according to the specification and that these agents will obey the specified scenario  For this, these applications must be built upon a law-governed architecture 
Minsky published the first ideas about laws in 1987 BIBREF21   Considering that a law is a set of norms that govern the interaction, afterwards, he published a seminal paper with the Law-Governed Interaction (LGI) conceptual model about the role of interaction laws on distributed systems BIBREF22   Since then, he conducted further work and experimentation based on those ideas BIBREF23   Although at the low level a multiparty conversation system is a distributed system and the LGI conceptual model can be used in a variety of application domains, it is composed of abstractions basically related to low level information about communication issues of distributed systems (like the primitives disconnected, reconnected, forward, and sending or receiving of messages), lacking the ability to express high level information of social systems 
Following the same approach, the Electronic Institution (EI) BIBREF24 solution also provides support for interaction norms  An EI has a set of high-level abstractions that allow for the specification of laws using concepts such as agent roles, norms and scenes 
Still at the agent level but more at the social level, the XMLaw description language and the M-Law framework BIBREF25 BIBREF26 were proposed and developed to support law-governed mechanism  They implement a law enforcement approach as an object-oriented framework and it allows normative behavior through the combination between norms and clocks  The M-Law framework BIBREF26 works by intercepting messages exchanged between agents, verifying the compliance of the messages with the laws and subsequently redirecting the message to the real addressee, if the laws allow it  If the message is not compliant, then the mediator blocks the message and applies the consequences specified in the law, if any  They are called laws in the sense that they enforce the norms, which represent what can be done (permissions), what cannot be done (prohibitions) and what must be done (obligations) 
Coordinated Aware Chatbots in a Multiparty Conversation
With regard to chatbot engines, there is a lack of research directed to building coordination laws integrated with natural language  To the best of our knowledge, the architecture proposed in this paper is the first one in the state of the art designed to support the design and development of coordinated aware chatbots in a multiparty conversation 
Types of Architectures
There are mainly three types of architectures when building conversational systems: totally rule-oriented, totally data-oriented, and a mix of rules and data-oriented 
Rule-oriented
A rule-oriented architecture provides a manually coded reply for each recognized utterance  Classical examples of rule-based chatbots include Eliza and Parry  Eliza could also extract some words from sentences and then create another sentence with these words based on their syntatic functions  It was a rule-based solution with no reasoning  Eliza could not "understand" what she was parsing  More sophisticated rule-oriented architectures contain grammars and mappings for converting sentences to appropriate sentences using some sort of knowledge  They can be implemented with propositional logic or first-order logic (FOL)  Propositional logic assumes the world contains facts (which refer to events, phenomena, symptoms or activities)  Usually, a set of facts (statements) is not sufficient to describe a domain in a complete manner  On the other hand, FOL assumes the world contains Objects (e g , people, houses, numbers, etc ), Relations (e g  red, prime, brother of, part of, comes between, etc ), and Functions (e g  father of, best friend, etc ), not only facts as in propositional logic  Moreover, FOL contains predicates, quantifiers and variables, which range over individuals (which are domain of discourse) 
Prolog (from French: Programmation en Logique) was one of the first logic programming languages (created in the 1970s), and it is one of the most important languages for expressing phrases, rules and facts  A Prolog program consists of logical formulas and running a program means proving a theorem  Knowledge bases, which include rules in addition to facts, are the basis for most rule-oriented chatbots created so far 
In general, a rule is presented as follows: DISPLAYFORM0
Prolog made it possible to perform the language of Horn clauses (implications with only one conclusion)  The concept of Prolog is based on predicate logic, and proving theorems involves a resolute system of denials  Prolog can be distinguished from classic programming languages due to its possibility of interpreting the code in both a procedural and declarative way  Although Prolog is a set of specifications in FOL, it adopts the closed-world assumption, i e  all knowledge of the world is present in the database  If a term is not in the database, Prolog assumes it is false 
In case of Prolog, the FOL-based set of specifications (formulas) together with the facts compose the knowledge base to be used by a rule-oriented chatbot  However an Ontology could be used  For instance, OntBot BIBREF27 uses mapping technique to transform ontologies and knowledge into relational database and then use that knowledge to drive its chats  One of the main issues currently facing such a huge amount of ontologies stored in a database is the lack of easy to use interfaces for data retrieval, due to the need to use special query languages or applications 
In rule-oriented chatbots, the degree of intelligent behavior depends on the knowledge base size and quality (which represents the information that the chatbot knows), poor ones lead to weak chatbot responses while good ones do the opposite  However, good knowledge bases may require years to be created, depending on the domain 
Data-oriented
As opposed to rule-oriented architectures, where rules have to be explicitly defined, data-oriented architectures are based on learning models from samples of dialogues, in order to reproduce the behavior of the interaction that are observed in the data  Such kind of learning can be done by means of machine learning approach, or by simply extracting rules from data instead of manually coding them 
Among the different technologies on which these system can be based, we can highlight classical information retrieval algorithms, neural networks BIBREF28 , Hidden Markov Models (HMM) BIBREF29 , and Partially Observable Markov Decision Process (POMDP) BIBREF30   Examples include Cleverbot and Tay BIBREF31   Tay was a chatbot developed by Microsoft that after one day live learning from interaction with teenagers on Twitter, started replying impolite utterances  Microsoft has developed others similar chatbots in China (Xiaoice) and in Japan (Rinna)  Microsoft has not associated its publications with these chatbots, but they have published a data-oriented approach BIBREF32 that proposes a unified multi-turn multi-task spoken language understanding (SLU) solution capable of handling multiple context sensitive classification (intent determination) and sequence labeling (slot filling) tasks simultaneously  The proposed architecture is based on recurrent convolutional neural networks (RCNN) with shared feature layers and globally normalized sequence modeling components 
A survey of public available corpora for can be found in BIBREF33   A corpus can be classified into different categories, according to: the type of data, whether it is spoken dialogues, transcripts of spoken dialogues, or directly written; the type of interaction, if it is human-human or human-machine; and the domain, whether it is restricted or unconstrained  Two well-known corpora are the Switchboard dataset, which consists of transcripts of spoken, unconstrained, dialogues, and the set of tasks for the Dialog State Tracking Challenge (DSTC), which contain more constrained tasks, for instance the restaurant and travel information sets 
Rule and Data-oriented
The model of learning in current A L I C E  BIBREF13 is incremental or/and interactive learning because a person monitors the robot's conversations and creates new AIML content to make the responses more appropriate, accurate, believable, "human", or whatever he/she intends  There are algorithms for automatic detection of patterns in the dialogue data and this process provides the person with new input patterns that do not have specific replies yet, permitting a process of almost continuous supervised refinement of the bot 
As already mentioned, A L I C E  consists of roughly 41,000 elements called categories which is the basic unit of knowledge in AIML  Each category consists of an input question, an output answer, and an optional context  The question, or stimulus, is called the pattern  The answer, or response, is called the template  The two types of optional context are called that and topic  The keyword that refers to the robot's previous utterance  The AIML pattern language consists only of words, spaces, and the wildcard symbols "_" and "*"  The words may consist only of letters and numerals  The pattern language is case invariant  Words are separated by a single space, and the wildcard characters function like words, similar to the initial pattern matching strategy of the Eliza system  More generally, AIML tags transform the reply into a mini computer program which can save data, activate other programs, give conditional responses, and recursively call the pattern matcher to insert the responses from other categories  Most AIML tags in fact belong to this template side sublanguage BIBREF13  
AIML language allows:
Symbolic reduction: Reduce complex grammatical forms to simpler ones 
Divide and conquer: Split an input into two or more subparts, and combine the responses to each 
Synonyms: Map different ways of saying the same thing to the same reply 
Spelling or grammar corrections: the bot both corrects the client input and acts as a language tutor 
Detecting keywords anywhere in the input that act like triggers for a reply 
Conditionals: Certain forms of branching to produce a reply 
Any combination of (1)-(6) 
When the bot chats with multiple clients, the predicates are stored relative to each client ID  For example, the markup INLINEFORM0 set name INLINEFORM1 "name" INLINEFORM2 Matthew INLINEFORM3 /set INLINEFORM4 stores the string Matthew under the predicate named "name"  Subsequent activations of INLINEFORM5 get name="name" INLINEFORM6 return "Matthew"  In addition, one of the simple tricks that makes ELIZA and A L I C E  so believable is a pronoun swapping substitution  For instance:
U: My husband would like to invest with me 
S: Who else in your family would like to invest with you?
Types of Intentions
According to the types of intentions, conversational systems can be classified into two categories: a) goal-driven or task oriented, and b) non-goal-driven or end-to-end systems 
In a goal-driven system, the main objective is to interact with the user so that back-end tasks, which are application specific, are executed by a supporting system  As an example of application we can cite technical support systems, for instance air ticket booking systems, where the conversation system must interact with the user until all the required information is known, such as origin, destination, departure date and return date, and the supporting system must book the ticket  The most widely used approaches for developing these systems are Partially-observed Decision Processes (POMDP) BIBREF30 , Hidden Markov Models (HMM) BIBREF29 , and more recently, Memory Networks BIBREF28   Given that these approaches are data-oriented, a major issue is to collect a large corpora of annotated task-specific dialogs  For this reason, it is not trivial to transfer the knowledge from one to domain to another  In addition, it might be difficult to scale up to larger sets of tasks 
Non-goal-driven systems (also sometimes called reactive systems), on the other hand, generate utterances in accordance to user input, e g  language learning tools or computer games characters  These systems have become more popular in recent years, mainly owning to the increase of popularity of Neural Networks, which is also a data-oriented approach  The most recent state of the art to develop such systems have employed Recurrent Neural Networs (RNN) BIBREF34 , Dynamic Context-Sensitive Generation BIBREF35 , and Memory Networks BIBREF36 , just to name a few  Nevertheless, probabilistic methods such as Hidden Topic Markov Models (HTMM) BIBREF37 have also been evaluated  Goal-driven approach can create both pro-active and reactive chatbots, while non-goal-driven approach creates reactive chatbots  In addition, they can serve as a tool to goal-driven systems as in BIBREF28   That is, when trained on corpora of a goal-driven system, non-goal-driven systems can be used to simulate user interaction to then train goal-driven models 
Types of Context Reasoning
A dialogue system may support the context reasoning or not  Context reasoning is necessary in many occasions  For instance, when partial information is provided the chatbot needs to be able to interact one or more turns in order to get the complete information in order to be able to properly answer  In BIBREF38 , the authors present a taxonomy of errors in conversational systems  The ones regarding context-level errors are the ones that are perceived as the top-10 confusing and they are mainly divided into the following:
Excess/lack of proposition: the utterance does not provide any new proposition to the discourse context or provides excessive information than required 
Contradiction: the utterance contains propositions that contradict what has been said by the system or by the user 
Non-relevant topic: the topic of the utterance is irrelevant to the current context such as when the system suddenly jumps to some other topic triggered by some particular word in the previous user utterance 
Unclear relation: although the utterance might relate to the previous user utterance, its relation to the current topic is unclear 
Topic switch error: the utterance displays the fact that the system missed the switch in topic by the user, continuing with the previous topic 
Rule-oriented
In the state of the art most of the proposed approaches for context reasoning lies on rules using logics and knowledge bases as described in the Rule-oriented architecture sub-section  Given a set of facts extracted from the dialogue history and encoded in, for instance, FOL statements, a queries can be posed to the inference engine and produce answers  For instance, see the example in Table TABREF37   The sentences were extracted from BIBREF36 (which does not use a rule-oriented approach), and the first five statements are their respective facts  The system then apply context reasoning for the query Q: Where is the apple 
If statements above are received on the order present in Table TABREF37 , if the query Q: Where is the apple is sent, the inference engine will produce the answer A: Bedroom (i e , the statement INLINEFORM0 is found by the model and returned as True) 
Nowadays, the most common way to store knowledge bases is on triple stores, or RDF (Resource Description Framework) stores  A triple store is a knowledge base for the storage and retrieval of triples through semantic queries  A triple is a data entity composed of subject-predicate-object, like "Sam is at the kitchen" or "The apple is with Sam", for instance  A query language is needed for storing and retrieving data from a triple store  While SPARQL is a RDF query language, Rya is an open source scalable RDF triple store built on top of Apache Accumulo  Originally developed by the Laboratory for Telecommunication Sciences and US Naval Academy, Rya is currently being used by a number of american government agencies for storing, inferencing, and querying large amounts of RDF data 
A SPARQL query has a SQL-like syntax for finding triples matching specific patterns  For instance, see the query below  It retrieves all the people that works at IBM and lives in New York:
SELECT ?people
WHERE {
?people <worksAt> <IBM>  
?people <livesIn> <New York> 
}
Since triple stores can become huge, Rya provides three triple table index BIBREF39 to help speeding up queries:
SPO: subject, predicate, object
POS: predicate, object, subject
OSP: object, subject, predicate
While Rya is an example of an optimized triple store, a rule-oriented chatbot can make use of Rya or any triple store and can call the semantic search engine in order to inference and generate proper answers 
Data-oriented
Recent papers have used neural networks to predict the next utterance on non-goal-driven systems considering the context, for instance with Memory Networks BIBREF40   In this work BIBREF36 , for example the authors were able to generate answers for dialogue like below:
Sam walks into the kitchen 
Sam picks up an apple 
Sam walks into the bedroom 
Sam drops the apple 
Q: Where is the apple?
A: Bedroom
Sukhbaatar's model represents the sentence as a vector in a way that the order of the words matter, and the model encodes the temporal context enhancing the memory vector with a matrix that contains the temporal information  During the execution phase, Sukhbaatar's model takes a discrete set of inputs INLINEFORM0 that are to be stored in the memory, a query INLINEFORM1 , and outputs an answer INLINEFORM2   Each of the INLINEFORM3 , INLINEFORM4 , and INLINEFORM5 contains symbols coming from a dictionary with INLINEFORM6 words  The model writes all INLINEFORM7 to the memory up to a fixed buffer size, and then finds a continuous representation for the INLINEFORM8 and INLINEFORM9   The continuous representation is then processed via multiple computational steps to output INLINEFORM10   This allows back propagation of the error signal through multiple memory accesses back to the input during training  Sukhbaatar's also presents the state of the art of recent efforts that have explored ways to capture dialogue context, treated as long-term structure within sequences, using RNNs or LSTM-based models  The problem of this approach is that it is has not been tested for goal-oriented systems  In addition, it works with a set of sentences but not necessary from multi-party bots 
Platforms
Regarding current platforms to support the development of conversational systems, we can categorize them into three types: platforms for plugging chatbots, for creating chatbots and for creating service chatbots  The platforms for plugging chatbots provide tools for integrating them another system, like Slack  The chatbots need to receive and send messages in a specific way, which depends on the API and there is no support for actually helping on building chatbots behavior with natural language understanding  The platforms for creating chatbots mainly provide tools for adding and training intentions together with dialogue flow specification and some entities extraction, with no reasoning support  Once the models are trained and the dialogue flow specified, the chatbots are able to reply to the received intention  The platforms for creating service chatbots provide the same functionalities as the last one and also provide support for defining actions to be executed by the chatbots when they are answering to an utterance  Table TABREF43 summarizes current platforms on the market accordingly to these categories  There is a lack on platforms that allow to create chatbots that can be coordinated in a multiparty chat with governance or mediation 
A Conceptual Architecture for Multiparty-Aware Chatbots
In this section the conceptual architecture for creating a hybrid rule and machine learning-based MPCS is presented  The MPCS is defined by the the entities and relationships illustrated in Fig  FIGREF44 which represents the chatbot's knowledge  A Chat Group contains several Members that join the group with a Role  The role may constrain the behavior of the member in the group  Chatbot is a type of Role, to differentiate from persons that may also join with different roles  For instance, a person may assume the role of the owner of the group, or someone that was invited by the owner, or a domain role like an expert, teacher or other 
When a Member joins the Chat Group, it/he/she can send Utterances  The Member then classifies each Utterance with an Intent which has a Speech Act  The Intent class, Speech Act class and the Intent Flow trigger the Action class to be executed by the Member that is a Chatbot  The Chatbots associated to the Intention are the only ones that know how to answer to it by executing Actions  The Action, which implements one Speech Act, produces answers which are Utterances, so, for instance, the Get_News action produces an Utterance for which Intention's speech act is Inform_News  The Intent Flow holds the intent's class conversation graph which maps the dialog state as a decision tree  The answer's intention class is mapped in the Intent Flow as a directed graph G defined as following: DISPLAYFORM0
From the graph definitions, INLINEFORM0 is for vertices and INLINEFORM1 is for relations, which are the arrows in the graph  And in Equation EQREF46 :
INLINEFORM0 is the set of intentions pairs,
INLINEFORM0 is the set of paths to navigate through the intentions,
INLINEFORM0 is the arrow's head, and
INLINEFORM0 is the arrow's tail 
This arrow represents a turn from an utterance with INLINEFORM0 intention class which is replying to an utterance with INLINEFORM1 intention class to the state which an utterance with INLINEFORM2 intention's class is sent 
INLINEFORM0 is the intention class of the answer to be provided to the received INLINEFORM1 intention class 
In addition, each intent's class may refer to many Entities which, in turn, may be associated to several Features  For instance, the utterance
"I would like to invest USD10,000 in Savings Account for 2 years"
contains one entity – the Savings Account's investment option – and two features – money (USD10,000) and period of time (2 years)  The Intent Flow may need this information to choose the next node which will give the next answer  Therefore, if the example is changed a little, like
"I would like to invest in Savings Account",
INLINEFORM0 is constrained by the "Savings Account" entity which requires the two aforementioned features  Hence, a possible answer by one Member of the group would be
"Sure, I can simulate for you, what would be the initial amount and the period of time of the investment?"
With these conceptual model's elements, a MPCS system can be built with multiple chatbots  Next subsection further describes the components workflow 
Workflow
Figure FIGREF48 illustrates from the moment that an utterance is sent in a chat group to the moment a reply is generated in the same chat group, if the case  One or more person may be in the chat, while one or more chatbots too  There is a Hub that is responsible for broadcasting the messages to every Member in the group, if the case  The flow starts when a Member sends the utterance which goes to the Hub and, if allowed, is broadcasted  Many or none interactions norms can be enforced at this level depending on the application  Herein, a norm can be a prohibition, obligation or permission to send an utterance in the chat group 
Once the utterance is broadcasted, a chatbot needs to handle the utterance  In order to properly handle it, the chatbot parses the utterance with several parsers in the Parsing phase: a Topic Classifier, the Dependency Parsing, which includes Part-of-Speech tags and semantics tags, and any other that can extract metadata from the utterance useful for the reasoning  All these metadata, together with more criteria, may be used in the Frame parsing which is useful for context reasoning  All knowledge generated in this phase can be stored in the Context  Then, the Intent Classifier tries to detect the intent class of the utterance  If detected, the Speech Act is also retrieved  And an Event Detector can also check if there is any dialog inconsistency during this phase 
After that, the Filtering phase receives the object containing the utterance, the detected intent, and all metadata extracted so far and decides if an action should be performed to reply to the utterance  If yes, it is sent to the Acting phase which performs several steps  First the Action Classifier tries to detect the action to be performed
Context 3: Introduction
Nowadays deep learning techniques outperform the other conventional methods in most of the speech-related tasks  Training robust deep neural networks for each task depends on the availability of powerful processing GPUs, as well as standard and large scale datasets  In text-independent speaker verification, large-scale datasets are available, thanks to the NIST SRE evaluations and other data collection projects such as VoxCeleb BIBREF0 
In text-dependent speaker recognition, experiments with end-to-end architectures conducted on large proprietary databases have demonstrated their superiority over traditional approaches BIBREF1  Yet, contrary to text-independent speaker recognition, text-dependent speaker recognition lacks large-scale publicly available databases  The two most well-known datasets are probably RSR2015 BIBREF2 and RedDots BIBREF3  The former contains speech data collected from 300 individuals in a controlled manner, while the latter is used primarily for evaluation rather than training, due to its small number of speakers (only 64)  Motivated by this lack of large-scale dataset for text-dependent speaker verification, we chose to proceed with the collection of the DeepMine dataset, which we expect to become a standard benchmark for the task 
Apart from speaker recognition, large amounts of training data are required also for training automatic speech recognition (ASR) systems  Such datasets should not only be large in size, they should also be characterized by high variability with respect to speakers, age and dialects  While several datasets with these properties are available for languages like English, Mandarin, French, this is not the case for several other languages, such as Persian  To this end, we proceeded with collecting a large-scale dataset, suitable for building robust ASR models in Persian 
The main goal of the DeepMine project was to collect speech from at least a few thousand speakers, enabling research and development of deep learning methods  The project started at the beginning of 2017, and after designing the database and the developing Android and server applications, the data collection began in the middle of 2017  The project finished at the end of 2018 and the cleaned-up and final version of the database was released at the beginning of 2019  In BIBREF4, the running project and its data collection scenarios were described, alongside with some preliminary results and statistics  In this paper, we announce the final and cleaned-up version of the database, describe its different parts and provide various evaluation setups for each part  Finally, since the database was designed mainly for text-dependent speaker verification purposes, some baseline results are reported for this task on the official evaluation setups  Additional baseline results are also reported for Persian speech recognition  However, due to the space limitation in this paper, the baseline results are not reported for all the database parts and conditions  They will be defined and reported in the database technical documentation and in a future journal paper 
Data Collection
DeepMine is publicly available for everybody with a variety of licenses for different users  It was collected using crowdsourcing BIBREF4  The data collection was done using an Android application  Each respondent installed the application on his/her personal device and recorded several phrases in different sessions  The Android application did various checks on each utterance and if it passed all of them, the respondent was directed to the next phrase  For more information about data collection scenario, please refer to BIBREF4 
Data Collection ::: Post-Processing
In order to clean-up the database, the main post-processing step was to filter out problematic utterances  Possible problems include speaker word insertions (e g  repeating some part of a phrase), deletions, substitutions, and involuntary disfluencies  To detect these, we implemented an alignment stage, similar to the second alignment stage in the LibriSpeech project BIBREF5  In this method, a custom decoding graph was generated for each phrase  The decoding graph allows for word skipping and word insertion in the phrase 
For text-dependent and text-prompted parts of the database, such errors are not allowed  Hence, any utterances with errors were removed from the enrollment and test lists  For the speech recognition part, a sub-part of the utterance which is correctly aligned to the corresponding transcription is kept  After the cleaning step, around 190 thousand utterances with full transcription and 10 thousand with sub-part alignment have remained in the database 
Data Collection ::: Statistics
After processing the database and removing problematic respondents and utterances, 1969 respondents remained in the database, with 1149 of them being male and 820 female  297 of the respondents could not read English and have therefore read only the Persian prompts  About 13200 sessions were recorded by females and similarly, about 9500 sessions by males, i e  women are over-represented in terms of sessions, even though their number is 17% smaller than that of males  Other useful statistics related to the database are shown in Table TABREF4 
The last status of the database, as well as other related and useful information about its availability can be found on its website, together with a limited number of samples 
DeepMine Database Parts
The DeepMine database consists of three parts  The first one contains fixed common phrases to perform text-dependent speaker verification  The second part consists of random sequences of words useful for text-prompted speaker verification, and the last part includes phrases with word- and phoneme-level transcription, useful for text-independent speaker verification using a random phrase (similar to Part4 of RedDots)  This part can also serve for Persian ASR training  Each part is described in more details below  Table TABREF11 shows the number of unique phrases in each part of the database  For the English text-dependent part, the following phrases were selected from part1 of the RedDots database, hence the RedDots can be used as an additional training set for this part:
“My voice is my password ”
“OK Google ”
“Artificial intelligence is for real ”
“Actions speak louder than words ”
“There is no such thing as a free lunch ”
DeepMine Database Parts ::: Part1 - Text-dependent (TD)
This part contains a set of fixed phrases which are used to verify speakers in text-dependent mode  Each speaker utters 5 Persian phrases, and if the speaker can read English, 5 phrases selected from Part1 of the RedDots database are also recorded 
We have created three experimental setups with different numbers of speakers in the evaluation set  For each setup, speakers with more recording sessions are included in the evaluation set and the rest of the speakers are used for training in the background set (in the database, all background sets are basically training data)  The rows in Table TABREF13 corresponds to the different experimental setups and shows the numbers of speakers in each set  Note that, for English, we have filtered the (Persian native) speakers by the ability to read English  Therefore, there are fewer speakers in each set for English than for Persian  There is a small “dev” set in each setup which can be used for parameter tuning to prevent over-tuning on the evaluation set 
For each experimental setup, we have defined several official trial lists with different numbers of enrollment utterances per trial in order to investigate the effects of having different amounts of enrollment data  All trials in one trial list have the same number of enrollment utterances (3 to 6) and only one test utterance  All enrollment utterances in a trial are taken from different consecutive sessions and the test utterance is taken from yet another session  From all the setups and conditions, the 100-spk with 3-session enrollment (3-sess) is considered as the main evaluation condition  In Table TABREF14, the number of trials for Persian 3-sess are shown for the different types of trial in the text-dependent speaker verification (SV)  Note that for Imposter-Wrong (IW) trials (i e  imposter speaker pronouncing wrong phrase), we merely create one wrong trial for each Imposter-Correct (IC) trial to limit the huge number of possible trials for this case  So, the number of trials for IC and IW cases are the same 
DeepMine Database Parts ::: Part2 - Text-prompted (TP)
For this part, in each session, 3 random sequences of Persian month names are shown to the respondent in two modes: In the first mode, the sequence consists of all 12 months, which will be used for speaker enrollment  The second mode contains a sequence of 3 month names that will be used as a test utterance  In each 8 sessions received by a respondent from the server, there are 3 enrollment phrases of all 12 months (all in just one session), and $7 \times 3$ other test phrases, containing fewer words  For a respondent who can read English, 3 random sequences of English digits are also recorded in each session  In one of the sessions, these sequences contain all digits and the remaining ones contain only 4 digits 
Similar to the text-dependent case, three experimental setups with different number of speaker in the evaluation set are defined (corresponding to the rows in Table TABREF16)  However, different strategy is used for defining trials: Depending on the enrollment condition (1- to 3-sess), trials are enrolled on utterances of all words from 1 to 3 different sessions (i e  3 to 9 utterances)  Further, we consider two conditions for test utterances: seq test utterance with only 3 or 4 words and full test utterances with all words (i e  same words as in enrollment but in different order)  From all setups an all conditions, the 100-spk with 1-session enrolment (1-sess) is considered as the main evaluation condition for the text-prompted case  In Table TABREF16, the numbers of trials (sum for both seq and full conditions) for Persian 1-sess are shown for the different types of trials in the text-prompted SV  Again, we just create one IW trial for each IC trial 
DeepMine Database Parts ::: Part3 - Text-independent (TI)
In this part, 8 Persian phrases that have already been transcribed on the phone level are displayed to the respondent  These phrases are chosen mostly from news and Persian Wikipedia  If the respondent is unable to read English, instead of 5 fixed phrases and 3 random digit strings, 8 other Persian phrases are also prompted to the respondent to have exactly 24 phrases in each recording session 
This part can be useful at least for three potential applications  First, it can be used for text-independent speaker verification  The second application of this part (same as Part4 of RedDots) is text-prompted speaker verification using random text (instead of a random sequence of words)  Finally, the third application is large vocabulary speech recognition in Persian (explained in the next sub-section) 
Based on the recording sessions, we created two experimental setups for speaker verification  In the first one, respondents with at least 17 recording sessions are included to the evaluation set, respondents with 16 sessions to the development and the rest of respondents to the background set (can be used as training data)  In the second setup, respondents with at least 8 sessions are included to the evaluation set, respondents with 6 or 7 sessions to the development and the rest of respondents to the background set  Table TABREF18 shows numbers of speakers in each set of the database for text-independent SV case 
For text-independent SV, we have considered 4 scenarios for enrollment and 4 scenarios for test  The speaker can be enrolled using utterances from 1, 2 or 3 consecutive sessions (1sess to 3sess) or using 8 utterances from 8 different sessions  The test speech can be one utterance (1utt) for short duration scenario or all utterances in one session (1sess) for long duration case  In addition, test speech can be selected from 5 English phrases for cross-language testing (enrollment using Persian utterances and test using English utterances)  From all setups, 1sess-1utt and 1sess-1sess for 438-spk set are considered as the main evaluation setups for text-independent case  Table TABREF19 shows number of trials for these setups 
For text-prompted SV with random text, the same setup as text-independent case together with corresponding utterance transcriptions can be used 
DeepMine Database Parts ::: Part3 - Speech Recognition
As explained before, Part3 of the DeepMine database can be used for Persian read speech recognition  There are only a few databases for speech recognition in Persian BIBREF6, BIBREF7  Hence, this part can at least partly address this problem and enable robust speech recognition applications in Persian  Additionally, it can be used for speaker recognition applications, such as training deep neural networks (DNNs) for extracting bottleneck features BIBREF8, or for collecting sufficient statistics using DNNs for i-vector training 
We have randomly selected 50 speakers (25 for each gender) from the all speakers in the database which have net speech (without silence parts) between 25 minutes to 50 minutes as test speakers  For each speaker, the utterances in the first 5 sessions are included to (small) test-set and the other utterances of test speakers are considered as a large-test-set  The remaining utterances of the other speakers are included in the training set  The test-set, large-test-set and train-set contain 5 9, 28 5 and 450 hours of speech respectively 
There are about 8300 utterances in Part3 which contain only Persian full names (i e  first and family name pairs)  Each phrase consists of several full names and their phoneme transcriptions were extracted automatically using a trained Grapheme-to-Phoneme (G2P)  These utterances can be used to evaluate the performance of a systems for name recognition, which is usually more difficult than the normal speech recognition because of the lack of a reliable language model 
Experiments and Results
Due to the space limitation, we present results only for the Persian text-dependent speaker verification and speech recognition 
Experiments and Results ::: Speaker Verification Experiments
We conducted an experiment on text-dependent speaker verification part of the database, using the i-vector based method proposed in BIBREF9, BIBREF10 and applied it to the Persian portion of Part1  In this experiment, 20-dimensional MFCC features along with first and second derivatives are extracted from 16 kHz signals using HTK BIBREF11 with 25 ms Hamming windowed frames with 15 ms overlap 
The reported results are obtained with a 400-dimensional gender independent i-vector based system  The i-vectors are first length-normalized and are further normalized using phrase- and gender-dependent Regularized Within-Class Covariance Normalization (RWCCN) BIBREF10  Cosine distance is used to obtain speaker verification scores and phrase- and gender-dependent s-norm is used for normalizing the scores  For aligning speech frames to Gaussian components, monophone HMMs with 3 states and 8 Gaussian components in each state are used BIBREF10  We only model the phonemes which appear in the 5 Persian text-dependent phrases 
For speaker verification experiments, the results were reported in terms of Equal Error Rate (EER) and Normalized Detection Cost Function as defined for NIST SRE08 ($\mathrm {NDCF_{0 01}^{min}}$) and NIST SRE10 ($\mathrm {NDCF_{0 001}^{min}}$)  As shown in Table TABREF22, in text-dependent SV there are 4 types of trials: Target-Correct and Imposter-Correct refer to trials when the pass-phrase is uttered correctly by target and imposter speakers respectively, and in same manner, Target-Wrong and Imposter-Wrong refer to trials when speakers uttered a wrong pass-phrase  In this paper, only the correct trials (i e  Target-Correct as target trials vs Imposter-Correct as non-target trials) are considered for evaluating systems as it has been proved that these are the most challenging trials in text-dependent SV BIBREF8, BIBREF12 
Table TABREF23 shows the results of text-dependent experiments using Persian 100-spk and 3-sess setup  For filtering trials, the respondents' mobile brand and model were used in this experiment  In the table, the first two letters in the filter notation relate to the target trials and the second two letters (i e  right side of the colon) relate for non-target trials  For target trials, the first Y means the enrolment and test utterances were recorded using a device with the same brand by the target speaker  The second Y letter means both recordings were done using exactly the same device model  Similarly, the first Y for non-target trials means that the devices of target and imposter speakers are from the same brand (i e  manufacturer)  The second Y means that, in addition to the same brand, both devices have the same model  So, the most difficult target trials are “NN”, where the speaker has used different a device at the test time  In the same manner, the most difficult non-target trials which should be rejected by the system are “YY” where the imposter speaker has used the same device model as the target speaker (note that it does not mean physically the same device because each speaker participated in the project using a personal mobile device)  Hence, the similarity in the recording channel makes rejection more difficult 
The first row in Table TABREF23 shows the results for all trials  By comparing the results with the best published results on RSR2015 and RedDots BIBREF10, BIBREF8, BIBREF12, it is clear that the DeepMine database is more challenging than both RSR2015 and RedDots databases  For RSR2015, the same i-vector/HMM-based method with both RWCCN and s-norm has achieved EER less than 0 3% for both genders (Table VI in BIBREF10)  The conventional Relevance MAP adaptation with HMM alignment without applying any channel-compensation techniques (i e  without applying RWCCN and s-norm due to the lack of suitable training data) on RedDots Part1 for the male has achieved EER around 1 5% (Table XI in BIBREF10)  It is worth noting that EERs for DeepMine database without any channel-compensation techniques are 2 1 and 3 7% for males and females respectively 
One interesting advantage of the DeepMine database compared to both RSR2015 and RedDots is having several target speakers with more than one mobile device  This is allows us to analyse the effects of channel compensation methods  The second row in Table TABREF23 corresponds to the most difficult trials where the target trials come from mobile devices with different models while imposter trials come from the same device models  It is clear that severe degradation was caused by this kind of channel effects (i e  decreasing within-speaker similarities while increasing between-speaker similarities), especially for females 
The results in the third row show the condition when target speakers at the test time use exactly the same device that was used for enrollment  Comparing this row with the results in the first row proves how much improvement can be achieved when exactly the same device is used by the target speaker 
The results in the fourth row show the condition when imposter speakers also use the same device model at test time to fool the system  So, in this case, there is no device mismatch in all trials  By comparing the results with the third row, we can see how much degradation is caused if we only consider the non-target trials with the same device 
The fifth row shows similar results when the imposter speakers use device of the same brand as the target speaker but with a different model  Surprisingly, in this case, the degradation is negligible and it means that mobiles from a specific brand (manufacturer) have different recording channel properties 
The degraded female results in the sixth row as compared to the third row show the effect of using a different device model from the same brand for target trials  For males, the filters brings almost the same subsets of trials, which explains the very similar results in this case 
Looking at the first two and the last row of Table TABREF23, one can notice the significantly worse performance obtained for the female trials as compared to males  Note that these three rows include target trials where the devices used for enrollment do not necessarily match the devices used for recording test utterances  On the other hand, in rows 3 to 6, which exclude such mismatched trials, the performance for males and females is comparable  This suggest that the degraded results for females are caused by some problematic trials with device mismatch  The exact reason for this degradation is so far unclear and needs a further investigation 
In the last row of the table, the condition of the second row is relaxed: the target device should have different model possibly from the same brand and imposter device only needs to be from the same brand  In this case, as was expected, the performance degradation is smaller than in the second row 
Experiments and Results ::: Speech Recognition Experiments
In addition to speaker verification, we present several speech recognition experiments on Part3  The experiments were performed with the Kaldi toolkit BIBREF13  For training HMM-based MonoPhone model, only 20 thousands of shortest utterances are used and for other models the whole training data is used  The DNN based acoustic model is a time-delay DNN with low-rank factorized layers and skip connections without i-vector adaptation (a modified network from one of the best performing LibriSpeech recipes)  The network is shown in Table TABREF25: there are 16 F-TDNN layers, with dimension 1536 and linear bottleneck layers of dimension 256  The acoustic model is trained for 10 epochs using lattice-free maximum mutual information (LF-MMI) with cross-entropy regularization BIBREF14  Re-scoring is done using a pruned trigram language model and the size of the dictionary is around 90,000 words 
Table TABREF26 shows the results in terms of word error rate (WER) for different evaluated methods  As can be seen, the created database can be used to train well performing and practically usable Persian ASR models 
Conclusions
In this paper, we have described the final version of a large speech corpus, the DeepMine database  It has been collected using crowdsourcing and, according to the best of our knowledge, it is the largest public text-dependent and text-prompted speaker verification database in two languages: Persian and English  In addition, it is the largest text-independent speaker verification evaluation database, making it suitable to robustly evaluate state-of-the-art methods on different conditions  Alongside these appealing properties, it comes with phone-level transcription, making it suitable to train deep neural network models for Persian speech recognition 
We provided several evaluation protocols for each part of the database  The protocols allow researchers to investigate the performance of different methods in various scenarios and study the effects of channels, duration and phrase text on the performance  We also provide two test sets for speech recognition: One normal test set with a few minutes of speech for each speaker and one large test set with more (30 minutes on average) speech that can be used for any speaker adaptation method 
As baseline results, we reported the performance of an i-vector/HMM based method on Persian text-dependent part  Moreover, we conducted speech recognition experiments using conventional HMM-based methods, as well as state-of-the-art deep neural network based method using Kaldi toolkit with promising performance  Text-dependent results have shown that the DeepMine database is more challenging than RSR2015 and RedDots databases 
Acknowledgments
The data collection project was mainly supported by Sharif DeepMine company  The work on the paper was supported by Czech National Science Foundation (GACR) project "NEUREM3" No  19-26934X and the National Programme of Sustainability (NPU II) project "IT4Innovations excellence in science - LQ1602" 

ground_truth: Women represent 33.16% of the speakers

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What datasets are used?
Context 2:  If detected, the action is executed  At this step, many substeps may be performed, like searching for an information, computing maths, or generating information to create the answer  All of this may require a search in the Context and also may activate the Error Detector component to check if the dialog did not run into a wrong state  After the answer is generated, the Filtering phase is activated again to check if the reply should be really sent  If so, it is sent to the Hub which, again may check if it can be broadcasted before actually doing it 
The topic classifier is domain-dependent and is not mandatory  However, the chatbot can better react when the intent or action is not detected, which means that it does not know how to answer  Many reasons might explain this situation: the set of intents might be incomplete, the action might not have produced the proper behavior, misunderstanding might happen, or the chatbot was not designed to reply to a particular topic  In all cases, it must be able to produce a proper reply, if needed  Because this might happen throughout the workflow, the sooner that information is available, the better the chatbot reacts  Therefore it is one of the first executions of the flow 
Dependency is the notion that linguistic units, e g  words, are connected to each other by directed links  The (finite) verb is taken to be the structural center of clause structure  All other syntactic units (words) are either directly or indirectly connected to the verb in terms of the directed links, which are called dependencies  It is a one-to-one correspondence: for every element (e g  word or morph) in the sentence, there is exactly one node in the structure of that sentence that corresponds to that element  The result of this one-to-one correspondence is that dependency grammars are word (or morph) grammars  All that exist are the elements and the dependencies that connect the elements into a structure  Dependency grammar (DG) is a class of modern syntactic theories that are all based on the dependency relation 
Semantic dependencies are understood in terms of predicates and their arguments  Morphological dependencies obtain between words or parts of words  To facilitate future research in unsupervised induction of syntactic structure and to standardize best-practices, a tagset that consists of twelve universal part-of-speech categories was proposed BIBREF41  
Dependency parsers have to cope with a high degree of ambiguity and nondeterminism which let to different techniques than the ones used for parsing well-defined formal languages  Currently the mainstream approach uses algorithms that derive a potentially very large set of analyses in parallel and when disambiguation is required, this approach can be coupled with a statistical model for parse selection that ranks competing analyses with respect to plausibility BIBREF42  
Below we present an example of a dependency tree for the utterance:
"I want to invest 10 thousands":
[s]""blue[l]:black
"tree": {
"want VERB ROOT": {
"I PRON nsubj": {},
"to ADP mark": {},
"invest VERB nmod": {
"thousands NOUN nmod": {
"10 NUM nummod": {}
}
}
}
The coarse-grained part-of-speech tags, or morphological dependencies (VERB, PRON, ADP, NOUN and NUM) encode basic grammatical categories and the grammatical relationships (nsubjs, nmod, nummod) are defined in the Universal Dependencies project BIBREF41  
In this module, the dependency tree generated is used together with a set of rules to extract information that is saved in the context using the Frame-based approach  This approach fills the slots of the frame with the extracted values from the dialogue  Frames are like forms and slots are like fields  Using the knowledge's conceptual model, the fields are represented by the elements Entities and Features  In the dependency tree example, the entity would be the implicit concept: the investment option, and the feature is the implicit concept: initial amount – 10 thousands  Since the goal is to invest, and there are more entities needed for that (i e , fields to be filled), the next node in the Intention Flow tree would return an utterance which asks the user the time of investment, if he/she has not provided yet 
This module could be implemented using different approaches according to the domain, but tree search algorithms will be necessary for doing the tree parsing 
The Intent Classifier component aims at recognizing not only the Intent but the goal of the utterance sent by a Member, so it can properly react  The development of an intent classifier needs to deal with the following steps:
i) the creation of dataset of intents, to train the classification algorithm;
ii) the design of a classification algorithm that provides a reasonable level of accuracy;
iii) the creation of dataset of trees of intents, the same as defined in i) and which maps the goals;
iv) the design of a plan-graph search algorithm that maps the goal's state to a node in the graph;
There are several approaches to create training sets for dialogues: from an incremental approach to crowdsourcing  In the incremental approach, the Wizard of Oz method can be applied to a set of potential users of the system, and from this study, a set of questions that the users asked posted to the `fake' system can be collected  These questions have to be manually classified into a set of intent classes, and used to train the first version of the system  Next, this set has to be increased both in terms of number of classes and samples per class 
The Speech Act Classifier can be implemented with many speech act classes as needed by the application  The more classes, the more flexible the chatbot is  It can be built based on dictionaries, or a machine learning-based classifier can be trained  In the table below we present the main and more general speech act classes BIBREF43 used in the Chatbots with examples to differentiate one from another:
There are at least as many Action classes as Speech Act classes, since the action is the realization of a speech act  The domain specific classes, like "Inform_News" or "Inform_Factoids", enhance the capabilities of answering of a chatbot 
The Action Classifier can be defined as a multi-class classifier with the tuple DISPLAYFORM0
where INLINEFORM0 is the intent of the answer defined in ( EQREF46 ), INLINEFORM1 is the speech act of the answer, INLINEFORM2 and INLINEFORM3 are the sets of entities and features needed to produce the answer, if needed, respectively 
This component is responsible for implementing the behavior of the Action class  Basic behaviors may exist and be shared among different chatbots, like the ones that implement the greetings, thanks or not understood  Although they can be generic, they can also be personalized to differentiate the bot from one another and also to make it more "real"  Other cases like to inform, to send a query, to send a proposal, they are all domain-dependent and may require specific implementations 
Anyway, figure FIGREF59 shows at the high level the generic workflow  If action class detected is task-oriented, the system will implement the execution of the task, say to guide a car, to move a robot's arm, or to compute the return of investments  The execution might need to access an external service in the Internet in order to complete the task, like getting the inflation rate, or the interest rate, or to get information about the environment, or any external factor  During the execution or after it is finished, the utterance is generated as a reply and, if no more tasks are needed, the action execution is finished 
In the case of coordination of chatbots, one or more chatbots with the role of mediator may exist in the chat group and, at this step, it is able to invite one or more chatbots to the chat group and it is also able to redirect the utterances, if the case 
The proposed architecture addresses the challenges as the following:
What is the message/utterance about? solved by the Parsing phase;
Who should reply to the utterance? solved by the Filtering phase and may be enforced by the Hub;
How the reply should be built/generated? solved by the Acting phase;
When should the reply be sent? may be solved by the Acting phase or the Filtering phase, and may be enforced by the Hub;
And Context and Logging module is used throughout all phases 
Architecture Implementation and Evaluation
This section presents one implementation of the conceptual architecture presented in last section  After many refactorings, a framework called SABIA (Speech-Act-Based Intelligent Agents Framework) has been developed and CognIA (Cognitive Investment Advisor) application has been developed as an instantiation of SABIA framework  We present then the accuracy and some automated tests of this implementation 
Speech-Act-based Intelligent Agents Framework
SABIA was developed on top of Akka middleware  Akka is a toolkit and runtime that implements the Actor Model on the JVM  Akka's features, like concurrency, distributed computing, resilience, and message-passing were inspired by Erlang's actor model BIBREF44 BIBREF45   The actor model is a mathematical model of concurrent computation that treats "actors" as the universal primitives of concurrent computation  In response to a message that it receives, an actor can: make local decisions, create more actors, send more messages, and determine how to respond to the next received message  Actors may modify private state, but can only affect each other through messages (avoiding the need for any locks)  Akka middleware manages the actors life cycle and actors look up by theirs name, locally or remotely 
We implemented each Member of the Chat Group as an Actor by extending the UntypedActor class of Akka middleware  Yet, we created and implemented the SabiaActorSystem as a singleton (i e , a single instance of it exists in the system) BIBREF46 that has a reference to Akka's ActorSystem  During SabiaActorSystem's initialization, all parsers that consume too much memory during their initialization to load models are instantiated as singletons  In this way, we save time on their calls during the runtime  Moreover, all chat group management, like to join or leave the group, or to broadcast or filter a message at the Hub level is implemented in SABIA through the Chat Group behavior 
This is implemented in SABIA as a singleton that is initialized during the SabiaActorSystem initialization with the URL of the service that implements the dependency parsing and is used on each utterance's arrival through the execution of the tagUtterance method  The service must retrieve a JSON Object with the dependency tree which is then parsed using depth-first search 
SABIA does not support invariants for frame parsing  We are leaving this task to the instantiated application 
There are two intent classifiers that can be loaded with trained models in order to be ready to be used at runtime: the 1-nearest-neighbor (1NN) and the SVM-based classifier 
SABIA implements the Action Classifier assuming that the application uses a relational database with a data schema that implements the conceptual model presented in Figure FIGREF44   Then the invariants parts that use SQL are already present and the application only needs to implement the database connection and follow the required data schema 
SABIA provides partial implemented behavior for the Action through the Template method design pattern BIBREF46 , which implements the invariants parts of the action execution and leaves placeholders for customization 
CognIA: A Cognitive Investment Advisor
We developed CognIA, which is an instantiation of Sabia framework  A conversation is composed of a group chat that can contain multiple users and multiple chatbots  This example, in particular, has a mediator that can help users on financial matters, more specifically on investment options  For example, consider the following dialogue in the table below:
The Table TABREF71 shows an example that uses the mixed-initiative dialogue strategy, and a dialogue mediator to provide coordination control  In this example of an application, there are many types of intentions that should be answered: Q&A (question and answer) about definitions, investment options, and about the current finance indexes, simulation of investments, which is task-oriented and requires computation, and opinions, which can be highly subjective 
In Table SECREF72 , we present the interaction norms that were needed in Cognia  The Trigger column describes the event that triggers the Behavior specified in the third column  The Pre-Conditions column specifies what must happen in order to start the behavior execution  So, for instance, line 2, when the user sends an utterance in the chat group, an event is triggered and, if the utterance's topic is CDB (Certificate of Deposit which is a fixed rate investment) or if it is about the Savings Account investment option and the speech act is not Query_Calculation and the CDB and Savings Account members are not in the chat, then the behavior is activated  The bot members that implement these behaviors are called cdbguru and poupancaguru  Therefore these names are used when there is a mention 
Note that these interactions norms are not explicitly defined as obligations, permissions, and prohibitions  They are implict from the behavior described  During this implementation, we did not worry about explicitly defining the norms, because the goal was to evaluate the overall architecture, not to enhance the state of the art on norms specification for conversational systems  In addition, CognIA has only the presented interaction norms defined in Table SECREF72 , which is a very small set that that does not required model checking or verification of conflicts 
|p2cm|p5 0cm|p5 0cm|Cognia Interaction NormsCognia Interaction Norms
Trigger Pre-Conditions Behavior
On group chat creation Cognia chatbot is available Cognia chatbot joins the chat with the mediator role and user joins the chat with the owner_user role
On utterance sent by user Utterance's topic is CDB (cdbguru) or Savings Account (poupancaguru) and speech act is not Query_Calculation and they are not in the chat Cognia invites experts to the chat and repeats the utterance to them
On utterance sent by user Utterance's topic is CDB (cdbguru) or Savings Account (poupancaguru) and speech act is not Query_Calculation and they are in the chat Cognia waits for while and cdbguru or poupancaguru respectively handles the utterance  If they don't understand, they don't reply
On utterance sent by the experts If Cognia is waiting for them and has received both replies Cognia does not wait anymore
On utterance sent Utterance mentions cdbguru or poupancaguru cdbguru or poupancaguru respectively handles the utterance
On utterance sent Utterance mentions cdbguru or poupancaguru and they don't reply after a while and speech act is Query_Calculation Cognia sends I can only chat about investments   
On utterance sent Utterance mentions cdbguru or poupancaguru and they don't reply after while and speech act is not Query_Calculation Cognia sends I didn't understand
On utterance sent Utterance's speech act is Query_Calculation and period or initial amount of investment were not specified Cognia asks the user the missing information
On utterance sent Utterance's speech act is Query_Calculation and period and initial amount of investment were specified and the experts are not in the chat Cognia invites experts to the chat and repeats the utterance to them
On utterance sent Utterance's speech act is Query_Calculation and period and initial amount of investment were specified and the experts are in the chat Cognia repeats the utterance to experts
On utterance sent Utterance's speech act is Query_Calculation Cognia extracts variables and saves the context
On utterance sent Utterance's speech act is Query_Calculation and the experts are in the chat and the experts are mentioned Experts extract information, save in the context, compute calculation and send information
On utterance sent Utterance's speech act is Inform_Calculation and Cognia received all replies Cognia compares the results and inform comparison
On utterance sent Utterance mentions a chatbot but has no other text The chatbot replies How can I help you?
On utterance sent Utterance is not understood and speech act is Question The chatbot replies I don't know    I can only talk about topic X
On utterance sent Utterance is not understood and speech act is not Question The chatbot replies I didn't understand
On utterance sent Utterance's speech act is one of { Greetings, Thank, Bye } All chatbots reply to utterance
On group chat end All chatbots leave the chat, and the date and time of the end of chat is registered
We instantiated SABIA to develop CognIA as follows: the Mediator, Savings Account, CDB and User Actors are the Members of the Chat Group  The Hub was implemented using two servers: Socket io and Node JS which is a socket client of the Socket io server  The CognIA system has also one Socket Client for receiving the broadcast and forwarding to the Group Chat Manager  The former will actually do the broadcast to every member after enforcing the norms that applies specified in Table SECREF72   Each Member will behave according to this table too  For each user of the chat group, on a mobile or a desktop, there is its corresponding actor represented by the User Actor in the figure  Its main job is to receive Akka's broadcast and forward to the Socket io server, so it can be finally propagated to the users 
All the intents, actions, factual answers, context and logging data are saved in DashDB (a relational Database-as-a-Service system)  When an answer is not retrieved, a service which executes the module Search Finance on Social Media on a separate server is called  This service was implemented with the assumption that finance experts post relevant questions and answers on social media  Further details are explained in the Action execution sub-section 
We built a small dictionary-based topic classifier to identify if an utterance refers to finance or not, and if it refers to the two investment options (CDB or Savings Account) or not 
The dependency parsing is extremely important for computing the return of investment when the user sends an utterance with this intention  Our first implementation used regular expressions which led to a very fragile approach  Then we used a TensorFlow implementation BIBREF47 of a SyntaxNet model for Portuguese and used it to generate the dependency parse trees of the utterances  The SyntaxNet model is a feed-forward neural network that operates on a task-specific transition system and achieves the state-of-the-art on part-of-speech tagging, dependency parsing and sentence compression results BIBREF48   Below we present output of the service for the utterance:
"I want to invest 10 thousands in 40 months":
[s]""blue[l]:black
{ "original": "I would like to invest 10 thousands in 40 months",
"start_pos": [
23,
32],
"end_pos": [
27,
33],
"digits": [
10000,
40],
"converted": "I would like to invest 10000 in 40 months",
"tree": {
"like VERB ROOT": {
"I PRON nsubj": {},
"would MD aux":{
"invest VERB xcomp":{
"to TO aux": {},
"10000 NUM dobj": {},
"in IN prep": {
"months NOUN pobj":{
"40 NUM num": {}}}}}}}
The service returns a JSON Object containing six fields: original, start_pos, end_pos, digits, converted and tree  The original field contains the original utterance sent to the service  The converted field contains the utterance replaced with decimal numbers, if the case (for instance, "10 thousands" was converted to "10000" and replaced in the utterance)  The start_pos and end_pos are arrays that contain the start and end char positions of the numbers in the converted utterance  While the tree contains the dependency parse tree for the converted utterance 
Given the dependency tree, we implemented the frame parsing which first extracts the entities and features from the utterance and saves them in the context  Then, it replaces the extracted entities and features for reserved characters 
extract_period_of_investment (utteranceTree) [1] [t] numbersNodes INLINEFORM0 utteranceTree getNumbersNodes(); [t] foreach(numberNode in numbersNodes) do [t] parentsOfNumbersNode INLINEFORM1 numbersNode getParents() [t] foreach(parent in parentsOfNumbersNodes) do [t] if ( parent name contains { "day", "month", "year"} ) then [t] parentOfParent INLINEFORM2 parent getParent() [t] if ( parentOfParent is not null and
parentOfParent getPosTag==Verb and
parentOfParent name in investmentVerbsSet ) then [t] return numberNode
Therefore an utterance like "I would like to invest 10 thousands in 3 years" becomes "I would like to invest #v in #dt years"  Or "10 in 3 years" becomes "#v in #dt years", and both intents have the same intent class 
For doing that we implemented a few rules using a depth-first search algorithm combined with the rules as described in Algorithm UID79 , Algorithm UID79 and Algorithm UID79   Note that our parser works only for short texts on which the user's utterance mentions only one period of time and/ or initial amount of investment in the same utterance 
extract_initial_amount_of_investment (utteranceTree) [1] [t] numbersNodes INLINEFORM0 utteranceTree getNumbersNodes(); [t] foreach(numberNode in numbersNodes) do [t] parentsOfNumbersNode INLINEFORM1 numbersNode getParents() [t] foreach(parent in parentsOfNumbersNodes) do [t] if ( parent name does not contain { "day", "month", "year"} ) then [t] return numberNode
frame_parsing(utterance, utteranceTree) [1] [t] period INLINEFORM0 extract_period_of_investment (utteranceTree) [t] save_period_of_investment(period) [t] value INLINEFORM1 extract_initial_amount_of_investment (utteranceTree) [t] save_initial_amount_of_investment(value) [t] new_intent INLINEFORM2 replace(new_intent, period, "#dt") [t] new_intent INLINEFORM3 replace(new_intent, value, "#v")
In CognIA we have complemented the speech act classes with the ones related to the execution of specific actions  Therefore, if the chatbot needed to compute the return of investment, then, once it is computed, the speech act of the reply will be Inform_Calculation and the one that represents the query for that is Query_Calculation  In table TABREF81 we list the specific ones 
Given that there is no public dataset available with financial intents in Portuguese, we have employed the incremental approach to create our own training set for the Intent Classifier  First, we applied the Wizard of Oz method and from this study, we have collected a set of 124 questions that the users asked  Next, after these questions have been manually classified into a set of intent classes, and used to train the first version of the system, this set has been increased both in terms of number of classes and samples per class, resulting in a training set with 37 classes of intents, and a total 415 samples, with samples per class ranging from 3 to 37 
We have defined our classification method based on features extracted from word vectors  Word vectors consist of a way to encode the semantic meaning of the words, based on their frequency of co-occurrence  To create domain-specific word vectors, a set of thousand documents are needed related to desired domain  Then each intent from the training set needs to be encoded with its corresponding mean word vector  The mean word vector is then used as feature vector for standard classifiers 
We have created domain-specific word vectors by considering a set 246,945 documents, corresponding to of 184,001 Twitter posts and and 62,949 news articles, all related to finance  
The set of tweets has been crawled from the feeds of blog users who are considered experts in the finance domain  The news article have been extracted from links included in these tweets  This set contained a total of 63,270,124 word occurrences, with a vocabulary of 97,616 distinct words  With the aforementioned word vectors, each intent from the training set has been encoded with its corresponding mean word vector  The mean word vector has been then used as feature vector for standard classifiers 
As the base classifier, we have pursued with a two-step approach  In the first step, the main goal was to make use of a classifier that could be easily retrained to include new classes and intents  For this reason, the first implementation of the system considered an 1-nearest-neighbor (1NN) classifier, which is simply a K-nearest-neighbor classifier with K set to 1  With 1NN, the developer of the system could simply add new intents and classes to the classifier, by means of inserting new lines into the database storing the training set  Once we have considered that the training set was stable enough for the system, we moved the focus to an approach that would be able to provide higher accuracy rates than 1NN  For this, we have employed Support Vector Machines (SVM) with a Gaussian kernel, the parameters of which are optimized by means of a grid search 
We manually mapped the intent classes used to train the intent classifier to action classes and the dependent entities and features, when the case  Table TABREF85 summarizes the number of intent classes per action class that we used in CognIA 
For the majority of action classes we used SABIA's default behavior  For instance, Greet and Bye actions classes are implemented using rapport, which means that if the user says "Hi" the chatbot will reply "Hi" 
The Search News, Compute and Ask More classes are the ones that require specific implemention for CognIA as following:
Search News: search finance on social media service BIBREF49 , BIBREF50 receives the utterance as input, searches on previously indexed Twitter data for finance for Portuguese and return to the one which has the highest score, if found 
Ask More: If the user sends an utterance that has the intention class of simulating the return of investment, while not all variables to compute the return of investment are extracted from the dialogue, the mediator keeps asking the user these information before it actually redirects the query to the experts  This action then checks the state of the context given the specified intent flow as described in ( EQREF46 ) and ( EQREF57 ) in section SECREF4 to decide which variables are missing  For CognIA we manually added these dependencies on the database 
Compute: Each expert Chatbot implements this action according to its expertise  The savings account chatbot computes the formula ( EQREF90 ) and the certificate of deposit computes the formula ( EQREF92 )  Both are currently formulas for estimating in Brazil  DISPLAYFORM0
where INLINEFORM0 is the return of investment for the savings account, INLINEFORM1 is the initial value of investment, INLINEFORM2 is the savings account interest rate and INLINEFORM3 is the savings account rate base  DISPLAYFORM0
where INLINEFORM0 is the return of investment for certificate of deposit, INLINEFORM1 is the initial value of investment, INLINEFORM2 is the Interbank Deposit rate (DI in Portuguese), INLINEFORM3 is the ID's percentual payed by the bank (varies from 90% to 120%), INLINEFORM4 is the number of days the money is invested, and finally INLINEFORM5 is the income tax on the earnings 
Intention Classifier Accuracy
In Table TABREF95 we present the comparison of some distinct classification on the first version of the training set, i e  the set used to deploy the first classifier into the system  Roughly speaking, the 1NN classifier has been able to achieve a level of accuracy that is higher than other well-known classifiers, such as Logistic Regression and Naïve Bayes, showing that 1NN is suitable as a development classifier  Nevertheless, a SVM can perform considerable better than 1NN, reaching accuracies of about 12 percentage points higher, which demonstrates that this type of base classifier is a better choice to be deployed once the system is stable enough  It is worth mentioning that these results consider the leave-one-out validation procedure, given the very low number of samples in some classes 
As we mentioned, the use of an 1NN classifier has allowed the developer of the system to easily add new intent classes and samples whenever they judged it necessary, so that the system could present new actions, or the understanding of the intents could be improved  As a consequence, the initial training set grew from 37 to 63 classes, and from 415 to 659 samples, with the number of samples per class varying from 2 to 63  For visualizing the impact on the accuracy of the system, in Table TABREF96 we present the accuracy of the same classifiers used in the previous evaluation, in the new set  In this case, we observe some drop in accuracy for 1NN, showing that this classifier suffers in dealing with scalability  On the other hand, SVM has shown to scale very well to more classes and samples, since its accuracy kept at a very similar level than that with the other set, with a difference of about only 1 percentage point 
Testing SABIA
In this section, we describe the validation framework that we created for integration tests  For this, we developed it as a new component of SABIA's system architecture and it provides a high level language which is able to specify interaction scenarios that simulate users interacting with the deployed chatbots  The system testers provide a set of utterances and their corresponding expected responses, and the framework automatically simulates users interacting with the bots and collect metrics, such as time taken to answer an utterance and other resource consumption metrics (e g , memory, CPU, network bandwidth)  Our goal was to: (i) provide a tool for integration tests, (ii) to validate CognIA's implementation, and (iii) to support the system developers in understanding the behavior of the system and which aspects can be improved  Thus, whenever developers modify the system's source code, the modifications must first pass the automatic test before actual deployment 
The test framework works as follows  The system testers provide a set INLINEFORM0 of dialogues as input  Each dialogue INLINEFORM1 INLINEFORM2 INLINEFORM3 is an ordered set whose elements are represented by INLINEFORM4 , where INLINEFORM5 is the user utterance and INLINEFORM6 is an ordered set of pairs INLINEFORM7 that lists each response INLINEFORM8 each chatbot INLINEFORM9 should respond when the user says INLINEFORM10   For instance, Table UID98 shows a typical dialogue ( INLINEFORM11 ) between a user and the CognIA system  Note that we are omitting part of the expected answer with "   " just to better visualize the content of the table 
|p3 6cmp0 4cmp4 5cmp3 2cm|Content of dialogue INLINEFORM0 (example of dialogue in CognIA)Content of dialogue INLINEFORM1 (example of dialogue in CognIA
User utterance INLINEFORM0 rId Expected response INLINEFORM1 Chatbot INLINEFORM2
gray!25 hello 1 Hello Mediator
white what is cdb? 2 @CDBExpert what is cdb? Mediator
white 3 CDB is a type of investment that    CDB Expert
gray!25 which is better: cdb or savings account? 4 I found a post in the social media for     Mediator
white i would like to invest R$ 50 in six months 5 @SavingsAccountExpert and @CDBExpert, could you do a simulation    Mediator
white 6 If you invest in Savings Account,     Savings Account Exp 
white 7 If you invest in CDB,    CDB Expert
white 8 Thanks Mediator
white 9 @User, there is no significant difference   Mediator
gray!25 so i want to invest R$ 10000 in 2 years 10 @SavingsAccountExpert and @CDBExpert, could you do a simulation    Mediator
gray!25 11 If you invest in Savings Account,    Savings Account Exp 
gray!25 12 If you invest in CDB,    CDB Expert
gray!25 13 Thanks Mediator
gray!25 14 @User, in that case, it is better    Mediator
white what if i invest R$10,000 in 5 years? 15 @SavingsAccountExpert and @CDBExpert, could you do a simulation    Mediator
white 16 If you invest in Saving Account,    Savings Account Exp 
white 17 If you invest in CDB,    CDB Expert
white 18 Thanks Mediator
white 19 @User, in that case, it is better    Mediator
gray!25 how about 15 years? 20 @SavingsAccountExpert and @CDBExpert, could you do a simulation    Mediator
gray!25 21 If you invest in Savings Account,    Savings Account Exp
gray!25 22 If you invest in CDB,    CDB Expert
gray!25 23 Thanks Mediator
gray!25 24 @User, in that case, it is better    Mediator
white and 50,0000? 25 @SavingsAccountExpert and @CDBExpert, could you do a simulation    Mediator
white 26 If you invest in Savings Account,    Savings Account Exp 
white 27 If you invest in CDB,    CDB Expert
white 28 Thanks Mediator
white 29 @User, in that case, it is better   Mediator
gray!25 I want to invest in 50,000 for 15 years in CDB 30 Sure, follow this link to your bank    Mediator
white thanks 31 You are welcome  Mediator
The testers may also inform the number of simulated users that will concurrently use the platform  Then, for each simulated user, the test framework iterates over the dialogues in INLINEFORM0 and iterates over the elements in each dialogue to check if each utterance INLINEFORM1 was correctly responded with INLINEFORM2 by the chatbot INLINEFORM3   There is a maximum time to wait  If a bot does not respond with the expected response in the maximum time (defined by the system developers), an error is raised and the test is stopped to inform the developers about the error  Otherwise, for each correct bot response, the test framework collects the time taken to respond that specific utterance by the bot for that specific user and continues for the next user utterance  Other consumption resource metrics (memory, CPU, network, disk)  The framework is divided into two parts  One part is responsible to gather resource consumption metrics and it resides inside SABIA  The other part works as clients (users) interacting with the server  It collects information about time taken to answer utterances and checks if the utterances are answered correctly 
By doing this, we not only provide a sanity test for the domain application (CognIA) developed in SABIA framework, but also a performance analysis of the platform  That is, we can: validate if the bots are answering correctly given a pre-defined set of known dialogues, check if they are answering in a reasonable time, and verify the amount of computing resources that were consumed to answer a specific utterance  Given the complexity of CognIA, these tests enable debugging of specific features like: understanding the amount of network bandwidth to use external services, or analyzing CPU and memory consumption when responding a specific utterance  The later may happen when the system is performing more complex calculations to indicate the investment return, for instance 
CognIA was deployed on IBM Bluemix, a platform as a service, on a Liberty for Java Cloud Foundry app with 3 GB RAM memory and 1 GB disk  Each of the modules shown in Figure FIGREF74 are deployed on separate Bluemix servers  Node JS and Socket IO servers are both deployed as Node Cloud Foundry apps, with 256 MB RAM memory and 512 MB disk each  Search Finance on Social Media is on a Go build pack Cloud Foundry app with 128 MB RAM memory and 128 GB disk  For the framework part that simulates clients, we instantiated a virtual machine with 8 cores on IBM's SoftLayer that is able to communicate with Bluemix  Then, the system testers built two dialogues, i e , INLINEFORM0   The example shown in Table UID98 is the dialogue test INLINEFORM1   For the dialogue INLINEFORM2 , although it also has 10 utterances, the testers varied some of them to check if other utterances in the finance domain (different from the ones in dialogue INLINEFORM3 ) are being responded as expected by the bots  Then, two tests are performed and the results are analyzed next  All tests were repeated until the standard deviation of the values was less than 1%  The results presented next are the average of these values within the 1% margin 
Test 1: The first test consists of running both dialogues INLINEFORM0 and INLINEFORM1 for only one user for sanity check  We set 30 seconds as the maximum time a simulated user should wait for a bot correct response before raising an error  The result is that all chatbots (Mediator, CDBExpert, and SavingsAccountExpert) responded all expected responses before the maximum time  Additionally, the framework collected how long each chatbot took to respond an expected answer 
In Figure FIGREF101 , we show the results for those time measurements for dialogue INLINEFORM0 , as for the dialogue INLINEFORM1 the results are approximately the same  The x-axis (Response Identifier) corresponds to the second column (Resp  Id) in Table UID98   We can see, for example, that when the bot CDBExpert responds with the message 3 to the user utterance "what is cdb?", it is the only bot that takes time different than zero to answer, which is the expected behavior  We can also see that the Mediator bot is the one that takes the longest, as it is responsible to coordinate the other bots and the entire dialogue with the user  Moreover, when the expert bots (CDBExpert and SavingsAccountExpert) are called by the Mediator to respond to the simulation calculations (this happens in responses 6, 7, 11, 12, 16, 17, 21, 22, 26, 27), they take approximately the same to respond  Finally, we see that when the concluding responses to the simulation calculations are given by the Mediator (this happens in responses 9, 14, 19, 24, 29), the response times reaches the greatest values, being 20 seconds the greatest value in response 19  These results support the system developers to understand the behavior of the system when simulated users interact with it and then focus on specific messages that are taking longer 
Test 2: This test consists of running dialogue INLINEFORM0 , but now using eight concurrent simulated users  We set the maximum time to wait to 240 seconds, i e , eight times the maximum set up for the single user in Test 1  The results are illustrated in Figure FIGREF102 , where we show the median time for the eight users  The maximum and minimum values are also presented with horizontal markers  Note that differently than what has been shown in Figure FIGREF101 , where each series represents one specific chatbot, in Figure FIGREF102 , the series represents the median response time for the responses in the order (x-axis) they are responded, regardless the chatbot 
Comparing the results in Figure FIGREF102 with the ones in Figure FIGREF101 , we can see that the bots take longer to respond when eight users are concurrently using the platform than when a single user uses it, as expected  For example, CDBExpert takes approximately 5 times longer to respond response 3 to eight users than to respond to one user  On average, the concluding responses to the simulation questions (i e , responses 9, 14, 19, 24, 29) take approximately 7 3 times more to be responded with eight users than with one user, being the response 9 the one that presented greatest difference (11 4 times longer with eight users than with one)  These results help the system developers to diagnose the scalability of the system architecture and to plan sizing and improvements 
Conclusions and Future Work
In this article, we explored the challenges of engineering MPCS and we have presented a hybrid conceptual architecture and its implementation with a finance advisory system 
We are currently evolving this architecture to be able to support decoupled interaction norms specification, and we are also developing a multi-party governance service that uses that specification to enforce exchange of compliant utterances 
In addition, we are exploring a micro-service implementation of SABIA in order to increase its scalability and performance, so thousands of members can join the system within thousands of conversations 
Acknowledgments
The authors would like to thank Maximilien de Bayser, Ana Paula Appel, Flavio Figueiredo and Marisa Vasconcellos, who contributed with discussions during SABIA and CognIA's implementation 

ground_truth: Custom dataset with user questions; set of documents, twitter posts and news articles, all related to finance.

a self-collected financial intents dataset in Portuguese

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. what is the source of the data?
Context 4:  We can observe that for example pages about news tend to have more sadness and anger posts, while pages about cooking and tv-shows have a high percentage of joy posts  We will use this information to find the best set of pages for a given target domain (see Section SECREF5 ) 
Model
There are two main decisions to be taken in developing our model: (i) which Facebook pages to select as training data, and (ii) which features to use to train the model, which we discuss below  Specifically, we first set on a subset of pages and then experiment with features  Further exploration of the interaction between choice of pages and choice of features is left to future work, and partly discussed in Section SECREF6   For development, we use a small portion of the Affective data set described in Section SECREF4 , that is the portion that had been released as development set for SemEval's 2007 Task 14 BIBREF7 , which contains 250 annotated sentences (Affective development, Section SECREF4 )  All results reported in this section are on this dataset  The test set of Task 14 as well as the other two datasets described in Section SECREF3 will be used to evaluate the final models (Section SECREF4 ) 
Selecting Facebook pages
Although page selection is a crucial ingredient of this approach, which we believe calls for further and deeper, dedicated investigation, for the experiments described here we took a rather simple approach  First, we selected the pages that would provide training data based on intuition and availability, then chose different combinations according to results of a basic model run on development data, and eventually tested feature combinations, still on the development set 
For the sake of simplicity and transparency, we first trained an SVM with a simple bag-of-words model and default parameters as per the Scikit-learn implementation BIBREF13 on different combinations of pages  Based on results of the attempted combinations as well as on the distribution of emotions in the development dataset (Figure FIGREF9 ), we selected a best model (B-M), namely the combined set of Time, The Guardian and Disney, which yields the highest results on development data  Time and The Guardian perform well on most emotions but Disney helps to boost the performance for the Joy class 
Features
In selecting appropriate features, we mainly relied on previous work and intuition  We experimented with different combinations, and all tests were still done on Affective development, using the pages for the best model (B-M) described above as training data  Results are in Table TABREF20   Future work will further explore the simultaneous selection of features and page combinations 
We use a set of basic text-based features to capture the emotion class  These include a tf-idf bag-of-words feature, word (2-3) and character (2-5) ngrams, and features related to the presence of negation words, and to the usage of punctuation 
This feature is used in all unsupervised models as a source of information, and we mainly include it to assess its contribution, but eventually do not use it in our final model 
We used the NRC10 Lexicon because it performed best in the experiments by BIBREF10 , which is built around the emotions anger, anticipation, disgust, fear, joy, sadness, and surprise, and the valence values positive and negative  For each word in the lexicon, a boolean value indicating presence or absence is associated to each emotion  For a whole sentence, a global score per emotion can be obtained by summing the vectors for all content words of that sentence included in the lexicon, and used as feature 
As additional feature, we also included Word Embeddings, namely distributed representations of words in a vector space, which have been exceptionally successful in boosting performance in a plethora of NLP tasks  We use three different embeddings:
Google embeddings: pre-trained embeddings trained on Google News and obtained with the skip-gram architecture described in BIBREF14   This model contains 300-dimensional vectors for 3 million words and phrases 
Facebook embeddings: embeddings that we trained on our scraped Facebook pages for a total of 20,000 sentences  Using the gensim library BIBREF15 , we trained the embeddings with the following parameters: window size of 5, learning rate of 0 01 and dimensionality of 100  We filtered out words with frequency lower than 2 occurrences 
Retrofitted embeddings: Retrofitting BIBREF16 has been shown as a simple but efficient way of informing trained embeddings with additional information derived from some lexical resource, rather than including it directly at the training stage, as it's done for example to create sense-aware BIBREF17 or sentiment-aware BIBREF18 embeddings  In this work, we retrofit general embeddings to include information about emotions, so that emotion-similar words can get closer in space  Both the Google as well as our Facebook embeddings were retrofitted with lexical information obtained from the NRC10 Lexicon mentioned above, which provides emotion-similarity for each token  Note that differently from the previous two types of embeddings, the retrofitted ones do rely on handcrafted information in the form of a lexical resource 
Results on development set
We report precision, recall, and f-score on the development set  The average f-score is reported as micro-average, to better account for the skewed distribution of the classes as well as in accordance to what is usually reported for this task BIBREF19  
From Table TABREF20 we draw three main observations  First, a simple tf-idf bag-of-word mode works already very well, to the point that the other textual and lexicon-based features don't seem to contribute to the overall f-score (0 368), although there is a rather substantial variation of scores per class  Second, Google embeddings perform a lot better than Facebook embeddings, and this is likely due to the size of the corpus used for training  Retrofitting doesn't seem to help at all for the Google embeddings, but it does boost the Facebook embeddings, leading to think that with little data, more accurate task-related information is helping, but corpus size matters most  Third, in combination with embeddings, all features work better than just using tf-idf, but removing the Lexicon feature, which is the only one based on hand-crafted resources, yields even better results  Then our best model (B-M) on development data relies entirely on automatically obtained information, both in terms of training data as well as features 
Results
In Table TABREF26 we report the results of our model on the three datasets standardly used for the evaluation of emotion classification, which we have described in Section SECREF3  
Our B-M model relies on subsets of Facebook pages for training, which were chosen according to their performance on the development set as well as on the observation of emotions distribution on different pages and in the different datasets, as described in Section SECREF4   The feature set we use is our best on the development set, namely all the features plus Google-based embeddings, but excluding the lexicon  This makes our approach completely independent of any manual annotation or handcrafted resource  Our model's performance is compared to the following systems, for which results are reported in the referred literature  Please note that no other existing model was re-implemented, and results are those reported in the respective papers 
Discussion, conclusions and future work
We have explored the potential of using Facebook reactions in a distant supervised setting to perform emotion classification  The evaluation on standard benchmarks shows that models trained as such, especially when enhanced with continuous vector representations, can achieve competitive results without relying on any handcrafted resource  An interesting aspect of our approach is the view to domain adaptation via the selection of Facebook pages to be used as training data 
We believe that this approach has a lot of potential, and we see the following directions for improvement  Feature-wise, we want to train emotion-aware embeddings, in the vein of work by tang:14, and iacobacci2015sensembed  Retrofitting FB-embeddings trained on a larger corpus might also be successful, but would rely on an external lexicon 
The largest room for yielding not only better results but also interesting insights on extensions of this approach lies in the choice of training instances, both in terms of Facebook pages to get posts from, as well as in which posts to select from the given pages  For the latter, one could for example only select posts that have a certain length, ignore posts that are only quotes or captions to images, or expand posts by including content from linked html pages, which might provide larger and better contexts BIBREF23   Additionally, and most importantly, one could use an entropy-based measure to select only posts that have a strong emotion rather than just considering the majority emotion as training label  For the former, namely the choice of Facebook pages, which we believe deserves the most investigation, one could explore several avenues, especially in relation to stance-based issues BIBREF24   In our dataset, for example, a post about Chile beating Colombia in a football match during the Copa America had very contradictory reactions, depending on which side readers would cheer for  Similarly, the very same political event, for example, would get very different reactions from readers if it was posted on Fox News or The Late Night Show, as the target audience is likely to feel very differently about the same issue  This also brings up theoretical issues related more generally to the definition of the emotion detection task, as it's strongly dependent on personal traits of the audience  Also, in this work, pages initially selected on availability and intuition were further grouped into sets to make training data according to performance on development data, and label distribution  Another criterion to be exploited would be vocabulary overlap between the pages and the datasets 
Lastly, we could develop single models for each emotion, treating the problem as a multi-label task  This would even better reflect the ambiguity and subjectivity intrinsic to assigning emotions to text, where content could be at same time joyful or sad, depending on the reader 
Acknowledgements
In addition to the anonymous reviewers, we want to thank Lucia Passaro and Barbara Plank for insightful discussions, and for providing comments on draft versions of this paper 

ground_truth: Android application