prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How large is the Twitter dataset?
Context 1: Introduction
Recently, people have started looking at online forums either as a primary or secondary source of counseling services BIBREF0  BIBREF1 reported that over the first five years of operation (2011-2016), ReachOut com – Ireland's online youth mental health service – 62% of young people would visit a website for support when going through a tough time  With the expansion of the Internet, there has been a substantial growth in the number of users looking for psychological support online 
The importance of the on-line life of patients has been recognized in research as well  BIBREF2 stated that the online life of patients constitutes a major influence on their self-definition  Furthermore, according to BIBREF3, the social networking activities of an individual, offer an important reflection of their personality  While dealing with patients suffering from psychological problems, it is important that therapists do not ignore this pivotal source of information which can provide deep insights into their patients' mental conditions 
Acceptance of on-line support groups (OSG) by Mental Health Professionals is still not established BIBREF4  Since OSG can have double-edged effects on patients and the presence of professionals is often limited, we argue that their properties should be further studied  According to BIBREF5 OSG effectiveness is hard to assess, while some studies showed OSG's potential to change participants' attitudes, no such effect was observed in other studies (see Related Work Section for more details)  Furthermore the scope of previous work on analysis of users' behaviour in OSG has been limited by the fact that they relied on expert annotation of posts and comments BIBREF6 
We present a novel approach for automatically analysing online conversations for the presence of therapeutic factors of group therapy defined by BIBREF7 as “the actual mechanisms of effecting change in the patient”  The authors have identified 11 therapeutic factors in group therapy: Universality, Altruism, Instillation of Hope, Guidance, Imparting information, Developing social skills, Interpersonal learning, Cohesion, Catharsis, Existential factors, Imitative behavior and Corrective recapitulation of family of origin issues  In this paper, we focus on 3 therapeutic factors: Universality, Altruism and Instillation of Hope (listed below), as we believe that these can be approximated by using established NLP techniques (e g  Sentiment Analysis, Dialogue Act tagging etc ) 
Universality: the disconfirmation of a user's feelings of uniqueness of their mental health condition 
Altruism: others offer support, reassurance, suggestions and insight 
Instillation of Hope: inspiration provided to participants by their peers 
The selected therapeutic factors are analysed in terms of illocutionary force and attitude  Due to the multi-party and asynchronous nature of on-line social media conversations, prior to the analysis, we extract conversation threads among users – an essential prerequisite for any kind of higher-level dialogue analysis BIBREF10  Afterwards, the illocutionary force is identified using Dialogue Act tagging, whereas the attitude by using Sentiment Analysis  The quantitative analysis is then performed on these processed conversations 
Ideally, the analysis would require experts to annotate each post and comment on the presence of therapeutic factors  However, due to time and cost demands of this task, it is feasible to analyse only a small fraction of the available data  Compared to previous studies (e g  BIBREF6) that analysed few tens of conversations and several thousand lines of chat; using the proposed approach – application of Dialogue Acts and Sentiment Analysis – we were able to automatically analyse approximately 300 thousands conversations (roughly 1 5 million comments)  The rest of the paper is structured as follows  In Section 2 we introduce related work  Next, in Section 3 we describe the pre-processing pipeline and the methodology to perform thread extraction on asynchronous multi-party conversations  In Section 4 we provide the describe the final dataset used for the analysis, and in Section 5 we present the results of our analysis  Finally, in Section 6 we provide concluding remarks and future research directions 
Related Work
On-line support groups have been analyzed for various factors before  For instance, BIBREF11 analysed stress reduction in on-line support group chat-rooms, and the effects of on-line social interactions  Such studies mostly relied on questionnaires and were based on a small number of users  Nevertheless, in BIBREF11, the author showed that social support facilitates coping with distress, improves mood and expedites recovery from it  These findings highlight that, overall, on-line discussion boards appear to be therapeutic and constructive for individuals suffering alcohol-abuse 
Application of NLP to the analysis of mental health-related conversation has been studied as well (e g  BIBREF12, BIBREF13)  BIBREF6 applied sentiment-analysis combined with extensive turn-level annotation to investigate stress reduction in on-line support group chat-rooms, showing that sentiment-analysis is a good predictor of entrance stress level  Furthermore, similar to our setting, they applied automatic thread-extraction to determine conversation threads 
BIBREF14 have shown that on-line support group therapy increased the quality of life of patients with metastatic breast cancer  Since many original posters reported the benefits of group therapy on patients BIBREF15, BIBREF2, BIBREF16, BIBREF17, BIBREF18, BIBREF7, we evaluate the effect of the user interaction using sentiment scores of comments in on-line support groups 
According to BIBREF6, users with high incoming stress tend to request less information from others, as a percentage of their time, and share much more information, in absolute terms  In addition, high information sharing has been shown to be a good predictor of stress reduction at the end of the chat BIBREF6  Regarding information sharing, we rely on Dialogue Acts BIBREF19 to model the speaker's intention in producing an utterance  In particular, we are interested in Dialogue Act label that is defined to represent descriptive, narrative, or personal information – the statement 
Dialogue Acts have been applied to the analysis of spoken BIBREF20, BIBREF21 as well as on-line written synchronous conversations BIBREF22  We apply Dialogue Act tag set defined in BIBREF22 to the analysis of our on-line asynchronous conversations  We argue that Dialogue Acts can be used to analyse user behaviour in social media and verify the presence of therapeutic factors 
Methodology
We select the three therapeutic factors – Universality, Altruism and Instillation of Hope – that can be best approximated using NLP techniques: Sentiment Analysis and Dialogue Act tagging  We discuss each one of the selected therapeutic factors and the identified necessary conditions  The listed conditions, however, are not sufficient to attribute the presence of a therapeutic factor with high confidence, which only can be obtained using expert annotation  Our analysis focuses on the structure of conversations; though content plays an important role as well 
Universality consists in the disconfirmation of patients' belief of uniqueness of their disease  This therapeutic factor is shown to be a powerful source of relief for the patient, according to BIBREF7  From this definition, we can draw the following conditions that are applicable to our environment:
improvement of original poster's sentiment: we hypothesize that the discovery that other people passed through similar issues leads to a higher sentiment score;
posts containing negative personal experiences: to disconfirm the belief of uniqueness users have to share their story;
comments containing negative statements: to disconfirm the patient's feelings of uniqueness, the commenting user must tell a similar negative personal experience  This condition requires two sub-conditions: high presence of statements in comments and the presence of negative comments replying to negative posts 
Instillation of Hope is based on inspiration provided to participants by their peers  Through the inspiration provided by their peers, patients can increase their expectation on the therapy outcome  BIBREF7 in several studies have demonstrated that a high expectation of help before the start of a therapy is significantly correlated with a positive therapy outcome  The author states that many patients pointed out the importance of having observed the improvement of others  Therefore, the three main conditions are the following:
improvement of original poster's sentiment: we hypothesize that instillation of hope leads to a higher sentiment score;
posts containing negative personal experiences: hope can be instilled in someone who shares a negative personal experience;
comments containing positive personal experiences: in order to instill hope, commenting posters must show to original posters an overall positive personal experience  To detect positive personal experience, we require the presence of statements in comments and a positive sentiment of comments replying to negative posts 
Altruism consists of peers offering support, reassurance, suggestions and insight, since they share similar problems with one another BIBREF7  The experience of finding that a patient can be of value to others is refreshing and boosts self-esteem BIBREF7  However, in the current study we focus on testing whether commenting posters are altruists or not  We do not test whether the altruistic behavior leads to an improvement on the altruist itself  For these reasons, we define three main conditions:
improvement of original poster's sentiment: we hypothesize that supportive and reassuring statements improve the sentiment score of the original poster;
posts contains negative personal experiences: users offer support, reassurance and suggestion when facing a negative personal experience of the original poster;
comments containing positive statements: either supportive or reassuring statements show by definition a positive intended emotional communication  Thus comments to the post should consist of positive sentiment statements 
Consequently, a conversation containing the aforementioned therapeutic factors should satisfy the following conditions in terms of NLP: Sentiment Analysis and Dialogue Acts 
original posters have a higher sentiment score at the end of the thread than at the beginning;
the original post consists mostly of polarised statements;
the presence of a significant amount of statements in comments, since both support and sharing similar negative experiences can be represented as statements;
both negative and positive statements in comments lead to higher final sentiment score of the original poster 
Datasets
We verify the presence of therapeutic factors in two social media datasets: OSG and Twitter  The first dataset is crawled from an on-line support groups website, and the second dataset consists of a small sample of Twitter conversation threads  Since the former consists of multi-threaded conversations, we apply a pre-processing to extract conversation threads to provide a fair comparison with the Twitter dataset  An example conversation from each data source is presented in Figure FIGREF19 
Datasets ::: Twitter
We have downloaded 1,873 Twitter conversation threads, roughly 14k tweets, from a publicly available resource that were previously pre-processed and have conversation threads extracted  A conversation in the dataset consists of at least 4 tweets  Even though, according to BIBREF23, Twitter is broadly applicable to public health research, our expectation is that it contains less therapeutic conversations in comparison to specialized on-line support forums 
Datasets ::: OSG
Our data has been developed by crawling and pre-processing an OSG web forum  The forum has a great variety of different groups such as depression, anxiety, stress, relationship, cancer, sexually transmitted diseases, etc  Each conversation starts with one post and can contain multiple comments  Each post or comment is represented by a poster, a timestamp, a list of users it is referencing to, thread id, a comment id and a conversation id  The thread id is the same for comments replying to each other, otherwise it is different  The thread id is increasing with time  Thus, it provides ordering among threads; whereas the timestamp provides ordering in the thread 
Each conversation can belong to multiple groups  Consequently, the dataset needs to be processed to remove duplicates  The dataset resulting after de-duplication contains 295 thousand conversations, each conversation contains on average 6 comments  In total, there are 1 5 million comments  Since the created dataset is multi-threaded, we need to extract conversation threads, to eliminate paths not relevant to the original post 
Datasets ::: OSG ::: Conversation Thread Extraction
The thread extraction algorithm is heuristic-based and consists of two steps: (1) creation of a tree, based on a post written by a user and the related comments and (2) transformation of the tree into a list of threads 
The tree creation is an extension of the approach of BIBREF24, where first a graph of conversation is constructed  In the approach, direct replies to a post are attached to the first nesting level and subsequent comments to increasing nesting levels  In our approach, we also exploit comments' features 
The tree creation is performed without processing the content of comments, which allows us to process posts and comments of any length efficiently  The heuristic used in the process is based on three simplifying assumptions:
Unless there is a specific reference to another comment or a user, comments are attached to the original post 
When replying, the commenting poster is always replying to the original post or some other comment  Unless specified otherwise, it is assumed that it is a response to the previous (in time) post/comment 
Subsequent comments by the same poster are part of the same thread 
To evaluate the performance of the thread extraction algorithm, 2 annotators have manually constructed the trees for 100 conversations  The performance of the algorithm on this set of 100 conversations is evaluated using accuracy and standard Information Retrieval evaluation metrics of precision, recall, and F$_1$ measure  The results are reported in Table TABREF28 together with random and majority baselines  The turn-level percent agreement between the 2 annotators is 97 99% and Cohen's Kappa Coefficient is 83 80% 
Datasets ::: Data Representation
For both data sources, Twitter and OSG with extracted threads, posts and comments are tokenized and sentence split  Each sentence is passed through Sentiment Analysis and Dialogue Act tagging  Since a post or a comment can contain multiple sentences, therefore multiple Dialogue Acts, it is represented as as a one-hot encoding, where each position represents a Dialogue Act 
For Sentiment Analysis we use a lexicon-based sentiment analyser introduced by BIBREF25  For Dialogue Act tagging, on the other hand, we make use of a model trained on NPSChat corpus BIBREF22 following the approach of BIBREF26 
Analysis
As we mentioned in Section 3, the presence of each of the therapeutic conditions under analysis is a necessary for a conversation to be considered to have therapeutic factors  In this section we present the results of our analysis with respect to these conditions 
Analysis ::: Change in Sentiment score of Original Posters
The first condition which we test is the sentiment change in conversation threads, comparing the initial and final sentiment scores (i e  posts' scores) of the original poster  The results of the analysis are presented in Figure FIGREF33  In the figure we can observe that the distribution of the sentiment change in the two datasets is different  While in Twitter the amount of conversations that lead to the increase of sentiment score is roughly equal to the amount of conversations that lead to the decrease of sentiment score; the situation is different for OSG  In OSG, the amount of conversations that lead to the increase of sentiment score is considerably higher 
Figure FIGREF34 provides a more fine grained analysis, where we additionally analyse the sentiment change in nominal polarity terms – negative and positive  In OSG, the number of users that changed polarity from negative to positive is more than the double of the users that have changed the polarity from positive to negative  In Twitter, on the other hand, the users mostly changed polarity from positive to negative  Results of the analysis suggest that in OSG , sentiment increases and users tend to change polarity from negative to positive, whereas in Twitter sentiment tends to decrease  Verification of this condition alone indicates that the ratio of potentially therapeutic conversations in Twitter is lower 
Analysis ::: Structure of Posts and Comments
Table TABREF36 presents the distribution of automatically predicted per-sentence Dialogue Acts in the datasets  The most frequent tag is statement in both  In Table TABREF37, on the other hand, we present the distribution of post and comment structures in terms of automatically predicted Dialogue Act tags  The structure is an unordered set of tags in the post or comment  From the table we can observe that the distribution of tag sets is similar between posts and comments  In both cases the most common set is statement only  However, conversations containing only statement, emphasis or question posts and comments predominantly appear in Twitter  Which is expected due to the shorter length of Twitter posts and comments 
We can also observe that the original posters tend to ask more questions than the commenting posters – 19 83% for posts vs  11 21% for comments (summed)  This suggests that the original posters frequently ask either for suggestion or confirmation of their points of view or their disconfirmation  However, the high presence of personal experiences is supported by the high number of posts containing only statements 
High number of statement tags in comments suggests that users reply either with supporting or empathic statements or personal experience  However, 6 39% of comments contain accept and reject tags, which mark the degree to which a speaker accepts some previous proposal, plan, opinion, or statement BIBREF20  The described Dialogue Act tags are often used when commenting posters discuss original poster's point of view  For instance, “It's true  I felt the same ” – {Accept, Statement} or “Well no  You're not alone” – {Reject, Statement}  The datasets differ with respect to the distribution of these Dialogue Acts tags, they appear more frequently in OSG 
Analysis ::: Sentiment of Posts and Comments
Table TABREF39 presents the distribution of sentiment polarity in post and comment statements (i e  sentences tagged as statement)  For OSG, the predominant sentiment label of statements is positive and it is the highest for both posts and comments  However, the difference between the amounts of positive and negative statements is higher for the replying comments (34 5% vs  42 5%)  For Twitter, on the other hand, the predominant sentiment label of statements is neutral and the polarity distribution between posts and comments is very close  One particular observation is that the ratio of negative statements is higher in OSG for both posts and comments than in Twitter, which supports the idea of sharing negative experiences 
Further we analyze whether the sentiment of a comment (i e  the replying user) is affected by the sentiment of the original post (i e  the user being replied to), which will imply that the users adapt their behaviour with respect to the post's sentiment  For the analysis, we split the datasets into three buckets according to the posts' sentiment score – negative, neutral, or positive, and represent each conversation in terms of percentages of comments (replies) with each sentiment label  The buckets are then compared using t-test for statistically significant differences 
Table TABREF40 presents the distribution of sentiment labels with respect to the post's sentiment score  The patterns of distribution are similar across the datasets  We can observe that overall, replies tend to have a positive sentiment, which suggests that replying posters tend to have a positive attitude  However, the ratio of positive comments is higher for OSG than for Twitter 
The results of the Welch's t-test on OSG data reveal that there are statistically significant differences in the distribution of replying comments' sentiment between conversations with positive and negative starting posts  A positive post tends to get significantly more positive replies  Similarly, a negative post tends to get significantly more negative replies (both with $p < 0 01$) 
Table TABREF41 presents the distribution of the sentiment labels of the final text provided by the original poster with respect to the sentiment polarity of the comments  The results indicate that OSG participants are more supportive, as the majority of conversations end in a positive final sentiment regardless of the sentiment of comments  We can also observe that negative comments in OSG lead to positive sentiment, which supports the idea of sharing the negative experiences, thus presence of therapeutic factors  For Twitter, on the other hand, only positive comments lead to the positive final sentiments, whereas other comments lead predominantly to neutral final sentiments 
Our analysis in terms of sentiment and Dialogue Acts supports the presence of the three selected therapeutic factors – Universality, Altruism and Instillation of Hope – in OSG more than in Twitter  The main contributors to this conclusion are the facts that there is more positive change in the sentiment of the original posters in OSG (people seeking support) and that in OSG even negative and neutral comments are likely to lead to positive changes 
Conclusion
In this work, we propose a methodology to automatically analyse online social platforms for the presence of therapeutic factors (i e  Universality, Altruism and Instillation of Hope)  We evaluate our approach on two on-line platforms, Twitter and an OSG web forum  We apply NLP techniques of Sentiment Analysis and Dialogue Act tagging to automatically verify the presence of therapeutic factors, which allows us to analyse larger amounts of conversational data (as compared to previous studies) 
Our analysis indicates that OSG conversations satisfy higher number of conditions approximating therapeutic factors than Twitter conversations  Given this outcome, we postulate that users who join support group websites spontaneously seem to benefit from it  Indeed, as shown in Section SECREF5, the original posters who interact with others by replying to comments, have benefited from an improvement of their emotional state 
We would like to reemphasise that the conditions for the therapeutic factors are necessary but not sufficient; since our analysis focuses on the structure of conversations, being agnostic to the content  NLP, however, allows us to strengthen our approximations even further  Thus, the further extension of our work is also augmentation of our study with other language analysis metrics and their correlation with human annotation 
It should be noted that the proposed approach is an approximation of the tedious tasks of annotation of conversations by experts versed in the therapeutic factors and their associated theories  Even though we can use Sentiment Analysis to detect the existence of therapeutic factors, we cannot differentiate between Altruism and Instillation of Hope, as this requires differentiation between emotional state of the user and the intended emotional communication  Thus, the natural extensions of this work are differentiation between different therapeutic factors and comparison of the proposed analysis to the human evaluation 
Although we acknowledge that the proposed methodology does not serve as a replacement of manual analysis of OSG for the presence of therapeutic factors, we believe that it could facilitate and supplement this process  The method can serve as a tool for general practitioners and psychologists who can use it as an additional source of information regarding their patients’ condition and, in turn, offer a more personalised support that is better tailored to individual therapeutic needs 
Context 2: Introduction
Deep Learning approaches have achieved impressive results on various NLP tasks BIBREF0 , BIBREF1 , BIBREF2 and have become the de facto approach for any NLP task  However, these deep learning techniques have found to be less effective for low-resource languages when the available training data is very less BIBREF3   Recently, several approaches like Multi-task learning BIBREF4 , multilingual learning BIBREF5 , semi-supervised learning BIBREF2 , BIBREF6 and transfer learning BIBREF7 , BIBREF3 have been explored by the deep learning community to overcome data sparsity in low-resource languages  Transfer learning trains a model for a parent task and fine-tunes the learned parent model weights (features) for a related child task BIBREF7 , BIBREF8   This effectively reduces the requirement on training data for the child task as the model would have learned relevant features from the parent task data thereby, improving the performance on the child task 
Transfer learning has also been explored in the multilingual Neural Machine Translation BIBREF3 , BIBREF9 , BIBREF10   The goal is to improve the NMT performance on the source to target language pair (child task) using an assisting source language (assisting to target translation is the parent task)  Here, the parent model is trained on the assisting and target language parallel corpus and the trained weights are used to initialize the child model  The child model can now be fine-tuned on the source-target language pairs, if parallel corpus is available  The divergence between the source and the assisting language can adversely impact the benefits obtained from transfer learning  Multiple studies have shown that transfer learning works best when the languages are related BIBREF3 , BIBREF10 , BIBREF9   Several studies have tried to address lexical divergence between the source and the target languages BIBREF10 , BIBREF11 , BIBREF12   However, the effect of word order divergence and its mitigation has not been explored  In a practical setting, it is not uncommon to have source and assisting languages with different word order  For instance, it is possible to find parallel corpora between English and some Indian languages, but very little parallel corpora between Indian languages  Hence, it is natural to use English as an assisting language for inter-Indian language translation 
To see how word order divergence can be detrimental, let us consider the case of the standard RNN (Bi-LSTM) encoder-attention-decoder architecture BIBREF13   The encoder generates contextual representations (annotation vectors) for each source word, which are used by the attention network to match the source words to the current decoder state  The contextual representation is word-order dependent  Hence, if the assisting and the source languages do not have similar word order the generated contextual representations will not be consistent  The attention network (and hence the decoder) sees different contextual representations for similar words in parallel sentences across different languages  This makes it difficult to transfer knowledge learned from the assisting language to the source language 
We illustrate this by visualizing the contextual representations generated by the encoder of an English to Hindi NMT system for two versions of the English input: (a) original word order (SVO) (b) word order of the source language (SOV, for Bengali)  Figure FIGREF1 shows that the encoder representations obtained are very different  The attention network and the decoder now have to work with very different representations  Note that the plot below does not take into account further lexical and other divergences between source and assisting languages, since we demonstrated word order divergence with the same language on the source side 
To address this word order divergence, we propose to pre-order the assisting language sentences to match the word order of the source language  We consider an extremely resource constrained scenario, where we do not have any parallel corpus for the child task  We are limited to a bilingual dictionary for transfer information from the assisting to the source language  From our experiments, we show that there is a significant increase in the translation accuracy for the unseen source-target language pair 
Addressing Lexical Divergence
BIBREF3 explored transfer learning for NMT on low-resource languages  They studied the influence of language divergence between languages chosen for training the parent and child model, and showed that choosing similar languages for training the parent and child model leads to better improvements from transfer learning  A limitation of BIBREF3 approach is that they ignore the lexical similarity between languages and also the source language embeddings are randomly initialized  BIBREF10 , BIBREF11 , BIBREF12 take advantage of lexical similarity between languages in their work  BIBREF10 proposed to use Byte-Pair Encoding (BPE) to represent the sentences in both the parent and the child language to overcome the above limitation  They show using BPE benefits transfer learning especially when the involved languages are closely-related agglutinative languages  Similarly, BIBREF11 utilize lexical similarity between the source and assisting languages by training a character-level NMT system  BIBREF12 address lexical divergence by using bilingual embeddings and mixture of universal token embeddings  One of the languages' vocabulary, usually English vocabulary is considered as universal tokens and every word in the other languages is represented as a mixture of universal tokens  They show results on extremely low-resource languages 
Addressing Word Order Divergence
To the best of our knowledge, no work has addressed word order divergence in transfer learning for multilingual NMT  However, some work exists for other NLP tasks that could potentially address word order  For Named Entity Recognition (NER), BIBREF14 use a self-attention layer after the Bi-LSTM layer to address word-order divergence for Named Entity Recognition (NER) task  The approach does not show any significant improvements over multiple languages  A possible reason is that the divergence has to be addressed before/during construction of the contextual embeddings in the Bi-LSTM layer, and the subsequent self-attention layer does not address word-order divergence  BIBREF15 use adversarial training for cross-lingual question-question similarity ranking in community question answering  The adversarial training tries to force the encoder representations of similar sentences from different input languages to have similar representations 
Use of Pre-ordering
Pre-ordering the source language sentences to match the target language word order has been useful in addressing word-order divergence for Phrase-Based SMT BIBREF16 , BIBREF17 , BIBREF18 , BIBREF19   Recently, BIBREF20 proposed a way to measure and reduce the divergence between the source and target languages based on morphological and syntactic properties, also termed as anisomorphism  They demonstrated that by reducing the anisomorphism between the source and target languages, consistent improvements in NMT performance were obtained  The NMT system used additional features like word forms, POS tags and dependency relations in addition to parallel corpora  On the other hand, BIBREF21 observed a drop in performance due to pre-ordering for NMT  Unlike BIBREF20 , the NMT system was trained on pre-ordered sentences and no additional features were provided to the system  Note that all these works address source-target divergence, not divergence between source languages in multilingual NMT 
Proposed Solution
Consider the task of translating from an extremely low-resource language (source) to a target language  The parallel corpus between the two languages if available may be too small to train a NMT model  Similar to existing works BIBREF3 , BIBREF10 , BIBREF12 , we use transfer learning to overcome data sparsity and train a NMT model between the source and the target languages
Context 3: Introduction
Recent years have seen unprecedented progress for Natural Language Processing (NLP) on almost every NLP subtask  Even though low-resource settings have also been explored, this progress has overwhelmingly been observed in languages with significant data resources that can be leveraged to train deep neural networks  Low-resource languages still lag behind 
Endangered languages pose an additional challenge  The process of documenting an endangered language typically includes the creation of word lists, audio and video recordings, notes, or grammar fragments, with the created resources then stored in large online linguistics archives  This process is often hindered by the Transcription Bottleneck: the linguistic fieldworker and the language community may not have time to transcribe all of the recordings and may only transcribe segments that are linguistically salient for publication or culturally significant for the creation of community resources 
With this work we make publicly available a large corpus in Mapudungun, a language of the indigenous Mapuche people of southern Chile and western Argentina  We hope to ameliorate the resource gap and the transcription bottleneck in two ways  First, we are providing a larger data set than has previously been available, and second, we are providing baselines for NLP tasks (speech recognition, speech synthesis, and machine translation)  In providing baselines and datasets splits, we hope to further facilitate research on low-resource NLP for this language through our data set  Research on low-resource speech recognition is particularly important in relieving the transcription bottleneck, while tackling the research challenges that speech synthesis and machine translation pose for such languages could lead to such systems being deployed to serve more under-represented communities 
The Mapudungun Language
Mapudungun (iso 639-3: arn) is an indigenous language of the Americas spoken natively in Chile and Argentina, with an estimated 100 to 200 thousand speakers in Chile and 27 to 60 thousand speakers in Argentina BIBREF0  It is an isolate language and is classified as threatened by Ethnologue, hence the critical importance of all documentary efforts  Although the morphology of nouns is relatively simple, Mapudungun verb morphology is highly agglutinative and complex  Some analyses provide as many as 36 verb suffix slots BIBREF1  A typical complex verb form occurring in our corpus of spoken Mapudungun consists of five or six morphemes 
Mapudungun has several interesting grammatical properties  It is a polysynthetic language in the sense of BIBREF2; see BIBREF3 for explicit argumentation  As with other polysynthetic languages, Mapudungun has Noun Incorporation; however, it is unique insofar as the Noun appears to the right of the Verb, instead of to the left, as in most polysynthetic languages BIBREF4  One further distinction of Mapudungun is that, whereas other polysynthetic languages are characterized by a lack of infinitives, Mapudungun has infinitival verb forms; that is, while subordinate clauses in Mapudungun closely resemble possessed nominals and may occur with an analytic marker resembling possessor agreement, there is no agreement inflection on the verb itself  One further remarkable property of Mapudungun is its inverse voice system of agreement, whereby the highest agreement is with the argument highest in an animacy hierarchy regardless of thematic role BIBREF5 
The Resource
The resource is comprised of 142 hours of spoken Mapudungun that was recorded during the AVENUE project BIBREF6 in 2001 to 2005  The data was recorded under a partnership between the AVENUE project, funded by the US National Science Foundation at Carnegie Mellon University, the Chilean Ministry of Education (Mineduc), and the Instituto de Estudios Indígenas at Universidad de La Frontera, originally spanning 170 hours of audio  We have recently cleaned the data and are releasing it publicly for the first time (although it has been shared with individual researchers in the past) along with NLP baselines 
The recordings were transcribed and translated into Spanish at the Instituto de Estudios Indígenas at Universidad de La Frontera  The corpus covers three dialects of Mapudungun: about 110 hours of Nguluche, 20 hours of Lafkenche and 10 hours of Pewenche  The three dialects are quite similar, with some minor semantic and phonetic differences  The fourth traditionally distinguished dialect, Huilliche, has several grammatical differences from the other three and is classified by Ethnologue as a separate language, iso 639-3: huh, and as nearly extinct 
The recordings are restricted to a single domain: primary, preventive, and treatment health care, including both Western and Mapuche traditional medicine  The recording sessions were conducted as interactive conversations so as to be natural in Mapuche culture, and they were open-ended, following an ethnographic approach  The interviewer was trained in these methods along with the use of the digital recording systems that were available at the time  We also followed human subject protocol  Each person signed a consent form to release the recordings for research purposes and the data have been accordingly anonymized  Because Machi (traditional Mapuche healers) were interviewed, we asked the transcribers to delete any culturally proprietary knowledge that a Machi may have revealed during the conversation  Similarly, we deleted any names or any information that may identify the participants 
The corpus is culturally relevant because it was created by Mapuche people, using traditional ways of relating to each other in conversations  They discussed personal experiences with primary health care in the traditional Mapuche system and the Chilean health care system, talking about illnesses and the way they were cured  The participants ranged from 16 years old to 100 years old, almost in equal numbers of men and women, and they were all native speakers of Mapudungun 
The Resource ::: Orthography
At the time of the collection and transcription of the corpus, the orthography of Mapudungun was not standardized  The Mapuche team at the Instituto de Estudios Indígenas (IEI – Institute for Indigenous Studies) developed a supra-dialectal alphabet that comprises 28 letters that cover 32 phones used in the three Mapudungun variants  The main criterion for choosing alphabetic characters was to use the current Spanish keyboard that was available on all computers in Chilean offices and schools  The alphabet used the same letters used in Spanish for those phonemes that sound like Spanish phonemes  Diacritics such as apostrophes were used for sounds that are not found in Spanish 
As a result, certain orthographic conventions that were made at the time deviate from the now-standard orthography of Mapudungun, Azumchefe  We plan to normalize the orthography of the corpus, and in fact a small sample has already been converted to the modern orthography  However, we believe that the original transcriptions will also be invaluable for academic, historical, and cultural purposes, hence we release the corpus using these conventions 
The Resource ::: Additional Annotations
In addition, the transcription includes annotations for noises and disfluencies including aborted words, mispronunciations, poor intelligibility, repeated and corrected words, false starts, hesitations, undefined sound or pronunciations, non-verbal articulations, and pauses  Foreign words, in this case Spanish words, are also labelled as such 
The Resource ::: Cleaning
The dialogues were originally recorded using a Sony DAT recorder (48kHz), model TCD-D8, and Sony digital stereo microphone, model ECM-DS70P  Transcription was performed with the TransEdit transcription tool v 1 1 beta 10, which synchronizes the transcribed text and the wave files 
However, we found that a non-trivial number of the utterance boundaries and speaker annotations were flawed  Also some recording sessions did not have a complete set of matching audio, transcription, and translation files  Hence, in an effort to provide a relatively “clean" corpus for modern computational experiments, we converted the encoding of the textual transcription from Latin-1 to Unicode, DOS to UNIX line endings, a now more standard text encoding format than what was used when the data was first collected  Additionally, we renamed a small portion of files which had been misnamed and removed several duplicate files 
Although all of the data was recorded with similar equipment in relatively quiet environments, the acoustics are not as uniform as we would like for building speech synthesizers  Thus we applied standardized power normalization  We also moved the boundaries of the turns to standardize the amount of leading and trailing silence in each turn  This is a standard procedure for speech recognition and synthesis datasets  Finally we used the techniques in BIBREF7 for found data to re-align the text to the audio and find out which turns are best (or worst) aligned so that we can select segments that give the most accurate alignments  Some of the misalignments may in part be due to varied orthography, and we intend, but have not yet, to investigate normalization of orthography (i e  spelling correction) to mitigate this 
The Resource ::: Training, Dev, and Test Splits
We create two training sets, one appropriate for single-speaker speech synthesis experiments, and one appropriate for multiple-speaker speech recognition and machine translation experiments  In both cases, our training, development, and test splits are performed at the dialogue level, so that all examples from each dialogue belong to exactly one of these sets 
For single-speaker speech synthesis, we only use the dialog turns of the speaker with the largest volume of data (nmlch – one of the interviewers)  The training set includes $221 8$ thousand sentences from 285 dialogues, with 12 and 46 conversations reserved for the development and test set 
For speech recognition experiments, we ensure that our test set includes unique speakers as well as speakers that overlap with the training set, in order to allow for comparisons of the ability of the speech recognition system to generalize over seen and new speakers  For consistency, we use the same dataset splits for the machine translation experiments  The statistics in Table reflect this split 
Applications
Our resource has the potential to be the basis of computational research in Mapudungun across several areas  Since the collected audio has been transcribed, our resource is appropriate for the study of automatic speech recognition and speech synthesis  The Spanish translations enable the creation of machine translation systems between Mapudungun and Spanish, as well as end-to-end (or direct) speech translation  We in fact built such speech synthesis, speech recognition, and machine translation systems as a showcase of the usefulness of our corpus in that research direction 
Furthermore, our annotations of the Spanish words interspersed in Mapudungun speech could allow for a study of code-switching patterns within the Mapuche community  In addition, our annotations of non-standardized orthographic transcriptions could be extremely useful in the study of historical language and orthography change as a language moves from predominantly oral to being written in a standardized orthography, as well as in building spelling normalization and correction systems  The relatively large amount of data that we collected will also allow for the training of large language models, which in turn could be used as the basis for predictive keyboards tailored to Mapudungun  Last, since all data are dialogues annotated for the different speaker turns, they could be useful for building Mapudungun dialogue systems and chatbot-like applications 
The potential applications of our resource, however, are not exhausted in language technologies  The resource as a whole could be invaluable for ethnographic and sociological research, as the conversations contrast traditional and Western medicine practices, and they could reveal interesting aspects of the Mapuche culture 
In addition, the corpus is a goldmine of data for studying the morphostyntax of Mapudungun BIBREF8  As an isolate polysynthetic language, the study of Mapudungun can provide insights into the range of possibilities within human languages can work 
Baseline Results
Using the aforementioned higher quality portions of the corpus, we trained baseline systems for Mapudungun speech recognition and speech synthesis, as well as Machine Translation systems between Mapudungun and Spanish 
Baseline Results ::: Speech Synthesis
In our previous work on building speech systems on found data in 700 languages, BIBREF7, we addressed alignment issues (when audio is not segmented into turn/sentence sized chunks) and correctness issues (when the audio does not match the transcription)  We used the same techniques here, as described above 
For the best quality speech synthesis we need a few hours of phonetically-balanced, single-speaker, read speech  Our first step was to use the start and end points for each turn in the dialogues, and select those of the most frequent speaker, nmlch  This gave us around 18250 segments  We further automatically removed excessive silence from the start, middle and end of these turns (based on occurrence of F0)  This gave us 13 hours and 48 minutes of speech 
We phonetically aligned this data and built a speech clustergen statistical speech synthesizer BIBREF9 from all of this data  We resynthesized all of the data and measured the difference between the synthesized data and the original data using Mel Cepstral Distortion, a standard method for automatically measuring quality of speech generation BIBREF10  We then ordered the segments by their generation score and took the top 2000 turns to build a new synthesizer, assuming the better scores corresponded to better alignments, following the techniques of BIBREF7 
The initial build gave an MCD on held out data of 6 483  While the 2000 best segment dataset gives an MCD of 5 551, which is a large improvement  The quality of the generated speech goes from understandable, only if you can see the text, to understandable, and transcribable even for non-Mapudungun speakers  We do not believe we are building the best synthesizer with our current (non-neural) techniques, but we do believe we are selecting the best training data for other statistical and neural training techniques in both speech synthesis and speech recognition 
Baseline Results ::: Speech Recognition
For speech recognition (ASR) we used Kaldi BIBREF11  As we do not have access to pronunciation lexica for Mapudungun, we had to approximate them with two settings  In the first setting, we make the simple assumption that each character corresponds to a pronunced phoneme  In the second setting, we instead used the generated phonetic lexicon also used in the above-mentioned speech synthesis techniques  The train/dev/test splits are across conversations, as described above 
Under the first setting, we obtained a 60% character error rate, while the generated lexicon significantly boosts performance, as our systems achieve a notably reduced 30% phone error rate  Naturally, these results are relatively far from the quality of ASR systems trained on large amounts of clean data such as those available in English  Given the quality of the recordings, and the lack of additional resources, we consider our results fairly reasonable and they would still be usable for simple dialog-like tasks  We anticipate, though, that one could significantly improve ASR quality over our dataset, by using in-domain language models, or by training end-to-end neural recognizers leveraging languages with similar phonetic inventories BIBREF12 or by using the available Spanish translations in a multi-source scenario BIBREF13 
Baseline Results ::: Mapudungun–Spanish Machine Translation
We built neural end-to-end machine translation systems between Mapudungun and Spanish in both directions, using state-of-the-art Transformer architecture BIBREF14 with the toolkit of BIBREF15  We train our systems at the subword level using Byte-Pair Encoding BIBREF16 with a vocabulary of 5000 subwords, shared between the source and target languages  We use five layers for each of the encoder and the decoder, an embedding size of 512, feed forward transformation size of 2048, and eight attention heads  We use dropout BIBREF17 with $0 4$ probability as well as label smoothing set to $0 1$  We train with the Adam optimizer BIBREF18 for up to 200 epochs using learning decay with a patience of six epochs 
The baseline results using different portions of the training set (10k, 50k, 100k, and all (220k) parallel sentences) on both translation directions are presented in Table , using detokenized BLEU BIBREF19 (a standard MT metric) and chrF BIBREF20 (a metric that we consider to be more appropriate for polysynthetic languages, as it does not rely on word n-grams) computed with the sacreBLEU toolkit BIBREF21  It it worth noting the difference in quality between the two directions, with translation into Spanish reaching 20 4 (almost 21) BLEU points in the development set, while the opposite direction (translating into Mapudungun) shows about a 7 BLEU points worse performance  This is most likely due to Mapudungun being a polysynthetic language, with its complicated morphology posing a challenge for proper generation 
Related Work
Mapudungun grammar has been studied since the arrival of European missionaries and colonizers hundreds of years ago  More recent descriptions of Mapudungun grammar BIBREF1 and BIBREF0 informed the collection of the resource that we are presenting in this paper 
Portions of our resource have been used in early efforts to build language systems for Mapudungun  In particular, BIBREF22 focused on Mapudungun morphology in order to create spelling correction systems, while BIBREF23, BIBREF6, BIBREF24, and BIBREF25 developed hybrid rule- and phrase-based Statistical Machine Translation systems 
Naturally, similar works in collecting corpora in Indigenous languages of Latin America are abundant, but very few, if any, have the scale and potential of our resource to be useful in many downstream language-specific and inter-disciplinary applications  A general overview of the state of NLP for the under-represented languages of the Americas can be found at BIBREF26  To name a few of the many notable works, BIBREF27 created a parallel Mixtec-Spanish corpus for Machine Translation and BIBREF28 created lexical resources for Arapaho, while BIBREF29 and BIBREF30 focused on building speech corpora for Southern Quechua and Chatino respectively 
Conclusion
With this work we present a resource that will be extremely useful for building language systems in an endangered, under-represented language, Mapudungun  We benchmark NLP systems for speech synthesis, speech recognition, and machine translation, providing strong baseline results  The size of our resource (142 hours, more than 260k total sentences) has the potential to alleviate many of the issues faced when building language technologies for Mapudungun, in contrast to other indigenous languages of the Americas that unfortunately remain low-resource 
Our resource could also be used for ethnographic and anthropological research into the Mapuche culture, and has the potential to contribute to intercultural bilingual education, preservation activities and further general advancement of the Mapudungun-speaking community 
Acknowledgements
The data collection described in this paper was supported by NSF grants IIS-0121631 (AVENUE) and IIS-0534217 (LETRAS), with supplemental funding from NSF's Office of International Science and Education  Preliminary funding for work on Mapudungun was also provided by DARPA The experimental material is based upon work generously supported by the National Science Foundation under grant 1761548 

ground_truth: 1,873 Twitter conversation threads, roughly 14k tweets

1,873 Twitter conversation threads, roughly 14k tweets

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How do they match words before reordering them?
Context 2:  Specifically, the NMT model (parent model) is trained on the assisting language and target language pairs  We choose English as the assisting language in all our experiments  In our resource-scarce scenario, we have no parallel corpus for the child task  Hence, at test time, the source language sentence is translated using the parent model after performing a word-by-word translation into the assisting language 
Since the source language and the assisting language (English) have different word order, we hypothesize that it leads to inconsistencies in the contextual representations generated by the encoder for the two languages  In this paper, we propose to pre-order English sentences (assisting language sentences) to match the word-order of the source language and train the parent model on this pre-ordered corpus  In our experiments, we look at scenarios where the assisting language has SVO word order and the source language has SOV word order 
For instance, consider the English sentence Anurag will meet Thakur  One of the pre-ordering rule swaps the position of the noun phrase followed by a transitive verb with the transitive verb  The original and the resulting re-ordered parse tree will be as shown in the Table TABREF5   Applying this reordering rule to the above sentence Anurag will meet Thakur will yield the reordered sentence Anurag Thakur will meet  Additionally, the Table TABREF5 shows the parse trees for the above sentence with and without pre-ordering 
Pre-ordering should also be beneficial for other word order divergence scenarios (e g , SOV to SVO), but we leave verification of these additional scenarios for future work 
Experimental Setup
In this section, we describe the languages experimented with, datasets used, the network hyper-parameters used in our experiments 
Languages
We experimented with English INLINEFORM0 Hindi translation as the parent task  English is the assisting source language  Bengali, Gujarati, Marathi, Malayalam and Tamil are the primary source languages, and translation from these to Hindi constitute the child tasks  Hindi, Bengali, Gujarati and Marathi are Indo-Aryan languages, while Malayalam and Tamil are Dravidian languages  All these languages have a canonical SOV word order 
Datasets
For training English-Hindi NMT systems, we use the IITB English-Hindi parallel corpus BIBREF22 ( INLINEFORM0 sentences from the training set) and the ILCI English-Hindi parallel corpus ( INLINEFORM1 sentences)  The ILCI (Indian Language Corpora Initiative) multilingual parallel corpus BIBREF23 spans multiple Indian languages from the health and tourism domains  We use the 520-sentence dev-set of the IITB parallel corpus for validation  For each child task, we use INLINEFORM2 sentences from ILCI corpus as the test set 
Network
We use OpenNMT-Torch BIBREF24 to train the NMT system  We use the standard sequence-to-sequence architecture with attention BIBREF13   We use an encoder which contains two layers of bidirectional LSTMs with 500 neurons each  The decoder contains two LSTM layers with 500 neurons each  Input feeding approach BIBREF1 is used where the previous attention hidden state is fed as input to the decoder LSTM  We use a mini-batch of size 50 and use a dropout layer  We begin with an initial learning rate of INLINEFORM0 and decay the learning rate by a factor of INLINEFORM1 when the perplexity on validation set increases  The training is stopped when the learning rate falls below INLINEFORM2 or number of epochs=22  The English input is initialized with pre-trained embeddings trained using fastText BIBREF25  
English vocabulary consists of INLINEFORM0 tokens appearing at least 2 times in the English training corpus  For constructing the Hindi vocabulary we considered only those tokens appearing at least 5 times in the training split resulting in a vocabulary size of INLINEFORM1 tokens  For representing English and other source languages into a common space, we translate each word in the source language into English using a bilingual dictionary (Google Translate word translation in our case)  In an end-to-end solution, it would have been ideal to use bilingual embeddings or obtain word-by-word translations via bilingual embeddings BIBREF14   But, the quality of publicly available bilingual embeddings for English-Indian languages is very low for obtaining good-quality, bilingual representations BIBREF26 , BIBREF27   We also found that these embeddings were not useful for transfer learning 
We use the CFILT-preorder system for reordering English sentences to match the Indian language word order  It contains two re-ordering systems: (1) generic rules that apply to all Indian languages BIBREF17 , and (2) hindi-tuned rules which improve the generic rules by incorporating improvements found through an error analysis of English-Hindi reordering BIBREF28   These Hindi-tuned rules have been found to improve reordering for many English to Indian language pairs BIBREF29  
Results
In this section, we describe the results from our experiments on NMT task  We report the results on X-Hindi pair, where X is one of Bengali, Gujarati, Marathi, Tamil, and Malayalam  The results are presented in the Table TABREF6   We report BLEU scores and LeBLEU scores BIBREF30   We observe that both the pre-ordering configurations significantly improve the BLEU scores over the baseline scores  We observe larger gains when generic pre-ordering rules are used compared to the Hindi-tuned pre-ordering rules 
These results support our hypothesis that word-order divergence can limit the benefits of multilingual translation  Reducing the word order divergence can improve translation in extremely low-resource scenarios 
An analysis of the outputs revealed that pre-ordering significantly reducing the number of UNK tokens (placeholder for unknown words) in the test output (Table TABREF14 )  We hypothesize that due to word order divergence between English and Indian languages, the encoder representation generated is not consistent leading to decoder generating unknown words  However, the pre-ordered models generate better contextual representations leading to less number of unknown tokens and better translation which is also reflected in the BLEU scores 
Conclusion
In this paper, we show that handling word-order divergence between source and assisting languages is crucial for the success of multilingual NMT in an extremely low-resource setting  We show that pre-ordering the assisting language to match the word order of the source language significantly improves translation quality in an extremely low-resource setting  While the current work focused on Indian languages, we would like to validate the hypothesis on a more diverse set of languages 

ground_truth: Unanswerable

CFILT-preorder system

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How is non-standard pronunciation identified?
Context 4:  The two word representations INLINEFORM3 and INLINEFORM4 are projected to a space where their element wise difference can be used to model their dissimilarity and softmax applied on them can produce high probability to more dissimilar word pairs 
It is good to note that conflict suffers from the same limitation that attention suffers from  This is when a pair of sentences are highly matching especially with multiple associations  But when the two methods work together, each compensates for the other's shortcomings 
Combination of attention and conflict
We used two weighted representations of INLINEFORM0 using weights of attention and conflict as computed in Eqn  (4) and (8) respectively  Our final representation of a word representation INLINEFORM1 conditioned on INLINEFORM2 can be expressed as: DISPLAYFORM0
where A and C denote that they are from attention and conflict models respectively 
Relation to Multi-Head attention
Multi-head attention, as introduced in BIBREF3 , computes multiple identical attention mechanism parallelly on multiple linear projections of same inputs  The parameters of each attention and projections are different in each head  Finally, they concatenate all the attentions which is similar to how we concatenate conflict and attention  However, they use dot-product to compute each of the attention 
Our combined model that contains both attention and conflict can be thought of as a 2-head attention model but both heads are different  Our conflict head explicitly captures difference between the inputs 
Visualizing attention and conflict
We observe how our conflict model learns the dissimilarities between word representations  We achieve that by visualizing the heatmap of the weight matrix INLINEFORM0 for both attention and conflict from eqns  (3) and (8)  While attention successfully learns the alignments, conflict matrix also shows that our approach models the contradicting associations like "animal" and "lake" or "australia" and "world"  These two associations are the unique pairs which are instrumental in determining that the two queries are not similar 
The model
We create two models both of which constitutes of three main parts: encoder, interaction and classifier and take two sequences as input  Except interaction, all the other parts are exactly identical between the two models  The encoder is shared among the sequences simply uses two stacked GRU layers  The interaction part consists of only attention for one model while for the another one it consists of attention and conflict combined as shown in (eqn 11)   The classifier part is simply stacked fully-connected layers  Figure 3 shows a block diagram of how our model looks like 
Task 1: Quora Duplicate Question Pair Detection
The dataset includes pairs of questions labelled as 1 or 0 depending on whether a pair is duplicate or not respectively  This is a popular pair-level classification task on which extensive work has already been done before like BIBREF7 , BIBREF8   For this task, we make the output layer of our model to predict two probabilities for non-duplicate and duplicate  We sample the data from the original dataset so that it contains equal positive and negative classes  Original dataset has some class imbalance but for sake simplicity we don't consider it  The final data that we use has roughly 400,000 question pairs and we split this data into train and test using 8:2 ratio 
We train all our models for roughly 2 epochs with a batch size of 64  We use a hidden dimension of 150 throughout the model  The embedding layer uses ELMO BIBREF9 which has proven to be very useful in various downstream language understanding tasks  Our FC layers consists of four dense layers with INLINEFORM0 activation after each layer  The dropout rate is kept as 0 2 for every recurrent and FC linear layers  We use Adam optimizer in our experiment with epsilon=1e-8, beta=0 9 and learning rate=1e-3 
Task 2: Ranking questions in Bing's People Also Ask
People Also Ask is a feature in Bing search result page where related questions are recommended to the user  User may click on a question to view the answer  Clicking is a positive feedback that shows user's interest in the question  We use this click logs to build a question classifier using the same model in Figure 3  The problem statement is very similar to BIBREF10 where they use logistic regression to predict whether an user would click on ad  Our goal is to classify if a question is potential high-click question or not for a given query  For this, we first create a labelled data set using the click logs where any question having CTR lower than 0 3 is labelled as 0 and a question having CTR more than 0 7 as 1 
Our final data resembles that of a pair-level classifier, as in Task 1, where user query and candidate questions are input  With these data set, we train a binary classifier to detect high-click and low-click questions 
Quantitative Analysis
For both tasks, we compute classification accuracy using three model variants and report the results in Table 1 and Table 2  We observe that model with both attention and conflict combined gives the best results 
We also show the training loss curve for both the models having attention and attention combined with conflict respectively  Figure 4 and 5 shows these curves for Task 1 and Task 2 respectively  The curves are smoothed using moving average having an window size of 8  We notice that the conflict model has much steeper slope and converges to a much better minima in both the tasks  It can also be noticed that in the training procedure for the model which has both attention and conflict, the updates are much smoother 
Qualitative Comparison
We also show qualitative results where we can observe that our model with attention and conflict combined does better on cases where pairs are non-duplicate and has very small difference  We have observed that the conflict model is very sensitive to even minor differences and compensates in such cases where attention poses high bias towards similarities already there in the sequences 
Sequence 1: What are the best ways to learn French ?
Sequence 2: How do I learn french genders ?
Attention only: 1
Attention+Conflict: 0
Ground Truth: 0
Sequence 1: How do I prevent breast cancer ?
Sequence 2: Is breast cancer preventable ?
Attention only: 1
Attention+Conflict: 0
Ground Truth: 0
We provide two examples with predictions from the models with only attention and combination of attention and conflict  Each example is accompanied by the ground truth in our data 
Analyzing the gains
We analyzed the gains in Task 1 which we get from the attention-conflict model in order to ensure that they are not due to randomness in weight initialization or simply additional parameters  We particularly focused on the examples which were incorrectly marked in attention model but correctly in attention-conflict model  We saw that 70% of those cases are the ones where the pair was incorrectly marked as duplicate in the previous model but our combined model correctly marked them as non-duplicate 
Conclusion
In this work, we highlighted the limits of attention especially in cases where two sequences have a contradicting relationship based on the task it performs  To alleviate this problem and further improve the performance, we propose a conflict mechanism that tries to capture how two sequences repel each other  This acts like the inverse of attention and, empirically, we show that how conflict and attention together can improve the performance 
Future research work should be based on alternative design of conflict mechanism using other difference operators other than element wise difference which we use 

ground_truth: Unanswerable

Original transcription was labeled with additional labels in [] brackets with nonstandard pronunciation.