prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How large is their data set?
Context 1: Introduction
The task of document quality assessment is to automatically assess a document according to some predefined inventory of quality labels  This can take many forms, including essay scoring (quality = language quality, coherence, and relevance to a topic), job application filtering (quality = suitability for role + visual/presentational quality of the application), or answer selection in community question answering (quality = actionability + relevance of the answer to the question)  In the case of this paper, we focus on document quality assessment in two contexts: Wikipedia document quality classification, and whether a paper submitted to a conference was accepted or not 
Automatic quality assessment has obvious benefits in terms of time savings and tractability in contexts where the volume of documents is large  In the case of dynamic documents (possibly with multiple authors), such as in the case of Wikipedia, it is particularly pertinent, as any edit potentially has implications for the quality label of that document (and around 10 English Wikipedia documents are edited per second)  Furthermore, when the quality assessment task is decentralized (as in the case of Wikipedia and academic paper assessment), quality criteria are often applied inconsistently by different people, where an automatic document quality assessment system could potentially reduce inconsistencies and enable immediate author feedback 
Current studies on document quality assessment mainly focus on textual features  For example, BIBREF0 examine features such as the article length and the number of headings to predict the quality class of a Wikipedia article  In contrast to these studies, in this paper, we propose to combine text features with visual features, based on a visual rendering of the document  Figure 1 illustrates our intuition, relative to Wikipedia articles  Without being able to read the text, we can tell that the article in Figure 1 has higher quality than Figure 1 , as it has a detailed infobox, extensive references, and a variety of images  Based on this intuition, we aim to answer the following question: can we achieve better accuracy on document quality assessment by complementing textual features with visual features?
Our visual model is based on fine-tuning an Inception V3 model BIBREF1 over visual renderings of documents, while our textual model is based on a hierarchical biLSTM  We further combine the two into a joint model  We perform experiments on two datasets: a Wikipedia dataset novel to this paper, and an arXiv dataset provided by BIBREF2 split into three sub-parts based on subject category  Experimental results on the visual renderings of documents show that implicit quality indicators, such as images and visual layout, can be captured by an image classifier, at a level comparable to a text classifier  When we combine the two models, we achieve state-of-the-art results over 3/4 of our datasets 
This paper makes the following contributions:
All code and data associated with this research will be released on publication 
Related Work
A variety of approaches have been proposed for document quality assessment across different domains: Wikipedia article quality assessment, academic paper rating, content quality assessment in community question answering (cQA), and essay scoring  Among these approaches, some use hand-crafted features while others use neural networks to learn features from documents  For each domain, we first briefly describe feature-based approaches and then review neural network-based approaches  Wikipedia article quality assessment: Quality assessment of Wikipedia articles is a task that assigns a quality class label to a given Wikipedia article, mirroring the quality assessment process that the Wikipedia community carries out manually  Many approaches have been proposed that use features from the article itself, meta-data features (e g , the editors, and Wikipedia article revision history), or a combination of the two  Article-internal features capture information such as whether an article is properly organized, with supporting evidence, and with appropriate terminology  For example, BIBREF3 use writing styles represented by binarized character trigram features to identify featured articles  BIBREF4 and BIBREF0 explore the number of headings, images, and references in the article  BIBREF5 use nine readability scores, such as the percentage of difficult words in the document, to measure the quality of the article  Meta-data features, which are indirect indicators of article quality, are usually extracted from revision history, and the interaction between editors and articles  For example, one heuristic that has been proposed is that higher-quality articles have more edits BIBREF6 , BIBREF7   BIBREF8 use the percentage of registered editors and the total number of editors of an article  Article–editor dependencies have also been explored  For example, BIBREF9 use the authority of editors to measure the quality of Wikipedia articles, where the authority of editors is determined by the articles they edit  Deep learning approaches to predicting Wikipedia article quality have also been proposed  For example, BIBREF10 use a version of doc2vec BIBREF11 to represent articles, and feed the document embeddings into a four hidden layer neural network  BIBREF12 first obtain sentence representations by averaging words within a sentence, and then apply a biLSTM BIBREF13 to learn a document-level representation, which is combined with hand-crafted features as side information  BIBREF14 exploit two stacked biLSTMs to learn document representations 
Academic paper rating: Academic paper rating is a relatively new task in NLP/AI, with the basic formulation being to automatically predict whether to accept or reject a paper  BIBREF2 explore hand-crafted features, such as the length of the title, whether specific words (such as outperform, state-of-the-art, and novel) appear in the abstract, and an embedded representation of the abstract as input to different downstream learners, such as logistic regression, decision tree, and random forest  BIBREF15 exploit a modularized hierarchical convolutional neural network (CNN), where each paper section is treated as a module  For each paper section, they train an attention-based CNN, and an attentive pooling layer is applied to the concatenated representation of each section, which is then fed into a softmax layer 
Content quality assessment in cQA: Automatic quality assessment in cQA is the task of determining whether an answer is of high quality, selected as the best answer, or ranked higher than other answers  To measure answer content quality in cQA, researchers have exploited various features from different sources, such as the answer content itself, the answerer's profile, interactions among users, and usage of the content  The most common feature used is the answer length BIBREF16 , BIBREF17 , with other features including: syntactic and semantic features, such as readability scores  BIBREF18 ; similarity between the question and the answer at lexical, syntactic, and semantic levels BIBREF18 , BIBREF19 , BIBREF20 ; or user data (e g , a user's status points or the number of answers written by the user)  There have also been approaches using neural networks  For example, BIBREF21 combine CNN-learned representations with hand-crafted features to predict answer quality  BIBREF22 use a 2-dimensional CNN to learn the semantic relevance of an answer to the question, and apply an LSTM to the answer sequence to model thread context  BIBREF23 and BIBREF24 model the problem similarly to machine translation quality estimation, treating answers as competing translation hypotheses and the question as the reference translation, and apply neural machine translation to the problem  Essay scoring: Automated essay scoring is the task of assigning a score to an essay, usually in the context of assessing the language ability of a language learner  The quality of an essay is affected by the following four primary dimensions: topic relevance, organization and coherence, word usage and sentence complexity, and grammar and mechanics  To measure whether an essay is relevant to its “prompt” (the description of the essay topic), lexical and semantic overlap is commonly used BIBREF25 , BIBREF26   BIBREF27 explore word features, such as the number of verb formation errors, average word frequency, and average word length, to measure word usage and lexical complexity  BIBREF28 use sentence structure features to measure sentence variety  The effects of grammatical and mechanic errors on the quality of an essay are measured via word and part-of-speech $n$ -gram features and “mechanics” features BIBREF29 (e g , spelling, capitalization, and punctuation), respectively  BIBREF30 , BIBREF31 , and BIBREF32 use an LSTM to obtain an essay representation, which is used as the basis for classification  Similarly, BIBREF33 utilize a CNN to obtain sentence representation and an LSTM to obtain essay representation, with an attention layer at both the sentence and essay levels 
The Proposed Joint Model
We treat document quality assessment as a classification problem, i e , given a document, we predict its quality class (e g , whether an academic paper should be accepted or rejected)  The proposed model is a joint model that integrates visual features learned through Inception V3 with textual features learned through a biLSTM  In this section, we present the details of the visual and textual embeddings, and finally describe how we combine the two  We return to discuss hyper-parameter settings and the experimental configuration in the Experiments section 
Visual Embedding Learning
A wide range of models have been proposed to tackle the image classification task, such as VGG BIBREF34 , ResNet BIBREF35 , Inception V3 BIBREF1 , and Xception BIBREF36   However, to the best of our knowledge, there is no existing work that has proposed to use visual renderings of documents to assess document quality  In this paper, we use Inception V3 pretrained on ImageNet (“Inception” hereafter) to obtain visual embeddings of documents, noting that any image classifier could be applied to our task  The input to Inception is a visual rendering (screenshot) of a document, and the output is a visual embedding, which we will later integrate with our textual embedding 
Based on the observation that it is difficult to decide what types of convolution to apply to each layer (such as 3 $\times $ 3 or 5 $\times $ 5), the basic Inception model applies multiple convolution filters in parallel and concatenates the resulting features, which are fed into the next layer  This has the benefit of capturing both local features through smaller convolutions and abstracted features through larger convolutions  Inception is a hybrid of multiple Inception models of different architectures  To reduce computational cost, Inception also modifies the basic model by applying a 1 $\times $ 1 convolution to the input and factorizing larger convolutions into smaller ones 
Textual Embedding Learning
We adopt a bi-directional LSTM model to generate textual embeddings for document quality assessment, following the method of BIBREF12 (“biLSTM” hereafter)  The input to biLSTM is a textual document, and the output is a textual embedding, which will later integrate with the visual embedding 
For biLSTM, each word is represented as a word embedding BIBREF37 , and an average-pooling layer is applied to the word embeddings to obtain the sentence embedding, which is fed into a bi-directional LSTM to generate the document embedding from the sentence embeddings  Then a max-pooling layer is applied to select the most salient features from the component sentences 
The Joint Model
The proposed joint model (“Joint” hereafter) combines the visual and textual embeddings (output of Inception and biLSTM) via a simple feed-forward layer and softmax over the document label set, as shown in Figure 2   We optimize our model based on cross-entropy loss 
Experiments
In this section, we first describe the two datasets used in our experiments: (1) Wikipedia, and (2) arXiv  Then, we report the experimental details and results 
Datasets
The Wikipedia dataset consists of articles from English Wikipedia, with quality class labels assigned by the Wikipedia community  Wikipedia articles are labelled with one of six quality classes, in descending order of quality: Featured Article (“FA”), Good Article (“GA”), B-class Article (“B”), C-class Article (“C”), Start Article (“Start”), and Stub Article (“Stub”)  A description of the criteria associated with the different classes can be found in the Wikipedia grading scheme page  The quality class of a Wikipedia article is assigned by Wikipedia reviewers or any registered user, who can discuss through the article's talk page to reach consensus  We constructed the dataset by first crawling all articles from each quality class repository, e g , we get FA articles by crawling pages from the FA repository: https://en wikipedia org/wiki/Category:Featured_articles  This resulted in around 5K FA, 28K GA, 212K B, 533K C, 2 6M Start, and 3 2M Stub articles 
We randomly sampled 5,000 articles from each quality class and removed all redirect pages, resulting in a dataset of 29,794 articles  As the wikitext contained in each document contains markup relating to the document category such as {Featured Article} or {geo-stub}, which reveals the label, we remove such information  We additionally randomly partitioned this dataset into training, development, and test splits based on a ratio of 8:1:1  Details of the dataset are summarized in Table 1  
We generate a visual representation of each document via a 1,000 $\times $ 2,000-pixel screenshot of the article via a PhantomJS script over the rendered version of the article, ensuring that the screenshot and wikitext versions of the article are the same version  Any direct indicators of document quality (such as the FA indicator, which is a bronze star icon in the top right corner of the webpage) are removed from the screenshot 
The arXiv dataset BIBREF2 consists of three subsets of academic articles under the arXiv repository of Computer Science (cs), from the three subject areas of: Artificial Intelligence (cs ai), Computation and Language (cs cl), and Machine Learning (cs lg)  In line with the original dataset formulation BIBREF2 , a paper is considered to have been accepted (i e  is positively labeled) if it matches a paper in the DBLP database or is otherwise accepted by any of the following conferences: ACL, EMNLP, NAACL, EACL, TACL, NIPS, ICML, ICLR, or AAAI  Failing this, it is considered to be rejected (noting that some of the papers may not have been submitted to one of these conferences)  The median numbers of pages for papers in cs ai, cs cl, and cs lg are 11, 10, and 12, respectively  To make sure each page in the PDF file has the same size in the screenshot, we crop the PDF file of a paper to the first 12; we pad the PDF file with blank pages if a PDF file has less than 12 pages, using the PyPDF2 Python package  We then use ImageMagick to convert the 12-page PDF file to a single 1,000 $\times $ 2,000 pixel screenshot  Table 2 details this dataset, where the “Accepted” column denotes the percentage of positive instances (accepted papers) in each subset 
Experimental Setting
As discussed above, our model has two main components — biLSTM and Inception— which generate textual and visual representations, respectively  For the biLSTM component, the documents are preprocessed as described in BIBREF12 , where an article is divided into sentences and tokenized using NLTK BIBREF38   Words appearing more than 20 times are retained when building the vocabulary  All other words are replaced by the special UNK token  We use the pre-trained GloVe BIBREF39 50-dimensional word embeddings to represent words  For words not in GloVe, word embeddings are randomly initialized based on sampling from a uniform distribution $U(-1, 1)$   All word embeddings are updated in the training process  We set the LSTM hidden layer size to 256  The concatenation of the forward and backward LSTMs thus gives us 512 dimensions for the document embedding  A dropout layer is applied at the sentence and document level, respectively, with a probability of 0 5 
For Inception, we adopt data augmentation techniques in the training with a “nearest” filling mode, a zoom range of 0 1, a width shift range of 0 1, and a height shift range of 0 1  As the original screenshots have the size of 1,000 $\times 2$ ,000 pixels, they are resized to 500 $\times $ 500 to feed into Inception, where the input shape is (500, 500, 3)  A dropout layer is applied with a probability of 0 5  Then, a GlobalAveragePooling2D layer is applied, which produces a 2,048 dimensional representation 
For the Joint model, we get a representation of 2,560 dimensions by concatenating the 512 dimensional representation from the biLSTM with the 2,048 dimensional representation from Inception  The dropout layer is applied to the two components with a probability of 0 5  For biLSTM, we use a mini-batch size of 128 and a learning rate of 0 001  For both Inception and joint model, we use a mini-batch size of 16 and a learning rate of 0 0001  All hyper-parameters were set empirically over the development data, and the models were optimized using the Adam optimizer BIBREF40  
In the training phase, the weights in Inception are initialized by parameters pretrained on ImageNet, and the weights in biLSTM are randomly initialized (except for the word embeddings)  We train each model for 50 epochs  However, to prevent overfitting, we adopt early stopping, where we stop training the model if the performance on the development set does not improve for 20 epochs  For evaluation, we use (micro-)accuracy, following previous studies BIBREF5 , BIBREF2  
Baseline Approaches
We compare our models against the following five baselines:
Majority: the model labels all test samples with the majority class of the training data 
Benchmark: a benchmark method from the literature  In the case of Wikipedia, this is BIBREF5 , who use structural features and readability scores as features to build a random forest classifier; for arXiv, this is BIBREF2 , who use hand-crafted features, such as the number of references and TF-IDF weighted bag-of-words in abstract, to build a classifier based on the best of logistic regression, multi-layer perception, and AdaBoost 
Doc2Vec: doc2vec BIBREF11 to learn document embeddings with a dimension of 500, and a 4-layer feed-forward classification model on top of this, with 2000, 1000, 500, and 200 dimensions, respectively 
biLSTM: first derive a sentence representation by averaging across words in a sentence, then feed the sentence representation into a biLSTM and a maxpooling layer over output sequence to learn a document level representation with a dimension of 512, which is used to predict document quality 
Inception $_{\text{fixed}}$ : the frozen Inception model, where only parameters in the last layer are fine-tuned during training 
The hyper-parameters of Benchmark, Doc2Vec, and biLSTM are based on the corresponding papers except that: (1) we fine-tune the feed forward layer of Doc2Vec on the development set and train the model 300 epochs on Wikipedia and 50 epochs on arXiv; (2) we do not use hand-crafted features for biLSTM as we want the baselines to be comparable to our models, and the main focus of this paper is not to explore the effects of hand-crafted features (e g , see BIBREF12 ) 
Experimental Results
Table 3 shows the performance of the different models over our two datasets, in the form of the average accuracy on the test set (along with the standard deviation) over 10 runs, with different random initializations 
On Wikipedia, we observe that the performance of biLSTM, Inception, and Joint is much better than that of all four baselines  Inception achieves 2 9% higher accuracy than biLSTM  The performance of Joint achieves an accuracy of 59 4%, which is 5 3% higher than using textual features alone (biLSTM) and 2 4% higher than using visual features alone (Inception)  Based on a one-tailed Wilcoxon signed-rank test, the performance of Joint is statistically significant ( $p<0 05$ )  This shows that the textual and visual features complement each other, achieving state-of-the-art results in combination 
For arXiv, baseline methods Majority, Benchmark, and Inception $_{\text{fixed}}$ outperform biLSTM over cs ai, in large part because of the class imbalance in this dataset (90% of papers are rejected)  Surprisingly, Inception $_{\text{fixed}}$ is better than Majority and Benchmark over the arXiv cs lg subset, which verifies the usefulness of visual features, even when only the last layer is fine-tuned  Table 3 also shows that Inception and biLSTM achieve similar performance on arXiv, showing that textual and visual representations are equally discriminative: Inception and biLSTM are indistinguishable over cs cl; biLSTM achieves 1 8% higher accuracy over cs lg, while Inception achieves 1 3% higher accuracy over cs ai  Once again, the Joint model achieves the highest accuracy on cs ai and cs cl by combining textual and visual representations (at a level of statistical significance for cs ai)  This, again, confirms that textual and visual features complement each other, and together they achieve state-of-the-art results  On arXiv cs lg, Joint achieves a 0 6% higher accuracy than Inception by combining visual features and textual features, but biLSTM achieves the highest accuracy  One characteristic of cs lg documents is that they tend to contain more equations than the other two arXiv datasets, and preliminary analysis suggests that the biLSTM is picking up on a correlation between the volume/style of mathematical presentation and the quality of the document 
Analysis
In this section, we first analyze the performance of Inception and Joint  We also analyze the performance of different models on different quality classes  The high-level representations learned by different models are also visualized and discussed  As the Wikipedia test set is larger and more balanced than that of arXiv, our analysis will focus on Wikipedia 
Inception
To better understand the performance of Inception, we generated the gradient-based class activation map BIBREF41 , by maximizing the outputs of each class in the penultimate layer, as shown in Figure 3   From Figure 3 and Figure 3 , we can see that Inception identifies the two most important regions (one at the top corresponding to the table of contents, and the other at the bottom, capturing both document length and references) that contribute to the FA class prediction, and a region in the upper half of the image that contributes to the GA class prediction (capturing the length of the article body)  From Figure 3 and Figure 3 , we can see that the most important regions in terms of B and C class prediction capture images (down the left and right of the page, in the case of B and C), and document length/references  From Figure 3 and Figure 3 , we can see that Inception finds that images in the top right corner are the strongest predictor of Start class prediction, and (the lack of) images/the link bar down the left side of the document are the most important for Stub class prediction 
Joint
Table 4 shows the confusion matrix of Joint on Wikipedia  We can see that more than 50% of documents for each quality class are correctly classified, except for the C class where more documents are misclassified into B  Analysis shows that when misclassified, documents are usually misclassified into adjacent quality classes, which can be explained by the Wikipedia grading scheme, where the criteria for adjacent quality classes are more similar 
We also provide a breakdown of precision (“ $\mathcal {P}$ ”), recall (“ $\mathcal {R}$ ”), and F1 score (“ $\mathcal {F}_{\beta =1}$ ”) for biLSTM, Inception, and Joint across the quality classes in Table 5   We can see that Joint achieves the highest accuracy in 11 out of 18 cases  It is also worth noting that all models achieve higher scores for FA, GA, and Stub articles than B, C and Start articles  This can be explained in part by the fact that FA and GA articles must pass an official review based on structured criteria, and in part by the fact that Stub articles are usually very short, which is discriminative for Inception, and Joint  All models perform worst on the B and C quality classes  It is difficult to differentiate B articles from C articles even for Wikipedia contributors  As evidence of this, when we crawled a new dataset including talk pages with quality class votes from Wikipedia contributors, we found that among articles with three or more quality labels, over 20% percent of B and C articles have inconsistent votes from Wikipedia contributors, whereas for FA and GA articles the number is only 0 7% 
We further visualize the learned document representations of biLSTM, Inception, and Joint in the form of a t-SNE plot BIBREF42 in Figure 4   The degree of separation between Start and Stub achieved by Inception is much greater than for biLSTM, with the separation between Start and Stub achieved by Joint being the clearest among the three models  Inception and Joint are better than biLSTM at separating Start and C  Joint achieves slightly better performance than Inception in separating GA and FA  We can also see that it is difficult for all models to separate B and C, which is consistent with the findings of Tables 4 and 5  
Conclusions
We proposed to use visual renderings of documents to capture implicit document quality indicators, such as font choices, images, and visual layout, which are not captured in textual content  We applied neural network models to capture visual features given visual renderings of documents  Experimental results show that we achieve a 2 9% higher accuracy than state-of-the-art approaches based on textual features over Wikipedia, and performance competitive with or surpassing state-of-the-art approaches over arXiv  We further proposed a joint model, combining textual and visual representations, to predict the quality of a document  Experimental results show that our joint model outperforms the visual-only model in all cases, and the text-only model on Wikipedia and two subsets of arXiv  These results underline the feasibility of assessing document quality via visual features, and the complementarity of visual and textual document representations for quality assessment 
Context 2: Introduction
Knowledge Base Question Answering (KBQA) systems answer questions by obtaining information from KB tuples BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5   For an input question, these systems typically generate a KB query, which can be executed to retrieve the answers from a KB  Figure 1 illustrates the process used to parse two sample questions in a KBQA system: (a) a single-relation question, which can be answered with a single $<$ head-entity, relation, tail-entity $>$ KB tuple BIBREF6 , BIBREF7 , BIBREF2 ; and (b) a more complex case, where some constraints need to be handled for multiple entities in the question  The KBQA system in the figure performs two key tasks: (1) entity linking, which links $n$ -grams in questions to KB entities, and (2) relation detection, which identifies the KB relation(s) a question refers to 
The main focus of this work is to improve the relation detection subtask and further explore how it can contribute to the KBQA system  Although general relation detection methods are well studied in the NLP community, such studies usually do not take the end task of KBQA into consideration  As a result, there is a significant gap between general relation detection studies and KB-specific relation detection  First, in most general relation detection tasks, the number of target relations is limited, normally smaller than 100  In contrast, in KBQA even a small KB, like Freebase2M BIBREF2 , contains more than 6,000 relation types  Second, relation detection for KBQA often becomes a zero-shot learning task, since some test instances may have unseen relations in the training data  For example, the SimpleQuestions BIBREF2 data set has 14% of the golden test relations not observed in golden training tuples  Third, as shown in Figure 1 (b), for some KBQA tasks like WebQuestions BIBREF0 , we need to predict a chain of relations instead of a single relation  This increases the number of target relation types and the sizes of candidate relation pools, further increasing the difficulty of KB relation detection  Owing to these reasons, KB relation detection is significantly more challenging compared to general relation detection tasks 
This paper improves KB relation detection to cope with the problems mentioned above  First, in order to deal with the unseen relations, we propose to break the relation names into word sequences for question-relation matching  Second, noticing that original relation names can sometimes help to match longer question contexts, we propose to build both relation-level and word-level relation representations  Third, we use deep bidirectional LSTMs (BiLSTMs) to learn different levels of question representations in order to match the different levels of relation information  Finally, we propose a residual learning method for sequence matching, which makes the model training easier and results in more abstract (deeper) question representations, thus improves hierarchical matching 
In order to assess how the proposed improved relation detection could benefit the KBQA end task, we also propose a simple KBQA implementation composed of two-step relation detection  Given an input question and a set of candidate entities retrieved by an entity linker based on the question, our proposed relation detection model plays a key role in the KBQA process: (1) Re-ranking the entity candidates according to whether they connect to high confident relations detected from the raw question text by the relation detection model  This step is important to deal with the ambiguities normally present in entity linking results  (2) Finding the core relation (chains) for each topic entity selection from a much smaller candidate entity set after re-ranking  The above steps are followed by an optional constraint detection step, when the question cannot be answered by single relations (e g , multiple entities in the question)  Finally the highest scored query from the above steps is used to query the KB for answers 
Our main contributions include: (i) An improved relation detection model by hierarchical matching between questions and relations with residual learning; (ii) We demonstrate that the improved relation detector enables our simple KBQA system to achieve state-of-the-art results on both single-relation and multi-relation KBQA tasks 
Background: Different Granularity in KB Relations
Previous research BIBREF4 , BIBREF20 formulates KB relation detection as a sequence matching problem  However, while the questions are natural word sequences, how to represent relations as sequences remains a challenging problem  Here we give an overview of two types of relation sequence representations commonly used in previous work 
(1) Relation Name as a Single Token (relation-level)  In this case, each relation name is treated as a unique token  The problem with this approach is that it suffers from the low relation coverage due to limited amount of training data, thus cannot generalize well to large number of open-domain relations  For example, in Figure 1 , when treating relation names as single tokens, it will be difficult to match the questions to relation names “episodes_written” and “starring_roles” if these names do not appear in training data – their relation embeddings $\mathbf {h}^r$ s will be random vectors thus are not comparable to question embeddings $\mathbf {h}^q$ s 
(2) Relation as Word Sequence (word-level)  In this case, the relation is treated as a sequence of words from the tokenized relation name  It has better generalization, but suffers from the lack of global information from the original relation names  For example in Figure 1 (b), when doing only word-level matching, it is difficult to rank the target relation “starring_roles” higher compared to the incorrect relation “plays_produced”  This is because the incorrect relation contains word “plays”, which is more similar to the question (containing word “play”) in the embedding space  On the other hand, if the target relation co-occurs with questions related to “tv appearance” in training, by treating the whole relation as a token (i e  relation id), we could better learn the correspondence between this token and phrases like “tv show” and “play on” 
The two types of relation representation contain different levels of abstraction  As shown in Table 1 , the word-level focuses more on local information (words and short phrases), and the relation-level focus more on global information (long phrases and skip-grams) but suffer from data sparsity  Since both these levels of granularity have their own pros and cons, we propose a hierarchical matching approach for KB relation detection: for a candidate relation, our approach matches the input question to both word-level and relation-level representations to get the final ranking score  Section "Improved KB Relation Detection" gives the details of our proposed approach 
Improved KB Relation Detection
This section describes our hierarchical sequence matching with residual learning approach for relation detection  In order to match the question to different aspects of a relation (with different abstraction levels), we deal with three problems as follows on learning question/relation representations 
Relation Representations from Different Granularity
We provide our model with both types of relation representation: word-level and relation-level  Therefore, the input relation becomes $\mathbf {r}=\lbrace r^{word}_1,\cdots ,r^{word}_{M_1}\rbrace  \cup \lbrace r^{rel}_1,\cdots ,r^{rel}_{M_2}\rbrace $ , where the first $M_1$ tokens are words (e g  {episode, written}), and the last $M_2$ tokens are relation names, e g , {episode_written} or {starring_roles, series} (when the target is a chain like in Figure 1 (b))  We transform each token above to its word embedding then use two BiLSTMs (with shared parameters) to get their hidden representations $[\mathbf {B}^{word}_{1:M_1}:\mathbf {B}^{rel}_{1:M_2}]$ (each row vector $\mathbf {\beta }_i$ is the concatenation between forward/backward representations at $i$ )  We initialize the relation sequence LSTMs with the final state representations of the word sequence, as a back-off for unseen relations  We apply one max-pooling on these two sets of vectors and get the final relation representation $\mathbf {h}^r$  
Different Abstractions of Questions Representations
From Table 1 , we can see that different parts of a relation could match different contexts of question texts  Usually relation names could match longer phrases in the question and relation words could match short phrases  Yet different words might match phrases of different lengths 
As a result, we hope the question representations could also comprise vectors that summarize various lengths of phrase information (different levels of abstraction), in order to match relation representations of different granularity  We deal with this problem by applying deep BiLSTMs on questions  The first-layer of BiLSTM works on the word embeddings of question words $\mathbf {q}=\lbrace q_1,\cdots ,q_N\rbrace $ and gets hidden representations $\mathbf {\Gamma }^{(1)}_{1:N}=[\mathbf {\gamma }^{(1)}_1;\cdots ;\mathbf {\gamma }^{(1)}_N]$   The second-layer BiLSTM works on $\mathbf {\Gamma }^{(1)}_{1:N}$ to get the second set of hidden representations $\mathbf {\Gamma }^{(2)}_{1:N}$   Since the second BiLSTM starts with the hidden vectors from the first layer, intuitively it could learn more general and abstract information compared to the first layer 
Note that the first(second)-layer of question representations does not necessarily correspond to the word(relation)-level relation representations, instead either layer of question representations could potentially match to either level of relation representations  This raises the difficulty of matching between different levels of relation/question representations; the following section gives our proposal to deal with such problem 
Hierarchical Matching between Relation and Question
Now we have question contexts of different lengths encoded in $\mathbf {\Gamma }^{(1)}_{1:N}$ and $\mathbf {\Gamma }^{(2)}_{1:N}$   Unlike the standard usage of deep BiLSTMs that employs the representations in the final layer for prediction, here we expect that two layers of question representations can be complementary to each other and both should be compared to the relation representation space (Hierarchical Matching)  This is important for our task since each relation token can correspond to phrases of different lengths, mainly because of syntactic variations  For example in Table 1 , the relation word written could be matched to either the same single word in the question or a much longer phrase be the writer of 
We could perform the above hierarchical matching by computing the similarity between each layer of $\mathbf {\Gamma }$ and $\mathbf {h}^r$ separately and doing the (weighted) sum between the two scores  However this does not give significant improvement (see Table 2 )  Our analysis in Section "Relation Detection Results" shows that this naive method suffers from the training difficulty, evidenced by that the converged training loss of this model is much higher than that of a single-layer baseline model  This is mainly because (1) Deep BiLSTMs do not guarantee that the two-levels of question hidden representations are comparable, the training usually falls to local optima where one layer has good matching scores and the other always has weight close to 0  (2) The training of deeper architectures itself is more difficult 
To overcome the above difficulties, we adopt the idea from Residual Networks BIBREF23 for hierarchical matching by adding shortcut connections between two BiLSTM layers  We proposed two ways of such Hierarchical Residual Matching: (1) Connecting each $\mathbf {\gamma }^{(1)}_i$ and $\mathbf {\gamma }^{(2)}_i$ , resulting in a $\mathbf {\gamma }^{^{\prime }}_i=\mathbf {\gamma }^{(1)}_i + \mathbf {\gamma }^{(2)}_i$ for each position $i$   Then the final question representation $\mathbf {h}^q$ becomes a max-pooling over all $\mathbf {\gamma }^{^{\prime }}_i$ s, 1 $\le $ i $\le $ $N$   (2) Applying max-pooling on $\mathbf {\Gamma }^{(1)}_{1:N}$ and $\mathbf {\gamma }^{(2)}_i$0 to get $\mathbf {\gamma }^{(2)}_i$1 and $\mathbf {\gamma }^{(2)}_i$2 , respectively, then setting $\mathbf {\gamma }^{(2)}_i$3   Finally we compute the matching score of $\mathbf {\gamma }^{(2)}_i$4 given $\mathbf {\gamma }^{(2)}_i$5 as $\mathbf {\gamma }^{(2)}_i$6  
Intuitively, the proposed method should benefit from hierarchical training since the second layer is fitting the residues from the first layer of matching, so the two layers of representations are more likely to be complementary to each other  This also ensures the vector spaces of two layers are comparable and makes the second-layer training easier 
During training we adopt a ranking loss to maximizing the margin between the gold relation $\mathbf {r}^+$ and other relations $\mathbf {r}^-$ in the candidate pool $R$  
$$l_{\mathrm {rel}} = \max \lbrace 0, \gamma - s_{\mathrm {rel}}(\mathbf {r}^+; \mathbf {q}) + s_{\mathrm {rel}}(\mathbf {r}^-; \mathbf {q})\rbrace  \nonumber $$   (Eq  12)
where $\gamma $ is a constant parameter  Fig 2 summarizes the above Hierarchical Residual BiLSTM (HR-BiLSTM) model 
Another way of hierarchical matching consists in relying on attention mechanism, e g  BIBREF24 , to find the correspondence between different levels of representations  This performs below the HR-BiLSTM (see Table 2 ) 
KBQA Enhanced by Relation Detection
This section describes our KBQA pipeline system  We make minimal efforts beyond the training of the relation detection model, making the whole system easy to build 
Following previous work BIBREF4 , BIBREF5 , our KBQA system takes an existing entity linker to produce the top- $K$ linked entities, $EL_K(q)$ , for a question $q$ (“initial entity linking”)  Then we generate the KB queries for $q$ following the four steps illustrated in Algorithm "KBQA Enhanced by Relation Detection"  
[htbp] InputInput OutputOutput Top query tuple $(\hat{e},\hat{r}, \lbrace (c, r_c)\rbrace )$ Entity Re-Ranking (first-step relation detection): Use the raw question text as input for a relation detector to score all relations in the KB that are associated to the entities in $EL_K(q)$ ; use the relation scores to re-rank $EL_K(q)$ and generate a shorter list $EL^{\prime }_{K^{\prime }}(q)$ containing the top- $K^{\prime }$ entity candidates (Section "Entity Re-Ranking" ) Relation Detection: Detect relation(s) using the reformatted question text in which the topic entity is replaced by a special token $<$ e $>$ (Section "Relation Detection" ) Query Generation: Combine the scores from step 1 and 2, and select the top pair $(\hat{e},\hat{r})$ (Section "Query Generation" ) Constraint Detection (optional): Compute similarity between $q$ and any neighbor entity $c$ of the entities along $EL_K(q)$0 (connecting by a relation $EL_K(q)$1 ) , add the high scoring $EL_K(q)$2 and $EL_K(q)$3 to the query (Section "Constraint Detection" )  KBQA with two-step relation detection
Compared to previous approaches, the main difference is that we have an additional entity re-ranking step after the initial entity linking  We have this step because we have observed that entity linking sometimes becomes a bottleneck in KBQA systems  For example, on SimpleQuestions the best reported linker could only get 72 7% top-1 accuracy on identifying topic entities  This is usually due to the ambiguities of entity names, e g  in Fig 1 (a), there are TV writer and baseball player “Mike Kelley”, which is impossible to distinguish with only entity name matching 
Having observed that different entity candidates usually connect to different relations, here we propose to help entity disambiguation in the initial entity linking with relations detected in questions 
Sections "Entity Re-Ranking" and "Relation Detection" elaborate how our relation detection help to re-rank entities in the initial entity linking, and then those re-ranked entities enable more accurate relation detection  The KBQA end task, as a result, benefits from this process 
Entity Re-Ranking
In this step, we use the raw question text as input for a relation detector to score all relations in the KB with connections to at least one of the entity candidates in $EL_K(q)$   We call this step relation detection on entity set since it does not work on a single topic entity as the usual settings  We use the HR-BiLSTM as described in Sec  "Improved KB Relation Detection"   For each question $q$ , after generating a score $s_{rel}(r;q)$ for each relation using HR-BiLSTM, we use the top $l$ best scoring relations ( $R^{l}_q$ ) to re-rank the original entity candidates  Concretely, for each entity $e$ and its associated relations $R_e$ , given the original entity linker score $s_{linker}$ , and the score of the most confident relation $r\in R_q^{l} \cap R_e$ , we sum these two scores to re-rank the entities:
$$s_{\mathrm {rerank}}(e;q) =& \alpha \cdot s_{\mathrm {linker}}(e;q) \nonumber \\ + & (1-\alpha ) \cdot \max _{r \in R_q^{l} \cap R_e} s_{\mathrm {rel}}(r;q) \nonumber $$   (Eq

ground_truth: a sample of  29,794 wikipedia articles and 2,794 arXiv papers 

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What is te core component for KBQA?
Context 2:  15)
Finally, we select top $K^{\prime }$ $<$ $K$ entities according to score $s_{rerank}$ to form the re-ranked list $EL_{K^{\prime }}^{^{\prime }}(q)$  
We use the same example in Fig 1 (a) to illustrate the idea  Given the input question in the example, a relation detector is very likely to assign high scores to relations such as “episodes_written”, “author_of” and “profession”  Then, according to the connections of entity candidates in KB, we find that the TV writer “Mike Kelley” will be scored higher than the baseball player “Mike Kelley”, because the former has the relations “episodes_written” and “profession”  This method can be viewed as exploiting entity-relation collocation for entity linking 
Relation Detection
In this step, for each candidate entity $e \in EL_K^{\prime }(q)$ , we use the question text as the input to a relation detector to score all the relations $r \in R_e$ that are associated to the entity $e$ in the KB  Because we have a single topic entity input in this step, we do the following question reformatting: we replace the the candidate $e$ 's entity mention in $q$ with a token “ $<$ e $>$ ”  This helps the model better distinguish the relative position of each word compared to the entity  We use the HR-BiLSTM model to predict the score of each relation $r \in R_e$ : $s_{rel} (r;e,q)$  
Query Generation
Finally, the system outputs the $<$ entity, relation (or core-chain) $>$ pair $(\hat{e}, \hat{r})$ according to:
$$s(\hat{e}, \hat{r}; q) =& \max _{e \in EL_{K^{\prime }}^{^{\prime }}(q), r \in R_e} \left( \beta \cdot s_{\mathrm {rerank}}(e;q) \right  \nonumber \\ &\left + (1-\beta ) \cdot s_{\mathrm {rel}} (r;e,q) \right), \nonumber $$   (Eq  19)
where $\beta $ is a hyperparameter to be tuned 
Constraint Detection
Similar to BIBREF4 , we adopt an additional constraint detection step based on text matching  Our method can be viewed as entity-linking on a KB sub-graph  It contains two steps: (1) Sub-graph generation: given the top scored query generated by the previous 3 steps, for each node $v$ (answer node or the CVT node like in Figure 1 (b)), we collect all the nodes $c$ connecting to $v$ (with relation $r_c$ ) with any relation, and generate a sub-graph associated to the original query  (2) Entity-linking on sub-graph nodes: we compute a matching score between each $n$ -gram in the input question (without overlapping the topic entity) and entity name of $c$ (except for the node in the original query) by taking into account the maximum overlapping sequence of characters between them (see Appendix A for details and B for special rules dealing with date/answer type constraints)  If the matching score is larger than a threshold $\theta $ (tuned on training set), we will add the constraint entity $c$ (and $r_c$ ) to the query by attaching it to the corresponding node $v$ on the core-chain 
Experiments
Task Introduction & Settings
We use the SimpleQuestions BIBREF2 and WebQSP BIBREF25 datasets  Each question in these datasets is labeled with the gold semantic parse  Hence we can directly evaluate relation detection performance independently as well as evaluate on the KBQA end task 
SimpleQuestions (SQ): It is a single-relation KBQA task  The KB we use consists of a Freebase subset with 2M entities (FB2M) BIBREF2 , in order to compare with previous research  yin2016simple also evaluated their relation extractor on this data set and released their proposed question-relation pairs, so we run our relation detection model on their data set  For the KBQA evaluation, we also start with their entity linking results  Therefore, our results can be compared with their reported results on both tasks 
WebQSP (WQ): A multi-relation KBQA task  We use the entire Freebase KB for evaluation purposes  Following yih-EtAl:2016:P16-2, we use S-MART BIBREF26 entity-linking outputs  In order to evaluate the relation detection models, we create a new relation detection task from the WebQSP data set  For each question and its labeled semantic parse: (1) we first select the topic entity from the parse; and then (2) select all the relations and relation chains (length $\le $ 2) connected to the topic entity, and set the core-chain labeled in the parse as the positive label and all the others as the negative examples 
We tune the following hyper-parameters on development sets: (1) the size of hidden states for LSTMs ({50, 100, 200, 400}); (2) learning rate ({0 1, 0 5, 1 0, 2 0}); (3) whether the shortcut connections are between hidden states or between max-pooling results (see Section "Hierarchical Matching between Relation and Question" ); and (4) the number of training epochs 
For both the relation detection experiments and the second-step relation detection in KBQA, we have entity replacement first (see Section "Relation Detection" and Figure 1 )  All word vectors are initialized with 300- $d$ pretrained word embeddings BIBREF27   The embeddings of relation names are randomly initialized, since existing pre-trained relation embeddings (e g  TransE) usually support limited sets of relation names  We leave the usage of pre-trained relation embeddings to future work 
Relation Detection Results
Table 2 shows the results on two relation detection tasks  The AMPCNN result is from BIBREF20 , which yielded state-of-the-art scores by outperforming several attention-based methods  We re-implemented the BiCNN model from BIBREF4 , where both questions and relations are represented with the word hash trick on character tri-grams  The baseline BiLSTM with relation word sequence appears to be the best baseline on WebQSP and is close to the previous best result of AMPCNN on SimpleQuestions  Our proposed HR-BiLSTM outperformed the best baselines on both tasks by margins of 2-3% (p $<$ 0 001 and 0 01 compared to the best baseline BiLSTM w/ words on SQ and WQ respectively) 
Note that using only relation names instead of words results in a weaker baseline BiLSTM model  The model yields a significant performance drop on SimpleQuestions (91 2% to 88 9%)  However, the drop is much smaller on WebQSP, and it suggests that unseen relations have a much bigger impact on SimpleQuestions 
The bottom of Table 2 shows ablation results of the proposed HR-BiLSTM  First, hierarchical matching between questions and both relation names and relation words yields improvement on both datasets, especially for SimpleQuestions (93 3% vs  91 2/88 8%)  Second, residual learning helps hierarchical matching compared to weighted-sum and attention-based baselines (see Section "Hierarchical Matching between Relation and Question" )  For the attention-based baseline, we tried the model from BIBREF24 and its one-way variations, where the one-way model gives better results  Note that residual learning significantly helps on WebQSP (80 65% to 82 53%), while it does not help as much on SimpleQuestions  On SimpleQuestions, even removing the deep layers only causes a small drop in performance  WebQSP benefits more from residual and deeper architecture, possibly because in this dataset it is more important to handle larger scope of context matching 
Finally, on WebQSP, replacing BiLSTM with CNN in our hierarchical matching framework results in a large performance drop  Yet on SimpleQuestions the gap is much smaller  We believe this is because the LSTM relation encoder can better learn the composition of chains of relations in WebQSP, as it is better at dealing with longer dependencies 
Next, we present empirical evidences, which show why our HR-BiLSTM model achieves the best scores  We use WebQSP for the analysis purposes  First, we have the hypothesis that training of the weighted-sum model usually falls to local optima, since deep BiLSTMs do not guarantee that the two-levels of question hidden representations are comparable  This is evidenced by that during training one layer usually gets a weight close to 0 thus is ignored  For example, one run gives us weights of -75 39/0 14 for the two layers (we take exponential for the final weighted sum)  It also gives much lower training accuracy (91 94%) compared to HR-BiLSTM (95 67%), suffering from training difficulty 
Second, compared to our deep BiLSTM with shortcut connections, we have the hypothesis that for KB relation detection, training deep BiLSTMs is more difficult without shortcut connections  Our experiments suggest that deeper BiLSTM does not always result in lower training accuracy  In the experiments a two-layer BiLSTM converges to 94 99%, even lower than the 95 25% achieved by a single-layer BiLSTM  Under our setting the two-layer model captures the single-layer model as a special case (so it could potentially better fit the training data), this result suggests that the deep BiLSTM without shortcut connections might suffers more from training difficulty 
Finally, we hypothesize that HR-BiLSTM is more than combination of two BiLSTMs with residual connections, because it encourages the hierarchical architecture to learn different levels of abstraction  To verify this, we replace the deep BiLSTM question encoder with two single-layer BiLSTMs (both on words) with shortcut connections between their hidden states  This decreases test accuracy to 76 11%  It gives similar training accuracy compared to HR-BiLSTM, indicating a more serious over-fitting problem  This proves that the residual and deep structures both contribute to the good performance of HR-BiLSTM 
KBQA End-Task Results
Table 3 compares our system with two published baselines (1) STAGG BIBREF4 , the state-of-the-art on WebQSP and (2) AMPCNN BIBREF20 , the state-of-the-art on SimpleQuestions  Since these two baselines are specially designed/tuned for one particular dataset, they do not generalize well when applied to the other dataset  In order to highlight the effect of different relation detection models on the KBQA end-task, we also implemented another baseline that uses our KBQA system but replaces HR-BiLSTM with our implementation of AMPCNN (for SimpleQuestions) or the char-3-gram BiCNN (for WebQSP) relation detectors (second block in Table 3 ) 
Compared to the baseline relation detector (3rd row of results), our method, which includes an improved relation detector (HR-BiLSTM), improves the KBQA end task by 2-3% (4th row)  Note that in contrast to previous KBQA systems, our system does not use joint-inference or feature-based re-ranking step, nevertheless it still achieves better or comparable results to the state-of-the-art 
The third block of the table details two ablation tests for the proposed components in our KBQA systems: (1) Removing the entity re-ranking step significantly decreases the scores  Since the re-ranking step relies on the relation detection models, this shows that our HR-BiLSTM model contributes to the good performance in multiple ways  Appendix C gives the detailed performance of the re-ranking step  (2) In contrast to the conclusion in BIBREF4 , constraint detection is crucial for our system  This is probably because our joint performance on topic entity and core-chain detection is more accurate (77 5% top-1 accuracy), leaving a huge potential (77 5% vs  58 0%) for the constraint detection module to improve 
Finally, like STAGG, which uses multiple relation detectors (see yih2015semantic for the three models used), we also try to use the top-3 relation detectors from Section "Relation Detection Results"   As shown on the last row of Table 3 , this gives a significant performance boost, resulting in a new state-of-the-art result on SimpleQuestions and a result comparable to the state-of-the-art on WebQSP 
Conclusion
KB relation detection is a key step in KBQA and is significantly different from general relation extraction tasks  We propose a novel KB relation detection model, HR-BiLSTM, that performs hierarchical matching between questions and KB relations  Our model outperforms the previous methods on KB relation detection tasks and allows our KBQA system to achieve state-of-the-arts  For future work, we will investigate the integration of our HR-BiLSTM into end-to-end systems  For example, our model could be integrated into the decoder in BIBREF31 , to provide better sequence prediction  We will also investigate new emerging datasets like GraphQuestions BIBREF32 and ComplexQuestions BIBREF30 to handle more characteristics of general QA 
Context 3: Introduction
Recently, people have started looking at online forums either as a primary or secondary source of counseling services BIBREF0  BIBREF1 reported that over the first five years of operation (2011-2016), ReachOut com – Ireland's online youth mental health service – 62% of young people would visit a website for support when going through a tough time  With the expansion of the Internet, there has been a substantial growth in the number of users looking for psychological support online 
The importance of the on-line life of patients has been recognized in research as well  BIBREF2 stated that the online life of patients constitutes a major influence on their self-definition  Furthermore, according to BIBREF3, the social networking activities of an individual, offer an important reflection of their personality  While dealing with patients suffering from psychological problems, it is important that therapists do not ignore this pivotal source of information which can provide deep insights into their patients' mental conditions 
Acceptance of on-line support groups (OSG) by Mental Health Professionals is still not established BIBREF4  Since OSG can have double-edged effects on patients and the presence of professionals is often limited, we argue that their properties should be further studied  According to BIBREF5 OSG effectiveness is hard to assess, while some studies showed OSG's potential to change participants' attitudes, no such effect was observed in other studies (see Related Work Section for more details)  Furthermore the scope of previous work on analysis of users' behaviour in OSG has been limited by the fact that they relied on expert annotation of posts and comments BIBREF6 
We present a novel approach for automatically analysing online conversations for the presence of therapeutic factors of group therapy defined by BIBREF7 as “the actual mechanisms of effecting change in the patient”  The authors have identified 11 therapeutic factors in group therapy: Universality, Altruism, Instillation of Hope, Guidance, Imparting information, Developing social skills, Interpersonal learning, Cohesion, Catharsis, Existential factors, Imitative behavior and Corrective recapitulation of family of origin issues  In this paper, we focus on 3 therapeutic factors: Universality, Altruism and Instillation of Hope (listed below), as we believe that these can be approximated by using established NLP techniques (e g  Sentiment Analysis, Dialogue Act tagging etc ) 
Universality: the disconfirmation of a user's feelings of uniqueness of their mental health condition 
Altruism: others offer support, reassurance, suggestions and insight 
Instillation of Hope: inspiration provided to participants by their peers 
The selected therapeutic factors are analysed in terms of illocutionary force and attitude  Due to the multi-party and asynchronous nature of on-line social media conversations, prior to the analysis, we extract conversation threads among users – an essential prerequisite for any kind of higher-level dialogue analysis BIBREF10  Afterwards, the illocutionary force is identified using Dialogue Act tagging, whereas the attitude by using Sentiment Analysis  The quantitative analysis is then performed on these processed conversations 
Ideally, the analysis would require experts to annotate each post and comment on the presence of therapeutic factors  However, due to time and cost demands of this task, it is feasible to analyse only a small fraction of the available data  Compared to previous studies (e g  BIBREF6) that analysed few tens of conversations and several thousand lines of chat; using the proposed approach – application of Dialogue Acts and Sentiment Analysis – we were able to automatically analyse approximately 300 thousands conversations (roughly 1 5 million comments)  The rest of the paper is structured as follows  In Section 2 we introduce related work  Next, in Section 3 we describe the pre-processing pipeline and the methodology to perform thread extraction on asynchronous multi-party conversations  In Section 4 we provide the describe the final dataset used for the analysis, and in Section 5 we present the results of our analysis  Finally, in Section 6 we provide concluding remarks and future research directions 
Related Work
On-line support groups have been analyzed for various factors before  For instance, BIBREF11 analysed stress reduction in on-line support group chat-rooms, and the effects of on-line social interactions  Such studies mostly relied on questionnaires and were based on a small number of users  Nevertheless, in BIBREF11, the author showed that social support facilitates coping with distress, improves mood and expedites recovery from it  These findings highlight that, overall, on-line discussion boards appear to be therapeutic and constructive for individuals suffering alcohol-abuse 
Application of NLP to the analysis of mental health-related conversation has been studied as well (e g  BIBREF12, BIBREF13)  BIBREF6 applied sentiment-analysis combined with extensive turn-level annotation to investigate stress reduction in on-line support group chat-rooms, showing that sentiment-analysis is a good predictor of entrance stress level  Furthermore, similar to our setting, they applied automatic thread-extraction to determine conversation threads 
BIBREF14 have shown that on-line support group therapy increased the quality of life of patients with metastatic breast cancer  Since many original posters reported the benefits of group therapy on patients BIBREF15, BIBREF2, BIBREF16, BIBREF17, BIBREF18, BIBREF7, we evaluate the effect of the user interaction using sentiment scores of comments in on-line support groups 
According to BIBREF6, users with high incoming stress tend to request less information from others, as a percentage of their time, and share much more information, in absolute terms  In addition, high information sharing has been shown to be a good predictor of stress reduction at the end of the chat BIBREF6  Regarding information sharing, we rely on Dialogue Acts BIBREF19 to model the speaker's intention in producing an utterance  In particular, we are interested in Dialogue Act label that is defined to represent descriptive, narrative, or personal information – the statement 
Dialogue Acts have been applied to the analysis of spoken BIBREF20, BIBREF21 as well as on-line written synchronous conversations BIBREF22  We apply Dialogue Act tag set defined in BIBREF22 to the analysis of our on-line asynchronous conversations  We argue that Dialogue Acts can be used to analyse user behaviour in social media and verify the presence of therapeutic factors 
Methodology
We select the three therapeutic factors – Universality, Altruism and Instillation of Hope – that can be best approximated using NLP techniques: Sentiment Analysis and Dialogue Act tagging  We discuss each one of the selected therapeutic factors and the identified necessary conditions  The listed conditions, however, are not sufficient to attribute the presence of a therapeutic factor with high confidence, which only can be obtained using expert annotation  Our analysis focuses on the structure of conversations; though content plays an important role as well 
Universality consists in the disconfirmation of patients' belief of uniqueness of their disease  This therapeutic factor is shown to be a powerful source of relief for the patient, according to BIBREF7  From this definition, we can draw the following conditions that are applicable to our environment:
improvement of original poster's sentiment: we hypothesize that the discovery that other people passed through similar issues leads to a higher sentiment score;
posts containing negative personal experiences: to disconfirm the belief of uniqueness users have to share their story;
comments containing negative statements: to disconfirm the patient's feelings of uniqueness, the commenting user must tell a similar negative personal experience  This condition requires two sub-conditions: high presence of statements in comments and the presence of negative comments replying to negative posts 
Instillation of Hope is based on inspiration provided to participants by their peers  Through the inspiration provided by their peers, patients can increase their expectation on the therapy outcome  BIBREF7 in several studies have demonstrated that a high expectation of help before the start of a therapy is significantly correlated with a positive therapy outcome  The author states that many patients pointed out the importance of having observed the improvement of others  Therefore, the three main conditions are the following:
improvement of original poster's sentiment: we hypothesize that instillation of hope leads to a higher sentiment score;
posts containing negative personal experiences: hope can be instilled in someone who shares a negative personal experience;
comments containing positive personal experiences: in order to instill hope, commenting posters must show to original posters an overall positive personal experience  To detect positive personal experience, we require the presence of statements in comments and a positive sentiment of comments replying to negative posts 
Altruism consists of peers offering support, reassurance, suggestions and insight, since they share similar problems with one another BIBREF7  The experience of finding that a patient can be of value to others is refreshing and boosts self-esteem BIBREF7  However, in the current study we focus on testing whether commenting posters are altruists or not  We do not test whether the altruistic behavior leads to an improvement on the altruist itself  For these reasons, we define three main conditions:
improvement of original poster's sentiment: we hypothesize that supportive and reassuring statements improve the sentiment score of the original poster;
posts contains negative personal experiences: users offer support, reassurance and suggestion when facing a negative personal experience of the original poster;
comments containing positive statements: either supportive or reassuring statements show by definition a positive intended emotional communication  Thus comments to the post should consist of positive sentiment statements 
Consequently, a conversation containing the aforementioned therapeutic factors should satisfy the following conditions in terms of NLP: Sentiment Analysis and Dialogue Acts 
original posters have a higher sentiment score at the end of the thread than at the beginning;
the original post consists mostly of polarised statements;
the presence of a significant amount of statements in comments, since both support and sharing similar negative experiences can be represented as statements;
both negative and positive statements in comments lead to higher final sentiment score of the original poster 
Datasets
We verify the presence of therapeutic factors in two social media datasets: OSG and Twitter  The first dataset is crawled from an on-line support groups website, and the second dataset consists of a small sample of Twitter conversation threads  Since the former consists of multi-threaded conversations, we apply a pre-processing to extract conversation threads to provide a fair comparison with the Twitter dataset  An example conversation from each data source is presented in Figure FIGREF19 
Datasets ::: Twitter
We have downloaded 1,873 Twitter conversation threads, roughly 14k tweets, from a publicly available resource that were previously pre-processed and have conversation threads extracted  A conversation in the dataset consists of at least 4 tweets  Even though, according to BIBREF23, Twitter is broadly applicable to public health research, our expectation is that it contains less therapeutic conversations in comparison to specialized on-line support forums 
Datasets ::: OSG
Our data has been developed by crawling and pre-processing an OSG web forum  The forum has a great variety of different groups such as depression, anxiety, stress, relationship, cancer, sexually transmitted diseases, etc  Each conversation starts with one post and can contain multiple comments  Each post or comment is represented by a poster, a timestamp, a list of users it is referencing to, thread id, a comment id and a conversation id  The thread id is the same for comments replying to each other, otherwise it is different  The thread id is increasing with time  Thus, it provides ordering among threads; whereas the timestamp provides ordering in the thread 
Each conversation can belong to multiple groups  Consequently, the dataset needs to be processed to remove duplicates  The dataset resulting after de-duplication contains 295 thousand conversations, each conversation contains on average 6 comments  In total, there are 1 5 million comments  Since the created dataset is multi-threaded, we need to extract conversation threads, to eliminate paths not relevant to the original post 
Datasets ::: OSG ::: Conversation Thread Extraction
The thread extraction algorithm is heuristic-based and consists of two steps: (1) creation of a tree, based on a post written by a user and the related comments and (2) transformation of the tree into a list of threads 
The tree creation is an extension of the approach of BIBREF24, where first a graph of conversation is constructed  In the approach, direct replies to a post are attached to the first nesting level and subsequent comments to increasing nesting levels  In our approach, we also exploit comments' features

ground_truth: answer questions by obtaining information from KB tuples 

hierarchical matching between questions and relations with residual learning

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How large is the Twitter dataset?
Context 3: 
The tree creation is performed without processing the content of comments, which allows us to process posts and comments of any length efficiently  The heuristic used in the process is based on three simplifying assumptions:
Unless there is a specific reference to another comment or a user, comments are attached to the original post 
When replying, the commenting poster is always replying to the original post or some other comment  Unless specified otherwise, it is assumed that it is a response to the previous (in time) post/comment 
Subsequent comments by the same poster are part of the same thread 
To evaluate the performance of the thread extraction algorithm, 2 annotators have manually constructed the trees for 100 conversations  The performance of the algorithm on this set of 100 conversations is evaluated using accuracy and standard Information Retrieval evaluation metrics of precision, recall, and F$_1$ measure  The results are reported in Table TABREF28 together with random and majority baselines  The turn-level percent agreement between the 2 annotators is 97 99% and Cohen's Kappa Coefficient is 83 80% 
Datasets ::: Data Representation
For both data sources, Twitter and OSG with extracted threads, posts and comments are tokenized and sentence split  Each sentence is passed through Sentiment Analysis and Dialogue Act tagging  Since a post or a comment can contain multiple sentences, therefore multiple Dialogue Acts, it is represented as as a one-hot encoding, where each position represents a Dialogue Act 
For Sentiment Analysis we use a lexicon-based sentiment analyser introduced by BIBREF25  For Dialogue Act tagging, on the other hand, we make use of a model trained on NPSChat corpus BIBREF22 following the approach of BIBREF26 
Analysis
As we mentioned in Section 3, the presence of each of the therapeutic conditions under analysis is a necessary for a conversation to be considered to have therapeutic factors  In this section we present the results of our analysis with respect to these conditions 
Analysis ::: Change in Sentiment score of Original Posters
The first condition which we test is the sentiment change in conversation threads, comparing the initial and final sentiment scores (i e  posts' scores) of the original poster  The results of the analysis are presented in Figure FIGREF33  In the figure we can observe that the distribution of the sentiment change in the two datasets is different  While in Twitter the amount of conversations that lead to the increase of sentiment score is roughly equal to the amount of conversations that lead to the decrease of sentiment score; the situation is different for OSG  In OSG, the amount of conversations that lead to the increase of sentiment score is considerably higher 
Figure FIGREF34 provides a more fine grained analysis, where we additionally analyse the sentiment change in nominal polarity terms – negative and positive  In OSG, the number of users that changed polarity from negative to positive is more than the double of the users that have changed the polarity from positive to negative  In Twitter, on the other hand, the users mostly changed polarity from positive to negative  Results of the analysis suggest that in OSG , sentiment increases and users tend to change polarity from negative to positive, whereas in Twitter sentiment tends to decrease  Verification of this condition alone indicates that the ratio of potentially therapeutic conversations in Twitter is lower 
Analysis ::: Structure of Posts and Comments
Table TABREF36 presents the distribution of automatically predicted per-sentence Dialogue Acts in the datasets  The most frequent tag is statement in both  In Table TABREF37, on the other hand, we present the distribution of post and comment structures in terms of automatically predicted Dialogue Act tags  The structure is an unordered set of tags in the post or comment  From the table we can observe that the distribution of tag sets is similar between posts and comments  In both cases the most common set is statement only  However, conversations containing only statement, emphasis or question posts and comments predominantly appear in Twitter  Which is expected due to the shorter length of Twitter posts and comments 
We can also observe that the original posters tend to ask more questions than the commenting posters – 19 83% for posts vs  11 21% for comments (summed)  This suggests that the original posters frequently ask either for suggestion or confirmation of their points of view or their disconfirmation  However, the high presence of personal experiences is supported by the high number of posts containing only statements 
High number of statement tags in comments suggests that users reply either with supporting or empathic statements or personal experience  However, 6 39% of comments contain accept and reject tags, which mark the degree to which a speaker accepts some previous proposal, plan, opinion, or statement BIBREF20  The described Dialogue Act tags are often used when commenting posters discuss original poster's point of view  For instance, “It's true  I felt the same ” – {Accept, Statement} or “Well no  You're not alone” – {Reject, Statement}  The datasets differ with respect to the distribution of these Dialogue Acts tags, they appear more frequently in OSG 
Analysis ::: Sentiment of Posts and Comments
Table TABREF39 presents the distribution of sentiment polarity in post and comment statements (i e  sentences tagged as statement)  For OSG, the predominant sentiment label of statements is positive and it is the highest for both posts and comments  However, the difference between the amounts of positive and negative statements is higher for the replying comments (34 5% vs  42 5%)  For Twitter, on the other hand, the predominant sentiment label of statements is neutral and the polarity distribution between posts and comments is very close  One particular observation is that the ratio of negative statements is higher in OSG for both posts and comments than in Twitter, which supports the idea of sharing negative experiences 
Further we analyze whether the sentiment of a comment (i e  the replying user) is affected by the sentiment of the original post (i e  the user being replied to), which will imply that the users adapt their behaviour with respect to the post's sentiment  For the analysis, we split the datasets into three buckets according to the posts' sentiment score – negative, neutral, or positive, and represent each conversation in terms of percentages of comments (replies) with each sentiment label  The buckets are then compared using t-test for statistically significant differences 
Table TABREF40 presents the distribution of sentiment labels with respect to the post's sentiment score  The patterns of distribution are similar across the datasets  We can observe that overall, replies tend to have a positive sentiment, which suggests that replying posters tend to have a positive attitude  However, the ratio of positive comments is higher for OSG than for Twitter 
The results of the Welch's t-test on OSG data reveal that there are statistically significant differences in the distribution of replying comments' sentiment between conversations with positive and negative starting posts  A positive post tends to get significantly more positive replies  Similarly, a negative post tends to get significantly more negative replies (both with $p < 0 01$) 
Table TABREF41 presents the distribution of the sentiment labels of the final text provided by the original poster with respect to the sentiment polarity of the comments  The results indicate that OSG participants are more supportive, as the majority of conversations end in a positive final sentiment regardless of the sentiment of comments  We can also observe that negative comments in OSG lead to positive sentiment, which supports the idea of sharing the negative experiences, thus presence of therapeutic factors  For Twitter, on the other hand, only positive comments lead to the positive final sentiments, whereas other comments lead predominantly to neutral final sentiments 
Our analysis in terms of sentiment and Dialogue Acts supports the presence of the three selected therapeutic factors – Universality, Altruism and Instillation of Hope – in OSG more than in Twitter  The main contributors to this conclusion are the facts that there is more positive change in the sentiment of the original posters in OSG (people seeking support) and that in OSG even negative and neutral comments are likely to lead to positive changes 
Conclusion
In this work, we propose a methodology to automatically analyse online social platforms for the presence of therapeutic factors (i e  Universality, Altruism and Instillation of Hope)  We evaluate our approach on two on-line platforms, Twitter and an OSG web forum  We apply NLP techniques of Sentiment Analysis and Dialogue Act tagging to automatically verify the presence of therapeutic factors, which allows us to analyse larger amounts of conversational data (as compared to previous studies) 
Our analysis indicates that OSG conversations satisfy higher number of conditions approximating therapeutic factors than Twitter conversations  Given this outcome, we postulate that users who join support group websites spontaneously seem to benefit from it  Indeed, as shown in Section SECREF5, the original posters who interact with others by replying to comments, have benefited from an improvement of their emotional state 
We would like to reemphasise that the conditions for the therapeutic factors are necessary but not sufficient; since our analysis focuses on the structure of conversations, being agnostic to the content  NLP, however, allows us to strengthen our approximations even further  Thus, the further extension of our work is also augmentation of our study with other language analysis metrics and their correlation with human annotation 
It should be noted that the proposed approach is an approximation of the tedious tasks of annotation of conversations by experts versed in the therapeutic factors and their associated theories  Even though we can use Sentiment Analysis to detect the existence of therapeutic factors, we cannot differentiate between Altruism and Instillation of Hope, as this requires differentiation between emotional state of the user and the intended emotional communication  Thus, the natural extensions of this work are differentiation between different therapeutic factors and comparison of the proposed analysis to the human evaluation 
Although we acknowledge that the proposed methodology does not serve as a replacement of manual analysis of OSG for the presence of therapeutic factors, we believe that it could facilitate and supplement this process  The method can serve as a tool for general practitioners and psychologists who can use it as an additional source of information regarding their patients’ condition and, in turn, offer a more personalised support that is better tailored to individual therapeutic needs 

ground_truth: 1,873 Twitter conversation threads, roughly 14k tweets

1,873 Twitter conversation threads, roughly 14k tweets