prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How does Gaussian-masked directional multi-head attention works?
Context 1: Introduction
Chinese word segmentation (CWS) is a task for Chinese natural language process to delimit word boundary  CWS is a basic and essential task for Chinese which is written without explicit word delimiters and different from alphabetical languages like English  BIBREF0 treats Chinese word segmentation (CWS) as a sequence labeling task with character position tags, which is followed by BIBREF1, BIBREF2, BIBREF3  Traditional CWS models depend on the design of features heavily which effects the performance of model  To minimize the effort in feature engineering, some CWS models BIBREF4, BIBREF5, BIBREF6, BIBREF7, BIBREF8, BIBREF9, BIBREF10, BIBREF11 are developed following neural network architecture for sequence labeling tasks BIBREF12  Neural CWS models perform strong ability of feature representation, employing unigram and bigram character embedding as input and approach good performance 
The CWS task is often modeled as one graph model based on a scoring model that means it is composed of two parts, one part is an encoder which is used to generate the representation of characters from the input sequence, the other part is a decoder which performs segmentation according to the encoder scoring  Table TABREF1 summarizes typical CWS models according to their decoding ways for both traditional and neural models  Markov models such as BIBREF13 and BIBREF4 depend on the maximum entropy model or maximum entropy Markov model both with a Viterbi decoder  Besides, conditional random field (CRF) or Semi-CRF for sequence labeling has been used for both traditional and neural models though with different representations BIBREF2, BIBREF15, BIBREF10, BIBREF17, BIBREF18  Generally speaking, the major difference between traditional and neural network models is about the way to represent input sentences 
Recent works about neural CWS which focus on benchmark dataset, namely SIGHAN Bakeoff BIBREF21, may be put into the following three categories roughly 
Encoder  Practice in various natural language processing tasks has been shown that effective representation is essential to the performance improvement  Thus for better CWS, it is crucial to encode the input character, word or sentence into effective representation  Table TABREF2 summarizes regular feature sets for typical CWS models including ours as well  The building blocks that encoders use include recurrent neural network (RNN) and convolutional neural network (CNN), and long-term memory network (LSTM) 
Graph model  As CWS is a kind of structure learning task, the graph model determines which type of decoder should be adopted for segmentation, also it may limit the capability of defining feature, as shown in Table 2, not all graph models can support the word features  Thus recent work focused on finding more general or flexible graph model to make model learn the representation of segmentation more effective as BIBREF9, BIBREF11 
External data and pre-trained embedding  Whereas both encoder and graph model are about exploring a way to get better performance only by improving the model strength itself  Using external resource such as pre-trained embeddings or language representation is an alternative for the same purpose BIBREF22, BIBREF23  SIGHAN Bakeoff defines two types of evaluation settings, closed test limits all the data for learning should not be beyond the given training set, while open test does not take this limitation BIBREF21  In this work, we will focus on the closed test setting by finding a better model design for further CWS performance improvement 
Shown in Table TABREF1, different decoders have particular decoding algorithms to match the respective CWS models  Markov models and CRF-based models often use Viterbi decoders with polynomial time complexity  In general graph model, search space may be too large for model to search  Thus it forces graph models to use an approximate beam search strategy  Beam search algorithm has a kind low-order polynomial time complexity  Especially, when beam width $b$=1, the beam search algorithm will reduce to greedy algorithm with a better time complexity $O(Mn)$ against the general beam search time complexity $O(Mnb^2)$, where $n$ is the number of units in one sentences, $M$ is a constant representing the model complexity  Greedy decoding algorithm can bring the fastest speed of decoding while it is not easy to guarantee the precision of decoding when the encoder is not strong enough 
In this paper, we focus on more effective encoder design which is capable of offering fast and accurate Chinese word segmentation with only unigram feature and greedy decoding  Our proposed encoder will only consist of attention mechanisms as building blocks but nothing else  Motivated by the Transformer BIBREF24 and its strength of capturing long-range dependencies of input sentences, we use a self-attention network to generate the representation of input which makes the model encode sentences at once without feeding input iteratively  Considering the weakness of the Transformer to model relative and absolute position information directly BIBREF25 and the importance of localness information, position information and directional information for CWS, we further improve the architecture of standard multi-head self-attention of the Transformer with a directional Gaussian mask and get a variant called Gaussian-masked directional multi-head attention  Based on the newly improved attention mechanism, we expand the encoder of the Transformer to capture different directional information  With our powerful encoder, our model uses only simple unigram features to generate representation of sentences 
For decoder which directly performs the segmentation, we use the bi-affinal attention scorer, which has been used in dependency parsing BIBREF26 and semantic role labeling BIBREF27, to implement greedy decoding on finding the boundaries of words  In our proposed model, greedy decoding ensures a fast segmentation while powerful encoder design ensures a good enough segmentation performance even working with greedy decoder together  Our model will be strictly evaluated on benchmark datasets from SIGHAN Bakeoff shared task on CWS in terms of closed test setting, and the experimental results show that our proposed model achieves new state-of-the-art 
The technical contributions of this paper can be summarized as follows 
We propose a CWS model with only attention structure  The encoder and decoder are both based on attention structure 
With a powerful enough encoder, we for the first time show that unigram (character) featues can help yield strong performance instead of diverse $n$-gram (character and word) features in most of previous work 
To capture the representation of localness information and directional information, we propose a variant of directional multi-head self-attention to further enhance the state-of-the-art Transformer encoder 
Models
The CWS task is often modelled as one graph model based on an encoder-based scoring model  The model for CWS task is composed of an encoder to represent the input and a decoder based on the encoder to perform actual segmentation  Figure FIGREF6 is the architecture of our model  The model feeds sentence into encoder  Embedding captures the vector $e=(e_1,   ,e_n)$ of the input character sequences of $c=(c_1,   ,c_n)$  The encoder maps vector sequences of $ {e}=(e_1,  ,e_n)$ to two sequences of vector which are $ {v^b}=(v_1^b,   ,v_n^b)$ and ${v^f}=(v_1^f,   v_n^f)$ as the representation of sentences  With $v^b$ and $v^f$, the bi-affinal scorer calculates the probability of each segmentation gaps and predicts the word boundaries of input  Similar as the Transformer, the encoder is an attention network with stacked self-attention and point-wise, fully connected layers while our encoder includes three independent directional encoders 
Models ::: Encoder Stacks
In the Transformer, the encoder is composed of a stack of N identical layers and each layer has one multi-head self-attention layer and one position-wise fully connected feed-forward layer  One residual connection is around two sub-layers and followed by layer normalization BIBREF24  This architecture provides the Transformer a good ability to generate representation of sentence 
With the variant of multi-head self-attention, we design a Gaussian-masked directional encoder to capture representation of different directions to improve the ability of capturing the localness information and position information for the importance of adjacent characters  One unidirectional encoder can capture information of one particular direction 
For CWS tasks, one gap of characters, which is from a word boundary, can divide one sequence into two parts, one part in front of the gap and one part in the rear of it  The forward encoder and backward encoder are used to capture information of two directions which correspond to two parts divided by the gap 
One central encoder is paralleled with forward and backward encoders to capture the information of entire sentences  The central encoder is a special directional encoder for forward and backward information of sentences  The central encoder can fuse the information and enable the encoder to capture the global information 
The encoder outputs one forward information and one backward information of each positions  The representation of sentence generated by center encoder will be added to these information directly:
where $v^{b}=(v^b_1,   ,v^b_n)$ is the backward information, $v^{f}=(v^f_1,   ,v^f_n)$ is the forward information, $r^{b}=(r^b_1,   ,r^b_n)$ is the output of backward encoder, $r^{c}=(r^c_1,   ,r^c_n)$ is the output of center encoder and $r^{f}=(r^f_1,   ,r^f_n)$ is the output of forward encoder 
Models ::: Gaussian-Masked Directional Multi-Head Attention
Similar as scaled dot-product attention BIBREF24, Gaussian-masked directional attention can be described as a function to map queries and key-value pairs to the representation of input  Here queries, keys and values are all vectors  Standard scaled dot-product attention is calculated by dotting query $Q$ with all keys $K$, dividing each values by $\sqrt{d_k}$, where $\sqrt{d_k}$ is the dimension of keys, and apply a softmax function to generate the weights in the attention:
Different from scaled dot-product attention, Gaussian-masked directional attention expects to pay attention to the adjacent characters of each positions and cast the localness relationship between characters as a fix Gaussian weight for attention  We assume that the Gaussian weight only relys on the distance between characters 
Firstly we introduce the Gaussian weight matrix $G$ which presents the localness relationship between each two characters:
where $g_{ij}$ is the Gaussian weight between character $i$ and $j$, $dis_{ij}$ is the distance between character $i$ and $j$, $\Phi (x)$ is the cumulative distribution function of Gaussian, $\sigma $ is the standard deviation of Gaussian function and it is a hyperparameter in our method  Equation (DISPLAY_FORM13) can ensure the Gaussian weight equals 1 when $dis_{ij}$ is 0  The larger distance between charactersis, the smaller the weight is, which makes one character can affect its adjacent characters more compared with other characters 
To combine the Gaussian weight to the self-attention, we produce the Hadamard product of Gaussian weight matrix $G$ and the score matrix produced by $Q{K^{T}}$
where $AG$ is the Gaussian-masked attention  It ensures that the relationship between two characters with long distances is weaker than adjacent characters 
The scaled dot-product attention models the relationship between two characters without regard to their distances in one sequence  For CWS task, the weight between adjacent characters should be more important while it is hard for self-attention to achieve the effect explicitly because the self-attention cannot get the order of sentences directly  The Gaussian-masked attention adjusts the weight between characters and their adjacent character to a larger value which stands for the effect of adjacent characters 
For forward and backward encoder, the self-attention sublayer needs to use a triangular matrix mask to let the self-attention focus on different weights:
where $pos_i$ is the position of character $c_i$  The triangular matrix for forward and backward encode are:
$\left[ \begin{matrix} 1 & 0 & 0 & \cdots &0\\ 1 & 1 & 0 & \cdots &0\\ 1 & 1 & 1 & \cdots &0\\ \vdots &\vdots &\vdots &\ddots &\vdots \\ 1 & 1 & 1 & \cdots & 1\\ \end{matrix} \right]$ $\left[ \begin{matrix} 1 & 1 & 1 & \cdots &1 \\ 0 & 1 & 1 & \cdots &1 \\ 0 & 0& 1 & \cdots &1 \\ \vdots &\vdots &\vdots &\ddots &\vdots \\ 0 & 0 & 0 & \cdots & 1\\ \end{matrix}\right]$
Similar as BIBREF24, we use multi-head attention to capture information from different dimension positions as Figure FIGREF16 and get Gaussian-masked directional multi-head attention  With multi-head attention architecture, the representation of input can be captured by
where $MH$ is the Gaussian-masked multi-head attention, ${W_i^q, W_i^k,W_i^v} \in \mathbb {R}^{d_k \times d_h}$ is the parameter matrices to generate heads, $d_k$ is the dimension of model and $d_h$ is the dimension of one head 
Models ::: Bi-affinal Attention Scorer
Regarding word boundaries as gaps between any adjacent words converts the character labeling task to the gap labeling task  Different from character labeling task, gap labeling task requires information of two adjacent characters  The relationship between adjacent characters can be represented as the type of gap  The characteristic of word boundaries makes bi-affine attention an appropriate scorer for CWS task 
Bi-affinal attention scorer is the component that we use to label the gap  Bi-affinal attention is developed from bilinear attention which has been used in dependency parsing BIBREF26 and SRL BIBREF27  The distribution of labels in a labeling task is often uneven which makes the output layer often include a fixed bias term for the prior probability of different labels BIBREF27  Bi-affine attention uses bias terms to alleviate the burden of the fixed bias term and get the prior probability which makes it different from bilinear attention  The distribution of the gap is uneven that is similar as other labeling task which fits bi-affine 
Bi-affinal attention scorer labels the target depending on information of independent unit and the joint information of two units  In bi-affinal attention, the score $s_{ij}$ of characters $c_i$ and $c_j$ $(i < j)$ is calculated by:
where $v_i^f$ is the forward information of $c_i$ and $v_i^b$ is the backward information of $c_j$  In Equation (DISPLAY_FORM21), $W$, $U$ and $b$ are all parameters that can be updated in training  $W$ is a matrix with shape $(d_i \times N\times d_j)$ and $U$ is a $(N\times (d_i + d_j))$ matrix where $d_i$ is the dimension of vector $v_i^f$ and $N$ is the number of labels 
In our model, the biaffine scorer uses the forward information of character in front of the gap and the backward information of the character behind the gap to distinguish the position of characters  Figure FIGREF22 is an example of labeling gap  The method of using biaffine scorer ensures that the boundaries of words can be determined by adjacent characters with different directional information  The score vector of the gap is formed by the probability of being a boundary of word  Further, the model generates all boundaries using activation function in a greedy decoding way 
Experiments ::: Experimental Settings ::: Data
We train and evaluate our model on datasets from SIGHAN Bakeoff 2005 BIBREF21 which has four datasets, PKU, MSR, AS and CITYU  Table TABREF23 shows the statistics of train data  We use F-score to evaluate CWS models  To train model with pre-trained embeddings in AS and CITYU, we use OpenCC to transfer data from traditional Chinese to simplified Chinese 
Experiments ::: Experimental Settings ::: Pre-trained Embedding
We only use unigram feature so we only trained character embeddings  Our pre-trained embedding are pre-trained on Chinese Wikipedia corpus by word2vec BIBREF29 toolkit  The corpus used for pre-trained embedding is all transferred to simplified Chinese and not segmented  On closed test, we use embeddings initialized randomly 
Experiments ::: Experimental Settings ::: Hyperparameters
For different datasets, we use two kinds of hyperparameters which are presented in Table TABREF24  We use hyperparameters in Table TABREF24 for small corpora (PKU and CITYU) and normal corpora (MSR and AS)  We set the standard deviation of Gaussian function in Equation (DISPLAY_FORM13) to 2  Each training batch contains sentences with at most 4096 tokens 
Experiments ::: Experimental Settings ::: Optimizer
To train our model, we use the Adam BIBREF30 optimizer with $\beta _1=0 9$, $\beta _2=0 98$ and $\epsilon =10^{-9}$  The learning rate schedule is the same as BIBREF24:
where $d$ is the dimension of embeddings, $step$ is the step number of training and $warmup_step$ is the step number of warmup  When the number of steps is smaller than the step of warmup, the learning rate increases linearly and then decreases 
Experiments ::: Hardware and Implements
We trained our models on a single CPU (Intel i7-5960X) with an nVidia 1080 Ti GPU  We implement our model in Python with Pytorch 1 0 
Experiments ::: Results
Tables TABREF25 and TABREF26 reports the performance of recent models and ours in terms of closed test setting  Without the assistance of unsupervised segmentation features userd in BIBREF20, our model outperforms all the other models in MSR and AS except BIBREF18 and get comparable performance in PKU and CITYU  Note that all the other models for this comparison adopt various $n$-gram features while only our model takes unigram ones 
With unsupervised segmentation features introduced by BIBREF20, our model gets a higher result  Specially, the results in MSR and AS achieve new state-of-the-art and approaching previous state-of-the-art in CITYU and PKU  The unsupervised segmentation features are derived from the given training dataset, thus using them does not violate the rule of closed test of SIGHAN Bakeoff 
Table TABREF36 compares our model and recent neural models in terms of open test setting in which any external resources, especially pre-trained embeddings or language models can be used  In MSR and AS, our model gets a comparable result while our results in CITYU and PKU are not remarkable 
However, it is well known that it is always hard to compare models when using open test setting, especially with pre-trained embedding  Not all models may use the same method and data to pre-train  Though pre-trained embedding or language model can improve the performance, the performance improvement itself may be from multiple sources  It often that there is a success of pre-trained embedding to improve the performance, while it cannot prove that the model is better 
Compared with other LSTM models, our model performs better in AS and MSR than in CITYU and PKU  Considering the scale of different corpora, we believe that the size of corpus affects our model and the larger size is, the better model performs  For small corpus, the model tends to be overfitting 
Tables TABREF25 and TABREF26 also show the decoding time in different datasets  Our model finishes the segmentation with the least decoding time in all four datasets, thanks to the architecture of model which only takes attention mechanism as basic block 
Related Work ::: Chinese Word Segmentation
CWS is a task for Chinese natural language process to delimit word boundary  BIBREF0 for the first time formulize CWS as a sequence labeling task  BIBREF3 show that different character tag sets can make essential impact for CWS  BIBREF2 use CRFs as a model for CWS, achieving new state-of-the-art  Works of statistical CWS has built the basis for neural CWS 
Neural word segmentation has been widely used to minimize the efforts in feature engineering which was important in statistical CWS  BIBREF4 introduce the neural model with sliding-window based sequence labeling  BIBREF6 propose a gated recursive neural network (GRNN) for CWS to incorporate complicated combination of contextual character and n-gram features  BIBREF7 use LSTM to learn long distance information  BIBREF9 propose a neural framework that eliminates context windows and utilize complete segmentation history  BIBREF33 explore a joint model that performs segmentation, POS-Tagging and chunking simultaneously  BIBREF34 propose a feature-enriched neural model for joint CWS and part-of-speech tagging  BIBREF35 present a joint model to enhance the segmentation of Chinese microtext by performing CWS and informal word detection simultaneously  BIBREF17 propose a character-based convolutional neural model to capture $n$-gram features automatically and an effective approach to incorporate word embeddings  BIBREF11 improve the model in BIBREF9 and propose a greedy neural word segmenter with balanced word and character embedding inputs  BIBREF23 propose a novel neural network model to incorporate unlabeled and partially-labeled data  BIBREF36 propose two methods that extend the Bi-LSTM to perform incorporating dictionaries into neural networks for CWS  BIBREF37 propose Switch-LSTMs to segment words and provided a more flexible solution for multi-criteria CWS which is easy to transfer the learned knowledge to new criteria 
Related Work ::: Transformer
Transformer BIBREF24 is an attention-based neural machine translation model  The Transformer is one kind of self-attention networks (SANs) which is proposed in BIBREF38  Encoder of the Transformer consists of one self-attention layer and a position-wise feed-forward layer  Decoder of the Transformer contains one self-attention layer, one encoder-decoder attention layer and one position-wise feed-forward layer  The Transformer uses residual connections around the sublayers and then followed by a layer normalization layer 
Scaled dot-product attention is the key component in the Transformer  The input of attention contains queries, keys, and values of input sequences  The attention is generated using queries and keys like Equation (DISPLAY_FORM11)  Structure of scaled dot-product attention allows the self-attention layer generate the representation of sentences at once and contain the information of the sentence which is different from RNN that process characters of sentences one by one  Standard self-attention is similar as Gaussian-masked direction attention while it does not have directional mask and gaussian mask  BIBREF24 also propose multi-head attention which is better to generate representation of sentence by dividing queries, keys and values to different heads and get information from different subspaces 
Conclusion
In this paper, we propose an attention mechanism only based Chinese word segmentation model  Our model uses self-attention from the Transformer encoder to take sequence input and bi-affine attention scorer to predict the label of gaps  To improve the ability of capturing the localness and directional information of self-attention based encoder, we propose a variant of self-attention called Gaussian-masked directional multi-head attention to replace the standard self-attention  We also extend the Transformer encoder to capture directional features  Our model uses only unigram features instead of multiple $n$-gram features in previous work  Our model is evaluated on standard benchmark dataset, SIGHAN Bakeoff 2005, which shows not only our model performs segmentation faster than any previous models but also gives new higher or comparable segmentation performance against previous state-of-the-art models 
Context 2: Introduction
Knowledge Base Question Answering (KBQA) systems answer questions by obtaining information from KB tuples BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5   For an input question, these systems typically generate a KB query, which can be executed to retrieve the answers from a KB  Figure 1 illustrates the process used to parse two sample questions in a KBQA system: (a) a single-relation question, which can be answered with a single $<$ head-entity, relation, tail-entity $>$ KB tuple BIBREF6 , BIBREF7 , BIBREF2 ; and (b) a more complex case, where some constraints need to be handled for multiple entities in the question  The KBQA system in the figure performs two key tasks: (1) entity linking, which links $n$ -grams in questions to KB entities, and (2) relation detection, which identifies the KB relation(s) a question refers to 
The main focus of this work is to improve the relation detection subtask and further explore how it can contribute to the KBQA system  Although general relation detection methods are well studied in the NLP community, such studies usually do not take the end task of KBQA into consideration  As a result, there is a significant gap between general relation detection studies and KB-specific relation detection  First, in most general relation detection tasks, the number of target relations is limited, normally smaller than 100  In contrast, in KBQA even a small KB, like Freebase2M BIBREF2 , contains more than 6,000 relation types  Second, relation detection for KBQA often becomes a zero-shot learning task, since some test instances may have unseen relations in the training data  For example, the SimpleQuestions BIBREF2 data set has 14% of the golden test relations not observed in golden training tuples  Third, as shown in Figure 1 (b), for some KBQA tasks like WebQuestions BIBREF0 , we need to predict a chain of relations instead of a single relation  This increases the number of target relation types and the sizes of candidate relation pools, further increasing the difficulty of KB relation detection  Owing to these reasons, KB relation detection is significantly more challenging compared to general relation detection tasks 
This paper improves KB relation detection to cope with the problems mentioned above  First, in order to deal with the unseen relations, we propose to break the relation names into word sequences for question-relation matching  Second, noticing that original relation names can sometimes help to match longer question contexts, we propose to build both relation-level and word-level relation representations  Third, we use deep bidirectional LSTMs (BiLSTMs) to learn different levels of question representations in order to match the different levels of relation information  Finally, we propose a residual learning method for sequence matching, which makes the model training easier and results in more abstract (deeper) question representations, thus improves hierarchical matching 
In order to assess how the proposed improved relation detection could benefit the KBQA end task, we also propose a simple KBQA implementation composed of two-step relation detection  Given an input question and a set of candidate entities retrieved by an entity linker based on the question, our proposed relation detection model plays a key role in the KBQA process: (1) Re-ranking the entity candidates according to whether they connect to high confident relations detected from the raw question text by the relation detection model  This step is important to deal with the ambiguities normally present in entity linking results  (2) Finding the core relation (chains) for each topic entity selection from a much smaller candidate entity set after re-ranking  The above steps are followed by an optional constraint detection step, when the question cannot be answered by single relations (e g , multiple entities in the question)  Finally the highest scored query from the above steps is used to query the KB for answers 
Our main contributions include: (i) An improved relation detection model by hierarchical matching between questions and relations with residual learning; (ii) We demonstrate that the improved relation detector enables our simple KBQA system to achieve state-of-the-art results on both single-relation and multi-relation KBQA tasks 
Background: Different Granularity in KB Relations
Previous research BIBREF4 , BIBREF20 formulates KB relation detection as a sequence matching problem  However, while the questions are natural word sequences, how to represent relations as sequences remains a challenging problem  Here we give an overview of two types of relation sequence representations commonly used in previous work 
(1) Relation Name as a Single Token (relation-level)  In this case, each relation name is treated as a unique token  The problem with this approach is that it suffers from the low relation coverage due to limited amount of training data, thus cannot generalize well to large number of open-domain relations  For example, in Figure 1 , when treating relation names as single tokens, it will be difficult to match the questions to relation names “episodes_written” and “starring_roles” if these names do not appear in training data – their relation embeddings $\mathbf {h}^r$ s will be random vectors thus are not comparable to question embeddings $\mathbf {h}^q$ s 
(2) Relation as Word Sequence (word-level)  In this case, the relation is treated as a sequence of words from the tokenized relation name  It has better generalization, but suffers from the lack of global information from the original relation names  For example in Figure 1 (b), when doing only word-level matching, it is difficult to rank the target relation “starring_roles” higher compared to the incorrect relation “plays_produced”  This is because the incorrect relation contains word “plays”, which is more similar to the question (containing word “play”) in the embedding space  On the other hand, if the target relation co-occurs with questions related to “tv appearance” in training, by treating the whole relation as a token (i e  relation id), we could better learn the correspondence between this token and phrases like “tv show” and “play on” 
The two types of relation representation contain different levels of abstraction  As shown in Table 1 , the word-level focuses more on local information (words and short phrases), and the relation-level focus more on global information (long phrases and skip-grams) but suffer from data sparsity  Since both these levels of granularity have their own pros and cons, we propose a hierarchical matching approach for KB relation detection: for a candidate relation, our approach matches the input question to both word-level and relation-level representations to get the final ranking score  Section "Improved KB Relation Detection" gives the details of our proposed approach 
Improved KB Relation Detection
This section describes our hierarchical sequence matching with residual learning approach for relation detection  In order to match the question to different aspects of a relation (with different abstraction levels), we deal with three problems as follows on learning question/relation representations 
Relation Representations from Different Granularity
We provide our model with both types of relation representation: word-level and relation-level  Therefore, the input relation becomes $\mathbf {r}=\lbrace r^{word}_1,\cdots ,r^{word}_{M_1}\rbrace  \cup \lbrace r^{rel}_1,\cdots ,r^{rel}_{M_2}\rbrace $ , where the first $M_1$ tokens are words (e g  {episode, written}), and the last $M_2$ tokens are relation names, e g , {episode_written} or {starring_roles, series} (when the target is a chain like in Figure 1 (b))  We transform each token above to its word embedding then use two BiLSTMs (with shared parameters) to get their hidden representations $[\mathbf {B}^{word}_{1:M_1}:\mathbf {B}^{rel}_{1:M_2}]$ (each row vector $\mathbf {\beta }_i$ is the concatenation between forward/backward representations at $i$ )  We initialize the relation sequence LSTMs with the final state representations of the word sequence, as a back-off for unseen relations  We apply one max-pooling on these two sets of vectors and get the final relation representation $\mathbf {h}^r$  
Different Abstractions of Questions Representations
From Table 1 , we can see that different parts of a relation could match different contexts of question texts  Usually relation names could match longer phrases in the question and relation words could match short phrases  Yet different words might match phrases of different lengths 
As a result, we hope the question representations could also comprise vectors that summarize various lengths of phrase information (different levels of abstraction), in order to match relation representations of different granularity  We deal with this problem by applying deep BiLSTMs on questions  The first-layer of BiLSTM works on the word embeddings of question words $\mathbf {q}=\lbrace q_1,\cdots ,q_N\rbrace $ and gets hidden representations $\mathbf {\Gamma }^{(1)}_{1:N}=[\mathbf {\gamma }^{(1)}_1;\cdots ;\mathbf {\gamma }^{(1)}_N]$   The second-layer BiLSTM works on $\mathbf {\Gamma }^{(1)}_{1:N}$ to get the second set of hidden representations $\mathbf {\Gamma }^{(2)}_{1:N}$   Since the second BiLSTM starts with the hidden vectors from the first layer, intuitively it could learn more general and abstract information compared to the first layer 
Note that the first(second)-layer of question representations does not necessarily correspond to the word(relation)-level relation representations, instead either layer of question representations could potentially match to either level of relation representations  This raises the difficulty of matching between different levels of relation/question representations; the following section gives our proposal to deal with such problem 
Hierarchical Matching between Relation and Question
Now we have question contexts of different lengths encoded in $\mathbf {\Gamma }^{(1)}_{1:N}$ and $\mathbf {\Gamma }^{(2)}_{1:N}$   Unlike the standard usage of deep BiLSTMs that employs the representations in the final layer for prediction, here we expect that two layers of question representations can be complementary to each other and both should be compared to the relation representation space (Hierarchical Matching)  This is important for our task since each relation token can correspond to phrases of different lengths, mainly because of syntactic variations  For example in Table 1 , the relation word written could be matched to either the same single word in the question or a much longer phrase be the writer of 
We could perform the above hierarchical matching by computing the similarity between each layer of $\mathbf {\Gamma }$ and $\mathbf {h}^r$ separately and doing the (weighted) sum between the two scores  However this does not give significant improvement (see Table 2 )  Our analysis in Section "Relation Detection Results" shows that this naive method suffers from the training difficulty, evidenced by that the converged training loss of this model is much higher than that of a single-layer baseline model  This is mainly because (1) Deep BiLSTMs do not guarantee that the two-levels of question hidden representations are comparable, the training usually falls to local optima where one layer has good matching scores and the other always has weight close to 0  (2) The training of deeper architectures itself is more difficult 
To overcome the above difficulties, we adopt the idea from Residual Networks BIBREF23 for hierarchical matching by adding shortcut connections between two BiLSTM layers  We proposed two ways of such Hierarchical Residual Matching: (1) Connecting each $\mathbf {\gamma }^{(1)}_i$ and $\mathbf {\gamma }^{(2)}_i$ , resulting in a $\mathbf {\gamma }^{^{\prime }}_i=\mathbf {\gamma }^{(1)}_i + \mathbf {\gamma }^{(2)}_i$ for each position $i$   Then the final question representation $\mathbf {h}^q$ becomes a max-pooling over all $\mathbf {\gamma }^{^{\prime }}_i$ s, 1 $\le $ i $\le $ $N$   (2) Applying max-pooling on $\mathbf {\Gamma }^{(1)}_{1:N}$ and $\mathbf {\gamma }^{(2)}_i$0 to get $\mathbf {\gamma }^{(2)}_i$1 and $\mathbf {\gamma }^{(2)}_i$2 , respectively, then setting $\mathbf {\gamma }^{(2)}_i$3   Finally we compute the matching score of $\mathbf {\gamma }^{(2)}_i$4 given $\mathbf {\gamma }^{(2)}_i$5 as $\mathbf {\gamma }^{(2)}_i$6  
Intuitively, the proposed method should benefit from hierarchical training since the second layer is fitting the residues from the first layer of matching, so the two layers of representations are more likely to be complementary to each other  This also ensures the vector spaces of two layers are comparable and makes the second-layer training easier 
During training we adopt a ranking loss to maximizing the margin between the gold relation $\mathbf {r}^+$ and other relations $\mathbf {r}^-$ in the candidate pool $R$  
$$l_{\mathrm {rel}} = \max \lbrace 0, \gamma - s_{\mathrm {rel}}(\mathbf {r}^+; \mathbf {q}) + s_{\mathrm {rel}}(\mathbf {r}^-; \mathbf {q})\rbrace  \nonumber $$   (Eq  12)
where $\gamma $ is a constant parameter  Fig 2 summarizes the above Hierarchical Residual BiLSTM (HR-BiLSTM) model 
Another way of hierarchical matching consists in relying on attention mechanism, e g  BIBREF24 , to find the correspondence between different levels of representations  This performs below the HR-BiLSTM (see Table 2 ) 
KBQA Enhanced by Relation Detection
This section describes our KBQA pipeline system  We make minimal efforts beyond the training of the relation detection model, making the whole system easy to build 
Following previous work BIBREF4 , BIBREF5 , our KBQA system takes an existing entity linker to produce the top- $K$ linked entities, $EL_K(q)$ , for a question $q$ (“initial entity linking”)  Then we generate the KB queries for $q$ following the four steps illustrated in Algorithm "KBQA Enhanced by Relation Detection"  
[htbp] InputInput OutputOutput Top query tuple $(\hat{e},\hat{r}, \lbrace (c, r_c)\rbrace )$ Entity Re-Ranking (first-step relation detection): Use the raw question text as input for a relation detector to score all relations in the KB that are associated to the entities in $EL_K(q)$ ; use the relation scores to re-rank $EL_K(q)$ and generate a shorter list $EL^{\prime }_{K^{\prime }}(q)$ containing the top- $K^{\prime }$ entity candidates (Section "Entity Re-Ranking" ) Relation Detection: Detect relation(s) using the reformatted question text in which the topic entity is replaced by a special token $<$ e $>$ (Section "Relation Detection" ) Query Generation: Combine the scores from step 1 and 2, and select the top pair $(\hat{e},\hat{r})$ (Section "Query Generation" ) Constraint Detection (optional): Compute similarity between $q$ and any neighbor entity $c$ of the entities along $EL_K(q)$0 (connecting by a relation $EL_K(q)$1 ) , add the high scoring $EL_K(q)$2 and $EL_K(q)$3 to the query (Section "Constraint Detection" )  KBQA with two-step relation detection
Compared to previous approaches, the main difference is that we have an additional entity re-ranking step after the initial entity linking  We have this step because we have observed that entity linking sometimes becomes a bottleneck in KBQA systems  For example, on SimpleQuestions the best reported linker could only get 72 7% top-1 accuracy on identifying topic entities  This is usually due to the ambiguities of entity names, e g  in Fig 1 (a), there are TV writer and baseball player “Mike Kelley”, which is impossible to distinguish with only entity name matching 
Having observed that different entity candidates usually connect to different relations, here we propose to help entity disambiguation in the initial entity linking with relations detected in questions 
Sections "Entity Re-Ranking" and "Relation Detection" elaborate how our relation detection help to re-rank entities in the initial entity linking, and then those re-ranked entities enable more accurate relation detection  The KBQA end task, as a result, benefits from this process 
Entity Re-Ranking
In this step, we use the raw question text as input for a relation detector to score all relations in the KB with connections to at least one of the entity candidates in $EL_K(q)$   We call this step relation detection on entity set since it does not work on a single topic entity as the usual settings  We use the HR-BiLSTM as described in Sec  "Improved KB Relation Detection"   For each question $q$ , after generating a score $s_{rel}(r;q)$ for each relation using HR-BiLSTM, we use the top $l$ best scoring relations ( $R^{l}_q$ ) to re-rank the original entity candidates  Concretely, for each entity $e$ and its associated relations $R_e$ , given the original entity linker score $s_{linker}$ , and the score of the most confident relation $r\in R_q^{l} \cap R_e$ , we sum these two scores to re-rank the entities:
$$s_{\mathrm {rerank}}(e;q) =& \alpha \cdot s_{\mathrm {linker}}(e;q) \nonumber \\ + & (1-\alpha ) \cdot \max _{r \in R_q^{l} \cap R_e} s_{\mathrm {rel}}(r;q) \nonumber $$   (Eq  15)
Finally, we select top $K^{\prime }$ $<$ $K$ entities according to score $s_{rerank}$ to form the re-ranked list $EL_{K^{\prime }}^{^{\prime }}(q)$  
We use the same example in Fig 1 (a) to illustrate the idea  Given the input question in the example, a relation detector is very likely to assign high scores to relations such as “episodes_written”, “author_of” and “profession”  Then, according to the connections of entity candidates in KB, we find that the TV writer “Mike Kelley” will be scored higher than the baseball player “Mike Kelley”, because the former has the relations “episodes_written” and “profession”  This method can be viewed as exploiting entity-relation collocation for entity linking 
Relation Detection
In this step, for each candidate entity $e \in EL_K^{\prime }(q)$ , we use the question text as the input to a relation detector to score all the relations $r \in R_e$ that are associated to the entity $e$ in the KB  Because we have a single topic entity input in this step, we do the following question reformatting: we replace the the candidate $e$ 's entity mention in $q$ with a token “ $<$ e $>$ ”  This helps the model better distinguish the relative position of each word compared to the entity  We use the HR-BiLSTM model to predict the score of each relation $r \in R_e$ : $s_{rel} (r;e,q)$  
Query Generation
Finally, the system outputs the $<$ entity, relation (or core-chain) $>$ pair $(\hat{e}, \hat{r})$ according to:
$$s(\hat{e}, \hat{r}; q) =& \max _{e \in EL_{K^{\prime }}^{^{\prime }}(q), r \in R_e} \left( \beta \cdot s_{\mathrm {rerank}}(e;q) \right  \nonumber \\ &\left + (1-\beta ) \cdot s_{\mathrm {rel}} (r;e,q) \right), \nonumber $$   (Eq  19)
where $\beta $ is a hyperparameter to be tuned 
Constraint Detection
Similar to BIBREF4 , we adopt an additional constraint detection step based on text matching  Our method can be viewed as entity-linking on a KB sub-graph  It contains two steps: (1) Sub-graph generation: given the top scored query generated by the previous 3 steps, for each node $v$ (answer node or the CVT node like in Figure 1 (b)), we collect all the nodes $c$ connecting to $v$ (with relation $r_c$ ) with any relation, and generate a sub-graph associated to the original query  (2) Entity-linking on sub-graph nodes: we compute a matching score between each $n$ -gram in the input question (without overlapping the topic entity) and entity name of $c$ (except for the node in the original query) by taking into account the maximum overlapping sequence of characters between them (see Appendix A for details and B for special rules dealing with date/answer type constraints)  If the matching score is larger than a threshold $\theta $ (tuned on training set), we will add the constraint entity $c$ (and $r_c$ ) to the query by attaching it to the corresponding node $v$ on the core-chain 
Experiments
Task Introduction & Settings
We use the SimpleQuestions BIBREF2 and WebQSP BIBREF25 datasets  Each question in these datasets is labeled with the gold semantic parse  Hence we can directly evaluate relation detection performance independently as well as evaluate on the KBQA end task 
SimpleQuestions (SQ): It is a single-relation KBQA task  The KB we use consists of a Freebase subset with 2M entities (FB2M) BIBREF2 , in order to compare with previous research  yin2016simple also evaluated their relation extractor on this data set and released their proposed question-relation pairs, so we run our relation detection model on their data set  For the KBQA evaluation, we also start with their entity linking results  Therefore, our results can be compared with their reported results on both tasks 
WebQSP (WQ): A multi-relation KBQA task  We use the entire Freebase KB for evaluation purposes  Following yih-EtAl:2016:P16-2, we use S-MART BIBREF26 entity-linking outputs  In order to evaluate the relation detection models, we create a new relation detection task from the WebQSP data set  For each question and its labeled semantic parse: (1) we first select the topic entity from the parse; and then (2) select all the relations and relation chains (length $\le $ 2) connected to the topic entity, and set the core-chain labeled in the parse as the positive label and all the others as the negative examples 
We tune the following hyper-parameters on development sets: (1) the size of hidden states for LSTMs ({50, 100, 200, 400}); (2) learning rate ({0 1, 0 5, 1 0, 2 0}); (3) whether the shortcut connections are between hidden states or between max-pooling results (see Section "Hierarchical Matching between Relation and Question" ); and (4) the number of training epochs 
For both the relation detection experiments and the second-step relation detection in KBQA, we have entity replacement first (see Section "Relation Detection" and Figure 1 )  All word vectors are initialized with 300- $d$ pretrained word embeddings BIBREF27   The embeddings of relation names are randomly initialized, since existing pre-trained relation embeddings (e g  TransE) usually support limited sets of relation names  We leave the usage of pre-trained relation embeddings to future work 
Relation Detection Results
Table 2 shows the results on two relation detection tasks  The AMPCNN result is from BIBREF20 , which yielded state-of-the-art scores by outperforming several attention-based methods  We re-implemented the BiCNN model from BIBREF4 , where both questions and relations are represented with the word hash trick on character tri-grams  The baseline BiLSTM with relation word sequence appears to be the best baseline on WebQSP and is close to the previous best result of AMPCNN on SimpleQuestions  Our proposed HR-BiLSTM outperformed the best baselines on both tasks by margins of 2-3% (p $<$ 0 001 and 0 01 compared to the best baseline BiLSTM w/ words on SQ and WQ respectively) 
Note that using only relation names instead of words results in a weaker baseline BiLSTM model  The model yields a significant performance drop on SimpleQuestions (91 2% to 88 9%)  However, the drop is much smaller on WebQSP, and it suggests that unseen relations have a much bigger impact on SimpleQuestions 
The bottom of Table 2 shows ablation results of the proposed HR-BiLSTM  First, hierarchical matching between questions and both relation names and relation words yields improvement on both datasets, especially for SimpleQuestions (93 3% vs  91 2/88 8%)  Second, residual learning helps hierarchical matching compared to weighted-sum and attention-based baselines (see Section "Hierarchical Matching between Relation and Question" )  For the attention-based baseline, we tried the model from BIBREF24 and its one-way variations, where the one-way model gives better results  Note that residual learning significantly helps on WebQSP (80 65% to 82 53%), while it does not help as much on SimpleQuestions  On SimpleQuestions, even removing the deep layers only causes a small drop in performance  WebQSP benefits more from residual and deeper architecture, possibly because in this dataset it is more important to handle larger scope of context matching 
Finally, on WebQSP, replacing BiLSTM with CNN in our hierarchical matching framework results in a large performance drop  Yet on SimpleQuestions the gap is much smaller  We believe this is because the LSTM relation encoder can better learn the composition of chains of relations in WebQSP, as it is better at dealing with longer dependencies 
Next, we present empirical evidences, which show why our HR-BiLSTM model achieves the best scores  We use WebQSP for the analysis purposes  First, we have the hypothesis that training of the weighted-sum model usually falls to local optima, since deep BiLSTMs do not guarantee that the two-levels of question hidden representations are comparable  This is evidenced by that during training one layer usually gets a weight close to 0 thus is ignored  For example, one run gives us weights of -75 39/0 14 for the two layers (we take exponential for the final weighted sum)  It also gives much lower training accuracy (91 94%) compared to HR-BiLSTM (95 67%), suffering from training difficulty 
Second, compared to our deep BiLSTM with shortcut connections, we have the hypothesis that for KB relation detection, training deep BiLSTMs is more difficult without shortcut connections  Our experiments suggest that deeper BiLSTM does not always result in lower training accuracy  In the experiments a two-layer BiLSTM converges to 94 99%, even lower than the 95 25% achieved by a single-layer BiLSTM  Under our setting the two-layer model captures the single-layer model as a special case (so it could potentially better fit the training data), this result suggests that the deep BiLSTM without shortcut connections might suffers more from training difficulty 
Finally, we hypothesize that HR-BiLSTM is more than combination of two BiLSTMs with residual connections, because it encourages the hierarchical architecture to learn different levels of abstraction  To verify this, we replace the deep BiLSTM question encoder with two single-layer BiLSTMs (both on words) with shortcut connections between their hidden states  This decreases test accuracy to 76 11%  It gives similar training accuracy compared to HR-BiLSTM, indicating a more serious over-fitting problem  This proves that the residual and deep structures both contribute to the good performance of HR-BiLSTM 
KBQA End-Task Results
Table 3 compares our system with two published baselines (1) STAGG BIBREF4 , the state-of-the-art on WebQSP and (2) AMPCNN BIBREF20 , the state-of-the-art on SimpleQuestions  Since these two baselines are specially designed/tuned for one particular dataset, they do not generalize well when applied to the other dataset  In order to highlight the effect of different relation detection models on the KBQA end-task, we also implemented another baseline that uses our KBQA system but replaces HR-BiLSTM with our implementation of AMPCNN (for SimpleQuestions) or the char-3-gram BiCNN (for WebQSP) relation detectors (second block in Table 3 ) 
Compared to the baseline relation detector (3rd row of results), our method, which includes an improved relation detector (HR-BiLSTM), improves the KBQA end task by 2-3% (4th row)  Note that in contrast to previous KBQA systems, our system does not use joint-inference or feature-based re-ranking step, nevertheless it still achieves better or comparable results to the state-of-the-art 
The third block of the table details two ablation tests for the proposed components in our KBQA systems: (1) Removing the entity re-ranking step significantly decreases the scores  Since the re-ranking step relies on the relation detection models, this shows that our HR-BiLSTM model contributes to the good performance in multiple ways  Appendix C gives the detailed performance of the re-ranking step  (2) In contrast to the conclusion in BIBREF4 , constraint detection is crucial for our system  This is probably because our joint performance on topic entity and core-chain detection is more accurate (77 5% top-1 accuracy), leaving a huge potential (77 5% vs  58 0%) for the constraint detection module to improve 
Finally, like STAGG, which uses multiple relation detectors (see yih2015semantic for the three models used), we also try to use the top-3 relation detectors from Section "Relation Detection Results"   As shown on the last row of Table 3 , this gives a significant performance boost, resulting in a new state-of-the-art result on SimpleQuestions and a result comparable to the state-of-the-art on WebQSP 
Conclusion
KB relation detection is a key step in KBQA and is significantly different from general relation extraction tasks  We propose a novel KB relation detection model, HR-BiLSTM, that performs hierarchical matching between questions and KB relations  Our model outperforms the previous methods on KB relation detection tasks and allows our KBQA system to achieve state-of-the-arts  For future work, we will investigate the integration of our HR-BiLSTM into end-to-end systems  For example, our model could be integrated into the decoder in BIBREF31 , to provide better sequence prediction  We will also investigate new emerging datasets like GraphQuestions BIBREF32 and ComplexQuestions BIBREF30 to handle more characteristics of general QA 
Context 3: Introduction
Microblogging such as Twitter and Weibo is a popular social networking service, which allows users to post messages up to 140 characters  There are millions of active users on the platform who stay connected with friends  Unfortunately, spammers also use it as a tool to post malicious links, send unsolicited messages to legitimate users, etc  A certain amount of spammers could sway the public opinion and cause distrust of the social platform  Despite the use of rigid anti-spam rules, human-like spammers whose homepages having photos, detailed profiles etc  have emerged  Unlike previous "simple" spammers, whose tweets contain only malicious links, those "smart" spammers are more difficult to distinguish from legitimate users via content-based features alone BIBREF0  
There is a considerable amount of previous work on spammer detection on social platforms  Researcher from Twitter Inc  BIBREF1 collect bot accounts and perform analysis on the user behavior and user profile features  Lee et al  lee2011seven use the so-called social honeypot by alluring social spammers' retweet to build a benchmark dataset, which has been extensively explored in our paper  Some researchers focus on the clustering of urls in tweets and network graph of social spammers BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 , showing the power of social relationship features As for content information modeling, BIBREF6 apply improved sparse learning methods  However, few studies have adopted topic-based features  Some researchers BIBREF7 discuss topic characteristics of spamming posts, indicating that spammers are highly likely to dwell on some certain topics such as promotion  But this may not be applicable to the current scenario of smart spammers 
In this paper, we propose an efficient feature extraction method  In this method, two new topic-based features are extracted and used to discriminate human-like spammers from legitimate users  We consider the historical tweets of each user as a document and use the Latent Dirichlet Allocation (LDA) model to compute the topic distribution for each user  Based on the calculated topic probability, two topic-based features, the Local Outlier Standard Score (LOSS) which captures the user's interests on different topics and the Global Outlier Standard Score (GOSS) which reveals the user's interests on specific topic in comparison with other users', are extracted  The two features contain both local and global information, and the combination of them can distinguish human-like spammers effectively 
To the best of our knowledge, it is the first time that features based on topic distributions are used in spammer classification  Experimental results on one public dataset and one self-collected dataset further validate that the two sets of extracted topic-based features get excellent performance on human-like spammer classification problem compared with other state-of-the-art methods  In addition, we build a Weibo dataset, which contains both legitimate users and spammers 
To summarize, our major contributions are two-fold:
In the following sections, we first propose the topic-based features extraction method in Section 2, and then introduce the two datasets in Section 3  Experimental results are discussed in Section 4, and we conclude the paper in Section 5  Future work is presented in Section 6 
Methodology
In this section, we first provide some observations we obtained after carefully exploring the social network, then the LDA model is introduced  Based on the LDA model, the ways to obtain the topic probability vector for each user and the two topic-based features are provided 
Observation
After exploring the homepages of a substantial number of spammers, we have two observations  1) social spammers can be divided into two categories  One is content polluters, and their tweets are all about certain kinds of advertisement and campaign  The other is fake accounts, and their tweets resemble legitimate users' but it seems they are simply random copies of others to avoid being detected by anti-spam rules  2) For legitimate users, content polluters and fake accounts, they show different patterns on topics which interest them 
Legitimate users mainly focus on limited topics which interest him  They seldom post contents unrelated to their concern 
Content polluters concentrate on certain topics 
Fake accounts focus on a wide range of topics due to random copying and retweeting of other users' tweets 
Spammers and legitimate users show different interests on some topics e g  commercial, weather, etc 
To better illustrate our observation, Figure  1 shows the topic distribution of spammers and legitimate users in two employed datasets(the Honeypot dataset and Weibo dataset)  We can see that on both topics (topic-3 and topic-11) there exists obvious difference between the red bars and green bars, representing spammers and legitimate users  On the Honeypot dataset, spammers have a narrower shape of distribution (the outliers on the red bar tail are not counted) than that of legitimate users  This is because there are more content polluters than fake accounts  In other word, spammers in this dataset tend to concentrate on limited topics  While on the Weibo dataset, fake accounts who are interested in different topics take large proportion of spammers  Their distribution is more flat (i e  red bars) than that of the legitimate users  Therefore we can detect spammers by means of the difference of their topic distribution patterns 
LDA model
Blei et al blei2003latent first presented Latent Dirichlet Allocation(LDA) as an example of topic model 
Each document $i$ is deemed as a bag of words $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $ and $M$ is the number of words

ground_truth: pays attentions to adjacent characters and casts a localness relationship between the characters as a fixed Gaussian weight assuming the weight relies on the distance between characters

Gaussian-masked directional attention can be described as a function to map queries and key-value pairs to the representation of input, Gaussian-masked directional attention expects to pay attention to the adjacent characters of each positions and cast the localness relationship between characters as a fix Gaussian weight for attention, Gaussian weight only relys on the distance between characters

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What is te core component for KBQA?
Context 3:  Each word is attributable to one of the document's topics $Z=\left\lbrace  z_{i1},z_{i2},   ,z_{iK}\right\rbrace $ and $K$ is the number of topics  $\psi _{k}$ is a multinomial distribution over words for topic $k$   $\theta _i$ is another multinomial distribution over topics for document $i$   The smoothed generative model is illustrated in Figure  2   $\alpha $ and $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $0 are hyper parameter that affect scarcity of the document-topic and topic-word distributions  In this paper, $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $1 , $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $2 and $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $3 are empirically set to 0 3, 0 01 and 15  The entire content of each Twitter user is regarded as one document  We adopt Gibbs Sampling BIBREF8 to speed up the inference of LDA  Based on LDA, we can get the topic probabilities for all users in the employed dataset as: $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $4 , where $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $5 is the number of users  Each element $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $6 is a topic probability vector for the $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $7 document  $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $8 is the raw topic probability vector and our features are developed on top of it 
Topic-based Features
Using the LDA model, each person in the dataset is with a topic probability vector $X_i$   Assume $x_{ik}\in X_{i}$ denotes the likelihood that the $\emph {i}^{th}$ tweet account favors $\emph {k}^{th}$ topic in the dataset  Our topic based features can be calculated as below 
Global Outlier Standard Score measures the degree that a user's tweet content is related to a certain topic compared to the other users  Specifically, the "GOSS" score of user $i$ on topic $k$ can be calculated as Eq ( 12 ):
$$\centering \begin{array}{ll} \mu \left(x_{k}\right)=\frac{\sum _{i=1}^{n} x_{ik}}{n},\\ GOSS\left(x_{ik}\right)=\frac{x_{ik}-\mu \left(x_k\right)}{\sqrt{\underset{i}{\sum }\left(x_{ik}-\mu \left(x_{k}\right)\right)^{2}}}  \end{array}$$   (Eq  12)
The value of $GOSS\left(x_{ik}\right)$ indicates the interesting degree of this person to the $\emph {k}^{th}$ topic  Specifically, if $GOSS\left(x_{ik}\right)$ > $GOSS\left(x_{jk}\right)$ , it means that the $\emph {i}^{th}$ person has more interest in topic $k$ than the $\emph {j}^{th}$ person  If the value $GOSS\left(x_{ik}\right)$ is extremely high or low, the $\emph {i}^{th}$ person showing extreme interest or no interest on topic $k$ which will probably be a distinctive pattern in the fowllowing classfication  Therefore, the topics interested or disliked by the $\emph {k}^{th}$0 person can be manifested by $\emph {k}^{th}$1 , from which the pattern of the interested topics with regarding to this person is found  Denote $\emph {k}^{th}$2 our first topic-based feature, and it hopefully can get good performance on spammer detection 
Local Outlier Standard Score measures the degree of interest someone shows to a certain topic by considering his own homepage content only  For instance, the "LOSS" score of account $i$ on topic $k$ can be calculated as Eq ( 13 ):
$$\centering \begin{array}{ll} \mu \left(x_{i}\right)=\frac{\sum _{k=1}^{K} x_{ik}}{K},\\ LOSS\left(x_{ik}\right)=\frac{x_{ik}-\mu \left(x_i\right)}{\sqrt{\underset{k}{\sum }\left(x_{ik}-\mu \left(x_{i}\right)\right)^{2}}}

ground_truth: answer questions by obtaining information from KB tuples 

hierarchical matching between questions and relations with residual learning

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What is the benchmark dataset and is its quality high?
Context 3:  \end{array}$$   (Eq  13)
$\mu (x_i)$ represents the averaged interesting degree for all topics with regarding to $\emph {i}^{th}$ user and his tweet content  Similarly to $GOSS$ , the topics interested or disliked by the $\emph {i}^{th}$ person via considering his single post information can be manifested by $f_{LOSS}^{i}=[LOSS(x_{i1})\cdots LOSS(x_{iK})]$ , and $LOSS$ becomes our second topic-based features for the $\emph {i}^{th}$ person 
Dataset
We use one public dataset Social Honeypot dataset and one self-collected dataset Weibo dataset to validate the effectiveness of our proposed features 
Social Honeypot Dataset: Lee et al  lee2010devils created and deployed 60 seed social accounts on Twitter to attract spammers by reporting back what accounts interact with them  They collected 19,276 legitimate users and 22,223 spammers in their datasets along with their tweet content in 7 months  This is our first test dataset 
Our Weibo Dataset: Sina Weibo is one of the most famous social platforms in China  It has implemented many features from Twitter  The 2197 legitimate user accounts in this dataset are provided by the Tianchi Competition held by Sina Weibo  The spammers are all purchased commercially from multiple vendors on the Internet  We checked them manually and collected 802 suitable "smart" spammers accounts 
Preprocessing: Before directly performing the experiments on the employed datasets, we first delete some accounts with few posts in the two employed since the number of tweets is highly indicative of spammers  For the English Honeypot dataset, we remove stopwords, punctuations, non-ASCII words and apply stemming  For the Chinese Weibo dataset, we perform segmentation with "Jieba", a Chinese text segmentation tool  After preprocessing steps, the Weibo dataset contains 2197 legitimate users and 802 spammers, and the honeypot dataset contains 2218 legitimate users and 2947 spammers  It is worth mentioning that the Honeypot dataset has been slashed because most of the Twitter accounts only have limited number of posts, which are not enough to show their interest inclination 
Evaluation Metrics
The evaluating indicators in our model are show in 2   We calculate precision, recall and F1-score (i e  F1 score) as in Eq  ( 19 )  Precision is the ratio of selected accounts that are spammers  Recall is the ratio of spammers that are detected so  F1-score is the harmonic mean of precision and recall 
$$precision =\frac{TP}{TP+FP}, recall =\frac{TP}{TP+FN}\nonumber \\ F1-score = \frac{2\times precision \times recall}{precision + recall}$$   (Eq  19)
Performance Comparisons with Baseline
Three baseline classification methods: Support Vector Machines (SVM), Adaboost, and Random Forests are adopted to evaluate our extracted features  We test each classification algorithm with scikit-learn BIBREF9 and run a 10-fold cross validation  On each dataset, the employed classifiers are trained with individual feature first, and then with the combination of the two features  From 1 , we can see that GOSS+LOSS achieves the best performance on F1-score among all others  Besides, the classification by combination of LOSS and GOSS can increase accuracy by more than 3% compared with raw topic distribution probability 
Comparison with Other Features
To compare our extracted features with previously used features for spammer detection, we use three most discriminative feature sets according to Lee et al  lee2011seven( 4 )  Two classifiers (Adaboost and SVM) are selected to conduct feature performance comparisons  Using Adaboost, our LOSS+GOSS features outperform all other features except for UFN which is 2% higher than ours with regard to precision on the Honeypot dataset  It is caused by the incorrectly classified spammers who are mostly news source after our manual check  They keep posting all kinds of news pieces covering diverse topics, which is similar to the behavior of fake accounts  However, UFN based on friendship networks is more useful for public accounts who possess large number of followers  The best recall value of our LOSS+GOSS features using SVM is up to 6% higher than the results by other feature groups  Regarding F1-score, our features outperform all other features  To further show the advantages of our proposed features, we compare our combined LOSS+GOSS with the combination of all the features from Lee et al  lee2011seven (UFN+UC+UH)  It's obvious that LOSS+GOSS have a great advantage over UFN+UC+UH in terms of recall and F1-score  Moreover, by combining our LOSS+GOSS features and UFN+UC+UH features together, we obtained another 7 1% and 2 3% performance gain with regard to precision and F1-score by Adaboost  Though there is a slight decline in terms of recall  By SVM, we get comparative results on recall and F1-score but about 3 5% improvement on precision 
Conclusion
In this paper, we propose a novel feature extraction method to effectively detect "smart" spammers who post seemingly legitimate tweets and are thus difficult to identify by existing spammer classification methods  Using the LDA model, we obtain the topic probability for each Twitter user  By utilizing the topic probability result, we extract our two topic-based features: GOSS and LOSS which represent the account with global and local information  Experimental results on a public dataset and a self-built Chinese microblog dataset validate the effectiveness of the proposed features 
Future Work
In future work, the combination method of local and global information can be further improved to maximize their individual strengths  We will also apply decision theory to enhancing the performance of our proposed features  Moreover, we are also building larger datasets on both Twitter and Weibo to validate our method  Moreover, larger datasets on both Twitter and Weibo will be built to further validate our method 

ground_truth: Social Honeypot dataset (public) and Weibo dataset (self-collected); yes

Social Honeypot, which is not of high quality