Introduction
AliMe Bot is a kind of retrieval-based online service of E-commerce which collects a lot of predefined question-answering pairs. Through data analysis, we find that many variants of a question exist which means a sort of questions can correspond to a same answer. Based on the observation, naturally, we can view these questions with the same answer as a bag. Obviously, the bag contains diverse expressions of a question, which may provide more matching evidence than only one question due to the rich information contained in the bag. Motivated by the fact, different from existing query-question (Q-Q) matching method, we propose a new query-bag matching approach for retrieval-based chatbots. Concretely, when a user raises a query, the query-bag matching model provides the most suitable bag and returns the corresponding answer of the bag. To our knowledge, there is no query-bag matching study exists, and we focus on the new approach in this paper.
Recalling the text matching task BIBREF0, recently, researchers have adopted the deep neural network to model the matching relationship. ESIM BIBREF1 judges the inference relationship between two sentences by enhanced LSTM and interaction space. SMN BIBREF2 performs the context-response matching for the open-domain dialog system. BIBREF3 BIBREF3 explores the usefulness of noisy pre-training in the paraphrase identification task. BIBREF4 BIBREF4 surveys the methods in query-document matching in web search which focuses on the topic model, the dependency model, etc. However, none of them pays attention to the query-bag matching which concentrates on the matching for a query and a bag containing multiple questions.
When a user poses a query to the bot, the bot searches the most similar bag and uses the corresponding answer to reply to the user. The more information in the query covered by the bag, the more likely the bag's corresponding answer answers the query. What's more, the bag should not have too much information exceeding the query. Thus modelling the bag-to-query and query-to-bag coverage is essential in this task.
In this paper, we propose a simple but effective mutual coverage component to model the above-mentioned problem. The coverage is based on the cross-attention matrix of the query-bag pair which indicates the matching degree of elements between the query and bag. The mutual coverage is performed by stacking the cross-attention matrix along two directions, i.e., query and bag, in the word level respectively. In addition to the mutual coverage, a bag representation in word level is issued to help discover the main points of a bag. The bag representation then provides new matching evidence to the query-bag matching model.
We conduct experiments on the AliMe and Quora dataset for the query-bag matching based information-seeking conversation. Compared with baselines, we verify the effectiveness of our model. Our model obtains 0.05 and 0.03 $\text{R}_{10}@1$ gains comparing to the strongest baseline in the two datasets. The ablation study shows the usefulness of the components. The contributions in this paper are summarized as follows: 1) To the best of our knowledge, we are the first to adopt query-bag matching in the information-seeking conversation. 2) We propose the mutual coverage model to measure the information coverage in the query-bag matching. 3) We release the composite Quora dataset to facilitate the research in this area.
Methodology
This task aims at predicting whether a query $q$ matches a bag $b$, where the bag is composed of some questions $b=\lbrace b_1, \dots , b_n \rbrace $ and $n$ is the number of questions in a bag. For the $q$ and $b_i$, an embedding layer is first applied to transform words to word embeddings via looking up word embedding table which is initialized by pre-trained word embeddings as in Section SECREF12.
In the following subsections, we will introduce our proposed Query-bag Matching (QBM) model which output is the matching probability indicating whether the query and bag are asking the same questions. The basic Q-Q (query-question) matching model hybrid CNN (hCNN) BIBREF5 is presented as the background. Then we will show the base model and its two components designed to promote the performance: Mutual Coverage and Bag Representation. For better understanding, the whole model is shown in Figure FIGREF2.
Methodology ::: Background: hCNN for Q-Q Matching
We adopt the hCNN model, which measures the relationship between query-question pairs, to obtain the Q-Q matching representation. The model can be easily adapted to other query-question matching models. hCNN is a CNN based matching model which is fast enough to work on the industry application. The input of hCNN is a query $q$ and the $i$-th question $b_i$ in the bag. $q$ and $b_i$ are fed into a CNN respectively. A cross-attention matrix $M^i$ is fed into another CNN to get the interaction representation between them. Each element of $M^i$ is defined as $M^i_{a,b}=q_a^\top \cdot b_{i,b}$ where $q_a$ is the word embedding of the $a$-th word in query $q$ and $b_{i,b}$ is the embedding of the $b$-th word in $b_i$. Finally, the outputs of CNNs are combined via Equation SECREF3 to get the representation $r_i$, which indicates the matching representation of the query $q$ and the $i$-th question $b_i$ in the bag. For the Q-Q matching task, the $r_i$ is fed into an MLP (Multi-Layer Perceptron) to predict the matching score. In our query-bag matching setting, we will aggregate the $\lbrace r_1, \dots , r_n\rbrace $ to predict the query-bag matching score. Due to the page limitation, please refer to BIBREF5 BIBREF5 for more details on hCNN. h1 = CNN1(q) h2i = CNN1(bi) hmi = CNN2(qbi)
ri = [h1; h2i; h1-h2i; h1 h2i;hmi]
Methodology ::: Base Model
After getting the Q-Q matching representation $r_i$, we combine the $\lbrace r_1, \dots , r_n\rbrace $ by element-wise max and mean pooling in order to get $r_p$ to represent the query-bag matching representation: rp = [ max_pooling { r1, ..., rn }; mean_pooling { r1, ..., rn } ] where [;] denotes concatenation. After that, an MLP with softmax is applied to predict whether the query and the bag is asking the same question. Finally, the loss function minimizes the cross entropy of the training data. Due to the out-of-order of the bag, we do not model the bag representation by CNN or LSTM, and experiments show the pooling-based method works well.
Methodology ::: Mutual Coverage
“How many parts of a query are covered by the bag?” and “Is all the information in the bag mentioned by the query?” are two important problems in the query-bag matching task. We propose a novel mutual coverage module to model the above-mentioned inter-projection problems.
Bag-to-query Considering the $i$-th question $b_i$ in the bag, the element-wise max pooling is performed on $\lbrace M^i_0, \cdots M^i_n \rbrace $ to get the $b_i$ to $q$ coverage $c_i=\text{max\_pooling}\lbrace M^i_0, \cdots M^i_n \rbrace $ where $M^i$ is the cross-attention matrix between $b_i$ and $q$ as in the background section, and $M^i_j$ is its $j$-th row. Each element $c_{i,j}$ represents how many information of the $j$-th word in $q$ is mentioned by the $i$-th question in the bag. To get the coverage from a bag instead of the $i$-th question in a bag, a new element-wise max pooling is applied on the generated $\lbrace c_1, \dots , c_n \rbrace $ to get bag-to-query coverage $c_q$. The process of the bag-to-query coverage is shown in Figure FIGREF2.
Query-to-bag The query-to-bag coverage is performed in a similar way. After getting the coverage $c_i$ from query $q$ to $b_i$. The concatenation of $\lbrace c_1, \dots , c_n \rbrace $ across all questions in a bag forms the query-to-bag coverage vector $c_b$.
In addition, not all words in a question should be treated equally. The word “the” contributes little to the matching degree in most cases. However, “package” is very important in the E-commerce scenario. We adopt the attention mechanism BIBREF6 to weight the coverage vector $c_q$ and $c_b$. The attention is calculated as follows (we take the bag-to-query coverage as an example): ej = MLP(qj) ej { ej } cq = e cq where $q_j$ is the embedding of $j$-th word in query. And the weighting of query-to-bag coverage performs in the same way. We call the mechanism coverage weighting.
The query-to-bag coverage, and bag-to-query coverage representation, and their summation are concatenated to the matching representation $r_p$ to predict the final score: [ rp ; cq ; cb ; sum(cq) ; sum(cb)]
Methodology ::: Bag Representation
All the questions in a bag follow the same question points because they are different variants of the same question. We model the question points by collecting the important words in the bag, forming the word-level bag representation. We collect the top-10 important words through TF-IDF algorithm, except stop words, in a bag to form a new “question” $b_r$, and an hCNN is used to model the relationship of the user query and the new “question” $b_r$ in order to obtain the matching representation $r_r$. The $r_r$ is then concatenated to the matching representation $r_p$ as a new feature to predict the query-bag matching degree. We also adopt the coverage mechanism discussed above over the cross-attention matrix between the query and the new “question”. The new coverage representation is also concatenated to the $r_p$.
Experiments ::: Dataset
We conduct experiments on two datasets: AliMe and Quora. The AliMe dataset is collected from the AliMe intelligent assistant system and the Quora dataset is composed of a public dataset.
AliMe For the AliMe service in E-commerce, we collect 8,004 query-bag pairs to form our dataset. Negative sampling is also an important part of the matching model. For each query, we use the Lucene to retrieval the top-20 most similar questions from the whole question candidates. Then we filter the questions which are in the corresponding right bag. After that, we randomly sample one in the retrieved candidate and use the bag that the retrieved candidate belongs to as the negative case. In the bag construction stage, the annotators have already merged all the questions of the same meaning, so we can ensure that the after filtering retrieved cases are negative in our setting. We also restrict the number of questions in a bag not more than 5 and discard the redundant questions. Finally, we get 12,008 training cases, 2,000 valid cases, and 10,000 test cases. Please notice, for the testing, we sampled 9 negative bags instead of 1, and thus formed 10 candidates for ranking.
Quora The Quora dataset is originally released for the duplicated question detection task. The dataset contains 400,000 question pairs and each pair is marked whether they are asking the same question. Due to the huge amount of duplicated question pairs, we group the questions as question bag via the union-find algorithm from the duplicated questions. We get 60,400 bags, and all the questions in a bag are asking the same question. We filter the bags that contain questions less than 3 to make the bag not too small. The new bag dataset will help similar questions recommendation on the Quora website. We then extract one question in the bag as query and the other questions make up the bag in our task. Considering the negative samples, we follow the same strategy as AliMe dataset. Finally, we get 20,354 training set, 2,000 validation set, and 10,000 test set. To facilitate the research in this area, the composed Quora dataset are released.
Experiments ::: Setup
We use the Adam optimizer with learning rate 0.0001 to optimize the parameters. The batch size is 32. The dropout rate is 0.5. The max length of the query and questions is 20 to cover most of the words in a sentence. We use padding to handle the various lengths of the text. The model checkpoint is chosen according to the best F-score on the validation set. The word embedding dimension is 300, and the pre-trained word embedding is from Sina and Glove for AliMe and Quora dataset respectively. Besides, the embedding is tuned while the model training to get better performance.
Experiments ::: Baselines
To prove the effectiveness of our models, we propose two baselines from different aspects: the Q-Q matching based baseline and the query-bag matching based baseline.
Q-Q Matching One starting point behind our work is that the query-bag matching may work better than the Q-Q matching for the information-seeking conversation. To verify such opinion, we propose the Q-Q matching based baseline and compare our model with two instances of the baseline. We extract the query-question pairs form the query-bag pair. The label of the query-bag pair is assigned to the new query-question pairs. An hCNN model is applied to train the new dataset. In the testing stage, each query-question pair is assigned with a probability indicating the matching degree. To compare with our model, we rank the bags based on the query-bag matching scores and the scores are defined as the max or mean matching probability of the query-question pairs in the query-bag pair. We name the two instances Q-Q Max and Q-Q Mean respectively.
Query-bag Matching To verify the effectiveness of our proposed models, We design a new query-bag matching based baseline. We concatenate the questions in the bag to form a new long “question”, then the hCNN model is applied to measure the matching degree of the original query and the new “question”, namely Bag-Con (Bag Concatenation).
Experiments ::: Evaluation
Following BIBREF7, we evaluate the model performance on five automatic evaluation metrics: MRR, $\text{R}_{10}@1$, $\text{R}_{10}@2$, $\text{R}_{10}@5$, and $\text{R}_{2}@1$. $\text{R}_n@k$ calculates the recall of the true positive pre-defined questions among the $k$ selected candidates from $n$ available candidates. And Mean Reciprocal Rank (MRR) is another popular measurement for ranking problems.
Results and Analysis
Results and Ablation Study The results are shown in Table TABREF6. Our model (QBM) performs best compared to baselines (Q-Q Mean, Q-Q Max, Bag-con). Comparing Bag-Con and Base model, we find that modelling the query-question relationship following aggregation works better. We assume that the pooling-based aggregation can reduce the redundant information cross sentences in a bag. Considering the Q-Q matching based methods and query-bag based methods. In AliMe dataset, the query-bag matching outperforms the Q-Q matching based methods which shows the necessity to perform query-bag matching. The ablation study shows that the mutual coverage component and bag representation component achieve better performance than the base model, especially in the Quora dataset. The two components work independently and their combination gets the best performance.
UTF8gbsn
Effectiveness of the Mutual Coverage To intuitively learn the coverage weight, we sample some words with their weights in Table TABREF17. It shows that the words like “The” have low weight, which confirms that they contribute little to the matching. “Refund” in E-commerce is a very important element in a user query sentence. And “America” is important in Quora, because question like “what is the capital in <location>?” is highly related to location “<location>”.
Analysis of the Bag Representation Coverage is also applied in the bag representation layer. The results of the bag representation without coverage component (Base+(BR w/o Cov)) is shown in Table TABREF6. Compared with the Base+BR and BR without coverage, it shows that the coverage component contributes a lot on both the two datasets. The bag representation with coverage (Base+BR) gains improvement over Base model, especially in Quora dataset.
Conclusion
In this paper, we propose the QBM model which performs the query-bag matching in information-seeking conversation. Experiments show that the proposed mutual coverage component improves the model performance. And the model can automatically discover important words in the query or bag from both the coverage weighting component and the word-level bag representation. This work also shows that learning the query-bag matching directly in some scenarios may outperform the query-question matching in ranking bags. One advantage of our model is that it is extensible in replacing the query-question matching component.

Question: Does the query-bag matching model use a neural network?
Answer: Yes

Yes

Yes

Question: What datasets are used for experiments?
Answer: the AliMe and Quora dataset

AliMe and Quora

AliMe , Quora

Question: Which natural language(s) is/are studied?
Answer: Unanswerable

Chinese and English

Unanswerable

Question: Is model compared to some baseline?
Answer: Yes

Question: What datasets are used in experiments?
Answer:  the AliMe and Quora dataset