Classification Betters Regression in Query-based Multi-document Summarisation Techniques for Question Answering: Macquarie University at BioASQ7b

Abstract: Task B Phase B of the 2019 BioASQ challenge focuses on biomedical question answering. Macquarie University's participation applies query-based multi-document extractive summarisation techniques to generate a multi-sentence answer given the question and the set of relevant snippets. In past participation we explored the use of regression approaches using deep learning architectures and a simple policy gradient architecture. For the 2019 challenge we experiment with the use of classification approaches with and without reinforcement learning. In addition, we conduct a correlation analysis between various ROUGE metrics and the BioASQ human evaluation scores.

The BioASQ Challenge includes a question answering task (Phase B, part B) where the aim is to find the “ideal answer” — that is, an answer that would normally be given by a person BIBREF0. This is in contrast with most other question answering challenges where the aim is normally to give an exact answer, usually a fact-based answer or a list. Given that the answer is based on an input that consists of a biomedical question and several relevant PubMed abstracts, the task can be seen as an instance of query-based multi-document summarisation.
As in past participation BIBREF1, BIBREF2, we wanted to test the use of deep learning and reinforcement learning approaches for extractive summarisation. In contrast with past years where the training procedure was based on a regression set up, this year we experiment with various classification set ups. The main contributions of this paper are:
We compare classification and regression approaches and show that classification produces better results than regression but the quality of the results depends on the approach followed to annotate the data labels.
We conduct correlation analysis between various ROUGE evaluation metrics and the human evaluations conducted at BioASQ and show that Precision and F1 correlate better than Recall.
Section SECREF2 briefly introduces some related work for context. Section SECREF3 describes our classification and regression experiments. Section SECREF4 details our experiments using deep learning architectures. Section SECREF5 explains the reinforcement learning approaches. Section SECREF6 shows the results of our correlation analysis between ROUGE scores and human annotations. Section SECREF7 lists the specific runs submitted at BioASQ 7b. Finally, Section SECREF8 concludes the paper.
The BioASQ challenge has organised annual challenges on biomedical semantic indexing and question answering since 2013 BIBREF0. Every year there has been a task about semantic indexing (task a) and another about question answering (task b), and occasionally there have been additional tasks. The tasks defined for 2019 are:
Large Scale Online Biomedical Semantic Indexing.
Biomedical Semantic QA involving Information Retrieval (IR), Question Answering (QA), and Summarisation.
Medical Semantic Indexing in Spanish.
BioASQ Task 7b consists of two phases. Phase A provides a biomedical question as an input, and participants are expected to find relevant concepts from designated terminologies and ontologies, relevant articles from PubMed, relevant snippets from the relevant articles, and relevant RDF triples from designated ontologies. Phase B provides a biomedical question and a list of relevant articles and snippets, and participant systems are expected to return the exact answers and the ideal answers. The training data is composed of the test data from all previous years, and amounts to 2,747 samples. There has been considerable research on the use of machine learning approaches for tasks related to text summarisation, especially on single-document summarisation. Abstractive approaches normally use an encoder-decoder architecture and variants of this architecture incorporate attention BIBREF3 and pointer-generator BIBREF4. Recent approaches leveraged the use of pre-trained models BIBREF5. Recent extractive approaches to summarisation incorporate recurrent neural networks that model sequences of sentence extractions BIBREF6 and may incorporate an abstractive component and reinforcement learning during the training stage BIBREF7. But relatively few approaches have been proposed for query-based multi-document summarisation. Table TABREF8 summarises the approaches presented in the proceedings of the 2018 BioASQ challenge.
Our past participation in BioASQ BIBREF1, BIBREF2 and this paper focus on extractive approaches to summarisation. Our decision to focus on extractive approaches is based on the observation that a relatively large number of sentences from the input snippets has very high ROUGE scores, thus suggesting that human annotators had a general tendency to copy text from the input to generate the target summaries BIBREF1. Our past participating systems used regression approaches using the following framework:
Train the regressor to predict the ROUGE-SU4 F1 score of the input sentence.
Produce a summary by selecting the top $n$ input sentences.
A novelty in the current participation is the introduction of classification approaches using the following framework.
Train the classifier to predict the target label (“summary” or “not summary”) of the input sentence.
Produce a summary by selecting all sentences predicted as “summary”.
If the total number of sentences selected is less than $n$, select $n$ sentences with higher probability of label “summary”.
Introducing a classifier makes labelling the training data not trivial, since the target summaries are human-generated and they do not have a perfect mapping to the input sentences. In addition, some samples have multiple reference summaries. BIBREF11 showed that different data labelling approaches influence the quality of the final summary, and some labelling approaches may lead to better results than using regression. In this paper we experiment with the following labelling approaches:
: Label as “summary” all sentences from the input text that have a ROUGE score above a threshold $t$.
: Label as “summary” the $m$ input text sentences with highest ROUGE score.
As in BIBREF11, The ROUGE score of an input sentence was the ROUGE-SU4 F1 score of the sentence against the set of reference summaries.
We conducted cross-validation experiments using various values of $t$ and $m$. Table TABREF26 shows the results for the best values of $t$ and $m$ obtained. The regressor and classifier used Support Vector Regression (SVR) and Support Vector Classification (SVC) respectively. To enable a fair comparison we used the same input features in all systems. These input features combine information from the question and the input sentence and are shown in Fig. FIGREF16. The features are based on BIBREF12, and are the same as in BIBREF1, plus the addition of the position of the input snippet. The best SVC and SVR parameters were determined by grid search.
Preliminary experiments showed a relatively high number of cases where the classifier did not classify any of the input sentences as “summary”. To solve this problem, and as mentioned above, the summariser used in Table TABREF26 introduces a backoff step that extracts the $n$ sentences with highest predicted values when the summary has less than $n$ sentences. The value of $n$ is as reported in our prior work and shown in Table TABREF25.
The results confirm BIBREF11's finding that classification outperforms regression. However, the actual choice of optimal labelling scheme was different: whereas in BIBREF11 the optimal labelling was based on a labelling threshold of 0.1, our experiments show a better result when using the top 5 sentences as the target summary. The reason for this difference might be the fact that BIBREF11 used all sentences from the abstracts of the relevant PubMed articles, whereas we use only the snippets as the input to our summariser. Consequently, the number of input sentences is now much smaller. We therefore report the results of using the labelling schema of top 5 snippets in all subsequent classifier-based experiments of this paper.
barchart=[fill=black!20,draw=black] errorbar=[very thin,draw=black!75] sscale=[very thin,draw=black!75]
Based on the findings of Section SECREF3, we apply minimal changes to the deep learning regression models of BIBREF2 to convert them to classification models. In particular, we add a sigmoid activation to the final layer, and use cross-entropy as the loss function. The complete architecture is shown in Fig. FIGREF28.
The bottom section of Table TABREF26 shows the results of several variants of the neural architecture. The table includes a neural regressor (NNR) and a neural classifier (NNC). The neural classifier is trained in two set ups: “NNC top 5” uses classification labels as described in Section SECREF3, and “NNC SU4 F1” uses the regression labels, that is, the ROUGE-SU4 F1 scores of each sentence. Of interest is the fact that “NNC SU4 F1” outperforms the neural regressor. We have not explored this further and we presume that the relatively good results are due to the fact that ROUGE values range between 0 and 1, which matches the full range of probability values that can be returned by the sigmoid activation of the classifier final layer.
Table TABREF26 also shows the standard deviation across the cross-validation folds. Whereas this standard deviation is fairly large compared with the differences in results, in general the results are compatible with the top part of the table and prior work suggesting that classification-based approaches improve over regression-based approaches.
We also experiment with the use of reinforcement learning techniques. Again these experiments are based on BIBREF2, who uses REINFORCE to train a global policy. The policy predictor uses a simple feedforward network with a hidden layer.
The results reported by BIBREF2 used ROUGE Recall and indicated no improvement with respect to deep learning architectures. Human evaluation results are preferable over ROUGE but these were made available after the publication of the paper. When comparing the ROUGE and human evaluation results (Table TABREF29), we observe an inversion of the results. In particular, the reinforcement learning approaches (RL) of BIBREF2 receive good human evaluation results, and as a matter of fact they are the best of our runs in two of the batches. In contrast, the regression systems (NNR) fare relatively poorly. Section SECREF6 expands on the comparison between the ROUGE and human evaluation scores.
Encouraged by the results of Table TABREF29, we decided to continue with our experiments with reinforcement learning. We use the same features as in BIBREF2, namely the length (in number of sentences) of the summary generated so far, plus the $tf.idf$ vectors of the following:
Candidate sentence;
Entire input to summarise;
Summary generated so far;
Candidate sentences that are yet to be processed; and
Question.
The reward used by REINFORCE is the ROUGE value of the summary generated by the system. Since BIBREF2 observed a difference between the ROUGE values of the Python implementation of ROUGE and the original Perl version (partly because the Python implementation does not include ROUGE-SU4), we compare the performance of our system when trained with each of them. Table TABREF35 summarises some of our experiments. We ran the version trained on Python ROUGE once, and the version trained on Perl twice. The two Perl runs have different results, and one of them clearly outperforms the Python run. However, given the differences of results between the two Perl runs we advice to re-run the experiments multiple times and obtain the mean and standard deviation of the runs before concluding whether there is any statistical difference between the results. But it seems that there may be an improvement of the final evaluation results when training on the Perl ROUGE values, presumably because the final evaluation results are measured using the Perl implementation of ROUGE.
We have also tested the use of word embeddings instead of $tf.idf$ as input features to the policy model, while keeping the same neural architecture for the policy (one hidden layer using the same number of hidden nodes). In particular, we use the mean of word embeddings using 100 and 200 dimensions. These word embeddings were pre-trained using word2vec on PubMed documents provided by the organisers of BioASQ, as we did for the architectures described in previous sections. The results, not shown in the paper, indicated no major improvement, and re-runs of the experiments showed different results on different runs. Consequently, our submission to BioASQ included the original system using $tf.idf$ as input features in all batches but batch 2, as described in Section SECREF7.
As mentioned in Section SECREF5, there appears to be a large discrepancy between ROUGE Recall and the human evaluations. This section describes a correlation analysis between human and ROUGE evaluations using the runs of all participants to all previous BioASQ challenges that included human evaluations (Phase B, ideal answers). The human evaluation results were scraped from the BioASQ Results page, and the ROUGE results were kindly provided by the organisers. We compute the correlation of each of the ROUGE metrics (recall, precision, F1 for ROUGE-2 and ROUGE-SU4) against the average of the human scores. The correlation metrics are Pearson, Kendall, and a revised Kendall correlation explained below.
The Pearson correlation between two variables is computed as the covariance of the two variables divided by the product of their standard deviations. This correlation is a good indication of a linear relation between the two variables, but may not be very effective when there is non-linear correlation.
The Spearman rank correlation and the Kendall rank correlation are two of the most popular among metrics that aim to detect non-linear correlations. The Spearman rank correlation between two variables can be computed as the Pearson correlation between the rank values of the two variables, whereas the Kendall rank correlation measures the ordinal association between the two variables using Equation DISPLAY_FORM36.
It is useful to account for the fact that the results are from 28 independent sets (3 batches in BioASQ 1 and 5 batches each year between BioASQ 2 and BioASQ 6). We therefore also compute a revised Kendall rank correlation measure that only considers pairs of variable values within the same set. The revised metric is computed using Equation DISPLAY_FORM37, where $S$ is the list of different sets.
Table TABREF38 shows the results of all correlation metrics. Overall, ROUGE-2 and ROUGE-SU4 give similar correlation values but ROUGE-SU4 is marginally better. Among precision, recall and F1, both precision and F1 are similar, but precision gives a better correlation. Recall shows poor correlation, and virtually no correlation when using the revised Kendall measure. For reporting the evaluation of results, it will be therefore more useful to use precision or F1. However, given the small difference between precision and F1, and given that precision may favour short summaries when used as a function to optimise in a machine learning setting (e.g. using reinforcement learning), it may be best to use F1 as the metric to optimise.
Fig. FIGREF40 shows the scatterplots of ROUGE-SU4 recall, precision and F1 with respect to the average human evaluation. We observe that the relation between ROUGE and the human evaluations is not linear, and that Precision and F1 have a clear correlation.
Table TABREF41 shows the results and details of the runs submitted to BioASQ. The table uses ROUGE-SU4 Recall since this is the metric available at the time of writing this paper. However, note that, as explained in Section SECREF6, these results might differ from the final human evaluation results. Therefore we do not comment on the results, other than observing that the “first $n$” baseline produces the same results as the neural regressor. As mentioned in Section SECREF3, the labels used for the classification experiments are the 5 sentences with highest ROUGE-SU4 F1 score.
Macquarie University's participation in BioASQ 7 focused on the task of generating the ideal answers. The runs use query-based extractive techniques and we experiment with classification, regression, and reinforcement learning approaches. At the time of writing there were no human evaluation results, and based on ROUGE-F1 scores under cross-validation on the training data we observed that classification approaches outperform regression approaches. We experimented with several approaches to label the individual sentences for the classifier and observed that the optimal labelling policy for this task differed from prior work.
We also observed poor correlation between ROUGE-Recall and human evaluation metrics and suggest to use alternative automatic evaluation metrics with better correlation, such as ROUGE-Precision or ROUGE-F1. Given the nature of precision-based metrics which could bias the system towards returning short summaries, ROUGE-F1 is probably more appropriate when using at development time, for example for the reward function used by a reinforcement learning system.
Reinforcement learning gives promising results, especially in human evaluations made on the runs submitted to BioASQ 6b. This year we introduced very small changes to the runs using reinforcement learning, and will aim to explore more complex reinforcement learning strategies and more complex neural models in the policy and value estimators.

Figure and table captions:
Table 1. Summarisation techniques used in BioASQ 6b for the generation of ideal answers. The evaluation result is the human evaluation of the best run.
Fig. 2. Architecture of the neural classification and regression systems. A matrix of pre-trained word embeddings (same pre-trained vectors as in Fig. 1) is used to find the embeddings of the words of the input sentence and the question. Then, LSTM chains are used to generate sentence embeddings — the weights of the LSTM chains of input sentence and question are not shared. Then, the sentence position is concatenated to the sentence embedding and the similarity of sentence and question embeddings, implemented as a product. A final layer predicts the label of the sentence.
Table 5. Experiments using Perl and Python versions of ROUGE. The Python version used the average of ROUGE-2 and ROUGE-L, whereas the Perl version used ROUGESU4.
Table 6. Correlation analysis of evaluation results
Table 7. Runs submitted to BioASQ 7b
Fig. 3. Scatterplots of ROUGE SU4 evaluation metrics against the average human evaluations.

Questions about this paper:
What approaches without reinforcement learning have been tried?

Answers:
classification, regression, neural methods