user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Domain-Relevant Embeddings for Question Similarity

Paper content:
The rate at which medical questions are asked online significantly exceeds the capacity of qualified people to answer them, leaving many questions unanswered or inadequately answered.
Many of these questions are not unique, and reliable identification of similar questions would enable more efficient and effective question answering schema.
While many research efforts have focused on the problem of general question similarity, these approaches do not generalize well to the medical domain, where medical expertise is often required to determine semantic similarity.
In this paper, we show how a semi-supervised approach of pre-training a neural network on medical question-answer pairs is a particularly useful intermediate task for the ultimate goal of determining medical question similarity.
While other pre-training tasks yield an accuracy below 78.7% on this task, our model achieves an accuracy of 82.6% with the same number of training examples, an accuracy of 80.0% with a much smaller training set, and an accuracy of 84.5% when the full corpus of medical question-answer data is used.
With the ubiquity of the Internet and the emergence of medical question-answering websites such as ADAM (www.adam.com), WebMD (www.webmd.com), and HealthTap (www.healthtap. com), people are increasingly searching online for answers to their medical questions.
However, the number of people asking medical questions online far exceeds the number of qualified experts -i.e doctors -answering them.
One way to address this imbalance is to build a system that can automatically match unanswered questions with semantically similar answered questions, or mark them as priority if no similar answered questions exist.
This approach uses doctor time more efficiently, reducing the number of unanswered questions and lowering the cost of providing online care.
Many of the individuals seeking medical advice online are otherwise reluctant to seek medical help due to cost, convenience, or embarrassment.
For these patients, an accurate online system is critical because it may be the only medical advice they receive.
Of course, some medical problems require in-person care, and an online system must indicate that.
Other patients use the internet in addition to in-person care either to determine when an appointment is needed or to follow up after visits when they have lingering questions.
For this second group, if the answers they see online do not match those given to them by their doctors, they are less likely to follow the advice of their doctors (Nosta, 2017) , which can have serious consequences.
Coming up with an accurate algorithm for finding similar medical questions, however, is difficult.
Simple heuristics such as word-overlap are ineffective because Can a menstrual blood clot travel to your heart or lungs like other blood clots can?
and Can clots from my period cause a stroke or embolism?
are similar questions with low overlap, but Is candida retested after treatment and Is Chlamydia retested after treatment?
are critically different and only one word apart.
Machine learning is a good candidate for such complex tasks, but requires labeled training data.
As no widely available data for this particular task exists, we generate and release our own dataset of medical question pairs such as the ones shown in Table 1 .
Given the recent success of pre-trained bi-directional transformer networks for natural language processing (NLP) outside the medical field (Peters et al., 2018; Devlin et al., 2018; Radford et al.; Yang et al., 2019; Liu et al., 2019) , most research efforts in medical NLP have tried to apply general .
However, these models are not trained on medical information, and make errors that reflect this.
In this work, we augment the features in these general language models using the depth of information that is stored within a medical question-answer pair to embed medical knowledge into the model.
Our models pre-trained on this task outperform models pre-trained on out-of-domain question similarity with high statistical significance, and the results show promise of generalizing to other domains as well.
The task of question-answer matching was specifically chosen because it is closely related to that of question similarity; one component of whether or not two questions are semantically similar is whether or not the answer to one also answers the other.
We show that the performance gains achieved by this particular task are not realized by other in-domain tasks, such as medical questioncategorization and medical answer completion.
The main contributions of this paper are:
• We release a dataset of medical question pairs generated and labeled by doctors that is based upon real, patient-asked questions
• We prove that, particularly for medical NLP, domain matters: pre-training on a different task in the same domain outperforms pre-training on the same task in a different domain
• We show that the task of question-answer matching embeds relevant medical information for question similarity that is not captured by other in-domain tasks 2 RELATED WORK 2.1 PRE-TRAINED NETWORKS FOR GENERAL LANGUAGE UNDERSTANDING NLP has undergone a transfer learning revolution in the past year, with several large pre-trained models earning state-of-the-art scores across many linguistic tasks.
Two such models that we use in our own experiments are BERT (Devlin et al., 2018) and XLNet (Yang et al., 2019) .
These models have been trained on semi-supervised tasks such as predicting a word that has been masked out from a random position in a sentence, and predicting whether or not one sentence is likely to follow another.
The corpus used to train BERT was exceptionally large (3.3 billion words), but all of the data came from BooksCorpus and Wikipedia.
Talmor & Berant (2019) recently found that BERT generalizes better to other datasets drawn from Wikipedia than to tasks using other web snippets.
This is consistent with our finding that pre-training domain makes a big difference.
To understand the broader applicability of our findings, we apply our approach to a non-medical domain: the AskUbuntu question-answer pairs from Lei et al. (2016) .
As before, we avoid making the pre-training task artificially easy by creating negatives from related questions.
This time, since there are no category labels, we index all of the data with Elasticsearch 1 .
For the question similarity task, the authors have released a candidate set of pairs that were human labeled as similar or dissimilar.
Without any pre-training (baseline), we observe an accuracy of 65.3% ± 1.2% on the question similarity task.
Pre-training on QQP leads to a significant reduction in accuracy to 62.3% ± 2.1% indicating that an out-of-domain pretraining task can actually hurt performance.
When the QA task is used for intermediate pre-training, the results improve to 66.6% ± 0.9%.
While this improvement may not be statistically significant, it is consistent with the main premise of our work that related tasks in the same domain can help performance.
We believe that the low accuracy on this task, as well as the small inter-model performance gains, may be due to the exceptionally long question lengths, some of which are truncated by the models during tokenization.
In the future, we would explore ways to reduce the length of these questions before feeding them into the model.
In this work, we release a medical question-pairs dataset and show that the semi-supervised approach of pre-training on in-domain question-answer matching (QA) is particularly useful for the difficult task of duplicate question recognition.
Although the QA model outperforms the out-of-domain same-task QQP model, there are a few examples where the QQP model seems to have learned information that is missing from the QA model (see Appendix A).
In the future, we can further explore whether these two models learned independently useful information from their pre-training tasks.
If they did, then we hope to be able to combine these features into one model with multitask learning.
An additional benefit of the error analysis is that we have a better understanding of the types of mistakes that even our best model is making.
It is therefore now easier to use weak supervision and augmentation rules to supplement our datasets to increase the number of training examples in those difficult regions of the data.
With both of these changes, we expect to be able to bump up accuracy on this task by several more percentage points.

assistant: text: We show that question-answer matching is a particularly good pre-training task for question-similarity and release a dataset for medical question similarity