Venue: NAACL 2016

# Bayesian Supervised Domain Adaptation for Short Text Similarity

Md Arafat Sultan\({}^{1,2}\)  Jordan Boyd-Graber\({}^{2}\)   Tamara Sumner\({}^{1,2}\)

\({}^{1}\)Institute of Cognitive Science

\({}^{2}\)Department of Computer Science

University of Colorado, Boulder, CO

{arafat.sultan,Jordan.Boyd.Graber,sumer}@colorado.edu

###### Abstract

Identification of short text similarity (sts) is a high-utility NLP task with applications in a variety of domains. We explore adaptation of sts algorithms to different target domains and applications. A two-level hierarchical Bayesian model is employed for domain adaptation (da) of a linear sts model to text from different sources (e.g., news, tweets). This model is then further extended for multitask learning (mtl) of three related tasks: sts, short answer scoring (sas) and answer sentence ranking (asr). In our experiments, the adaptive model demonstrates better overall cross-domain and cross-task performance over two non-adaptive baselines.

## 1 Short Text Similarity: The Need for Domain Adaptation

Given two snippets of text--neither longer than a few sentences--short text similarity (sts) determines how semantically close they are. sts has a broad range of applications: question answering Yao et al. (2013); Severyn and Moschitti (2015), text summarization Dasgupta et al. (2013); Wang et al. (2013), machine translation evaluation Chan and Ng (2008); Liu et al. (2011), and grading of student answers in academic tests Mohler et al. (2011); Ramachandran et al. (2015).

sts is typically viewed as a _supervised_ machine learning problem Bar et al. (2012); Lynum et al. (2014); Hanig et al. (2015). SemEval contests Agirre et al. (2012); Agirre et al. (2015) have spurred recent progress in sts and have provided valuable training data for these supervised approaches. However, similarity varies across domains, as does the underlying text; e.g., syntactically well-formed academic text versus informal English in forum qa.

Our goal is to effectively use domain adaptation (da) to transfer information from these disparate sts domains. While "domain" can take a range of meanings, we consider adaptation to different (1) sources of text (e.g., news headlines, tweets), and (2) applications of sts (e.g., qa vs. answer grading). Our goal is to improve performance in a new domain with few in-domain annotations by using many out-of-domain ones (Section 2).

In Section 3, we describe our Bayesian approach that posits that per-domain parameter vectors share a common Gaussian prior that represents the global parameter vector. Importantly, this idea can be extended with little effort to a nested domain hierarchy (domains within domains), which allows us to create a single, unified sts model that _generalizes across domains as well as tasks_, capturing the nuances that an sts system must have for tasks such as short answer scoring or question answering.

We compare our da methods against two baselines: (1) a domain-agnostic model that uses all training data and does not distinguish between in-domain and out-of-domain examples, and (2) a model that learns only from in-domain examples. Section 5 shows that across ten different sts domains, the adaptive model consistently outperforms the first baseline while performing at least as well as the second across training datasets of different sizes. Our multitask model also yields better overall results over the same baselines across three related tasks: (1) sts, (2) short answer scoring (sas), and (3) answer sentence ranking (asr) for question answering.



Tasks and Datasets

Short Text Similarity (sts)Given two short texts, sts provides a real-valued score that represents their degree of semantic similarity. Our sts datasets come from the SemEval 2012-2015 corpora, containing over 14,000 human-annotated sentence pairs (via Amazon Mechanical Turk) from domains like news, tweets, forum posts, and image descriptions.

For our experiments, we select ten datasets from ten different domains, containing 6,450 sentence pairs.1 This selection is intended to maximize (a) the number of domains, (b) domain uniqueness: of three different news headlines datasets, for example, we select the most recent [38], discarding older ones [37, 39], and (c) amount of per-domain data available: we exclude the fnwn[37] dataset with 189 annotations, for example, because it limits per-domain training data in our experiments. Sizes of the selected datasets range from 375 to 750 pairs. Average correlation (Pearson's \(r\)) among annotators ranges from 58.6% to 88.8% on individual datasets (above 70% for most) [1, 1, 13].

Footnote 1: [https://www.stsci.edu/~franck/](https://www.stsci.edu/~franck/)

Short Answer Scoring (ss)sas comes in different forms; we explore a form where for a short-answer question, a gold answer is provided, and the goal is to grade student answers based on how similar they are to the gold answer [15]. We use a dataset of undergraduate data structures questions and student responses graded by two judges [16]. These questions are spread across ten different assignments and two examinations, each on a related set of topics (e.g., programming basics, sorting algorithms). Inter-annotator agreement is 58.6% (Pearson's \(\rho\)) and 0.659 (rmse on a 5-point scale). We discard assignments with fewer than 200 pairs, retaining 1,182 student responses to forty questions spread across five assignments and tests.2

Footnote 2: [https://github.com/facebook/](https://github.com/facebook/)

Answer Sentence Ranking (asr)Given a factoid question and a set of candidate answer sentences, asr orders candidates so that sentences containing the answer are ranked higher. Text similarity is the foundation of most prior work: a candidate sentence's relevance is based on its similarity with the question [23, 24, 25].

Footnote 3: _what_, _when_, _who_ and _how many_.

For our asr experiments, we use factoid questions developed by wangetal07 from Text REtrieval Conferences (trec) 8-13. Candidate qa pairs of a question and a candidate were labeled with whether the candidate answers the question. The questions are of different types (e.g., _what_, _where_); we retain 2,247 qa pairs under four question types, each with at least 200 answer candidates in the combined development and test sets.3 Each question type represents a unique topical domain--_who_ questions are about persons and _how many_ questions are about quantities.

Footnote 3: _what_, _when_, _who_ and _how many_.

## 3 Bayesian Domain Adaptation for sts

We first discuss our base linear models for the three tasks: Bayesian \(L_{2}\)-regularized linear (for sts and sas) and logistic (for asr) regression. We extend these models for (1) adaptation across different short text similarity domains, and (2) multitask learning of short text similarity (sts), short answer scoring (sas), and answer sentence ranking (asr).

### Base Models

In our base models (Figure 1), the feature vector combines with the feature weight vector \(\mathbf{w}\) (including a bias term \(w_{0}\)) to form predictions. Each parameter \(w_{i}\in\mathbf{w}\) has its own zero-mean Gaussian prior with its standard deviation \(\sigma_{w_{i}}\) distributed uniformly in \([0,m_{\sigma_{w}}]\), the covariance matrix \(\mathbf{\Sigma}_{w}\) is diagonal, and the zero-mean prior \(L_{2}\) regularizes the model.

In the linear model (Figure 0(a)), \(S\) is the output (similarity score for sts; answer score for sas) and is normally distributed around the dot product \(\mathbf{w}^{T}\mathbf{f}\). The model error \(\sigma_{S}\) has a uniform prior over a pre-specified range \([0,m_{\sigma_{S}}]\). In the logistic model (Figure 0(b)) for asr, the probability \(p\) that the candidate sentence answers the question, is (1) the sigmoid of \(\mathbf{w}^{T}\mathbf{f}\), and (2) the Bernoulli prior of \(A\), whether or not the candidate answers the question.

The common vectors \(\mathbf{w}\) and \(\mathbf{f}\) in these models enable joint parameter learning and consequently multitask learning (Section 3.3).

 



### Adaptation to sts Domains

Domain adaptation for the linear model (Figure 0(a)) learns a separate weight vector \({\mathbf{w}_{d}}\) for each domain \(d\) (i.e., applied to similarity computations for test pairs in domain \(d\)) alongside a common, global domain-agnostic weight vector \({\mathbf{w}_{*}}\), which has a zero-mean Gaussian prior and serves as the Gaussian prior mean for each \({\mathbf{w}_{d}}\). Figure 2 shows the model. Both \({\mathbf{w}_{*}}\) and \({\mathbf{w}_{d}}\) have hyperpriors identical to \({\mathbf{w}}\) in Figure 0(a).4

Footnote 4: Results do not improve with individual domain-specific instances of \(\sigma_{S}\) and \({\mathbf{\sigma}_{w}}\), consistent with Finkel and Manning (2009) for dependency parsing and named entity recognition.

Each \({\mathbf{w}_{d}}\) depends not just on its domain-specific observations but also on information derived from the global, shared parameter \({\mathbf{w}_{*}}\). The balance between capturing in-domain information and inductive transfer is regulated by \(\Sigma_{w}\); larger variance allows \({\mathbf{w}_{d}}\) more freedom to reflect the domain.

### Multitask Learning

An advantage of hierarchical da is that it extends easily to arbitrarily nested domains. Our multitask learning model (Figure 3) models topical domains nested within one of three related tasks: sts, sas, and asr (Section 2). This model adds a level to the hierarchy of weight vectors: each domain-level \({\mathbf{w}_{d}}\) is now normally distributed around a task-level weight vector (e.g., \({\mathbf{w}_{\text{STS}}}\)), which in turn has global Gaussian mean \({\mathbf{w}_{*}}\).5 Like the da model, all weights in the same level share common variance hyperparameters while those across different levels are separate.

Footnote 5: We use the same variable for the domain-specific parameter \({\mathbf{w}_{d}}\) across tasks to simplify notation.

Again, this hierarchical structure (1) jointly learns global, task-level and domain-level feature weights enabling inductive transfer among tasks and domains while (2) retaining the distinction between in-domain and out-of-domain annotations. A task-specific model (Figure 1) that only learns from in-domain annotations supports only (2). On the other hand, a non-hierarchical joint model (Figure 4) supports only (1): it learns a single shared \({\mathbf{w}}\) applied to any test pair regardless of task or domain. We compare these models in Section 5.





## 4 Features

Any feature-based sts model can serve as the base model for a hierarchical Bayesian adaptation framework. For our experiments, we adopt the feature set of the ridge regression model in Sultan et al. (2015), the best-performing system at SemEval-2015 Agirre et al. (2015).

Input sentences \(S^{(1)}=(w_{1}^{(1)},...,w_{n}^{(1)})\) and \(S^{(2)}=(w_{1}^{(2)},...,w_{m}^{(2)})\) (where each \(w\) is a token) produce two similarity features. The first is the proportion of content words in \(S^{(1)}\) and \(S^{(2)}\) (combined) that have a semantically similar word--identified using a monolingual word aligner Sultan et al. (2014)--in the other sentence. The overall semantic similarity of a word pair \((w_{i}^{(1)},w_{j}^{(2)})\in S^{(1)}\times S^{(2)}\) is a weighted sum of lexical and contextual similarities: a paraphrase database Ganitkevitch et al. (2013); ppdb) identifies lexically similar words; contextual similarity is the average lexical similarity in (1) dependencies of \(w_{i}^{(1)}\) in \(S^{(1)}\) and \(w_{j}^{(2)}\) in \(S^{(2)}\), and (2) content words in [-3, 3] windows around \(w_{i}^{(1)}\) in \(S^{(1)}\) and \(w_{j}^{(2)}\) in \(S^{(2)}\). Lexical similarity scores of pairs in ppdb as well as weights of word and contextual similarities are optimized on an alignment dataset Brockett (2007). To avoid penalizing long answer snippets (that still have the desired semantic content) in sas and asr, word alignment proportions outside the reference (gold) answer (sas) and the question (asr) are ignored.

The second feature captures finer-grained similarities between related words (e.g., cell and organism). Given the 400-dimensional embedding Baroni et al. (2014) of each content word (lemmatized) in an input sentence, we compute a sentence vector by adding its content lemma vectors. The co

Figure 4: A non-hierarchical joint model for sts, sas and asr. A common weight vector \(\mathbf{w}\) is learned for all tasks and domains.



sine similarity between the \(S^{(1)}\) and \(S^{(2)}\) vectors is then used as an sts feature. Baroni et al. develop the word embeddings using word2vec6 from a corpus of about 2.8 billion tokens, using the Continuous Bag-of-Words (cbow) model proposed by Mikolov et al. (2013).

Footnote 5: [https://code.google.com/p/word2vec/](https://code.google.com/p/word2vec/)

## 5 Experiments

For each of the three tasks, we first assess the performance of our base model to (1) verify our sampling-based Bayesian implementations, and (2) compare to the state of the art. We train each model with a Metropolis-within-Gibbs sampler with 50,000 samples using PyMC (Patil et al., 2010; Salvatier et al., 2015), discarding the first half of the samples as burn-in. The variances \(m_{\sigma_{w}}\) and \(m_{\sigma_{S}}\) are both set to 100. Base models are evaluated on the entire test set for each task, and the same training examples as in the state-of-the-art systems are used. Table 1 shows the results.

Following SemEval, we report a weighted sum of correlations (Pearson's \(r\)) across all test sets for sts, where the weight of a test set is proportional to its number of pairs. Our model and Sultan et al. (2015) are almost identical on all twenty test sets from SemEval 2012-2015, supporting the correctness of our Bayesian implementation.

Following Mohler et al. (2011), for sas we use rmse and Pearson's \(r\) with gold scores over all answers. These metrics are complementary: correlation is a measure of consistency across students while error measures deviation from individual scores. Our model beats the state-of-the-art text matching model of Mohler et al. (2011) on both metrics.7

Footnote 7: Ramachandran et al. (2015) report better results; however, they evaluate on a much smaller random subset of the test data and use in-domain annotations for model training.

Finally, for ast, we adopt two metrics widely used in information retrieval: mean average precision (map) and mean reciprocal rank (mrr). map assesses the quality of the ranking as a whole whereas mrr evaluates only the top-ranked answer sentence. Severyn and Moschitti (2015) report a convolutional neural network model of text similarity which shows top asr results on the Wang et al. (2007) dataset. Our model outperforms this model on both metrics.

### Adaptation to sts Domains

Ideally, our domain adaptation (da) should allow the application of large amounts of out-of-domain training data along with few in-domain examples to improve in-domain performance. Given data from \(n\) domains, two other alternatives in such scenarios are: (1) to train a single _global_ model using all available training examples, and (2) to train \(n\)_individual_ models, one for each domain, using only in-domain examples. We present results from our da model and these two baselines on the ten sts datasets discussed in Section 2. We fix the training set size per domain and split each domain into train and test folds randomly.

Models have access to training data from all ten domains (thus nine times more out-of-domain examples than in-domain ones). Each model (global, individual, and adaptive) is trained on relevant annotations and applied to test pairs, and Pearson's \(r\) with gold scores is computed for each model on each individual test set. Since performance can vary across different splits, we average over 20 splits of the same train/test ratio per dataset. Finally, we evaluate each model with a weighted sum of average correlations across all test sets, where the weight of a test set is proportional to its number of pairs.

Figure 5 shows how models adapt as the training set grows. The global model clearly falters with larger training sets in comparison to the other two models. On the other hand, the domain-specific model (i.e., the ten individual models) performs poorly when in-domain annotations are scarce. Importantly, the adaptive model performs well across different amounts of available training data.

To gain a deeper understanding of model performance, we examine results in individual domains. A single performance score is computed for every model-domain pair by taking the model's average

\begin{table}
\begin{tabular}{l|c|c} \hline \hline
**Task** & **Current SOA** & **Our Model** \\ \hline \hline sts & Pearson’s \(r=\) 73.6\% & Pearson’s \(r=\) 73.7\% \\ \cline{2-4} sa & Pearson’s \(r=\) 51.8\% & Pearson’s \(r=\) 56.4\% \\  & rmse = 19.6\% & rmse = 18.1\% \\ \hline asr & Map = 74.6\% & map = 76.0\% \\  & mrr = 80.8\% & mrr = 82.8\% \\ \hline \hline \end{tabular}
\end{table}
Table 1: Our base linear models beat the state of the art in sts, sas and asr.

 

correlation in that domain over all seven training set sizes of Figure 5. We then normalize each score by dividing by the best score in that domain. Each cell in Table 2 shows this score for a model-domain pair. For example, Row 1 shows that--on average--the individual model performs the best (hence a correlation ratio of 1.0) on qa forum answer pairs while the global model performs the worst.

While the adaptive model is not the best in every domain, it has the best worst-case performance across domains. The global model suffers in domains that have unique parameter distributions (e.g., msrparser: a paraphrase dataset). The individual model performs poorly with few training examples and in domains with noisy annotations (e.g., smt: a machine translation evaluation dataset). The adaptive model is much less affected in such extreme cases. The summary statistics (weighted by dataset size) confirm that it not only stays the closest to the best model on average, but also deviates the least from its mean performance level.

#### 5.1.1 Qualitative Analysis

We further examine the models to understand _why_ the adaptive model performs well in different extreme scenarios, i.e., when one of the two baseline models performs worse than the other. Table 3 shows feature weights learned by each model from a split with seventy-five training pairs per domain and how well each model does.

All three domains have very different outcomes for the baseline models. We show weights for the alignment (\(w_{1}\)) and embedding features (\(w_{2}\)). In each domain, (1) the relative weights learned by the two baseline models are very different, and (2) the adaptive model learns relative weights that are closer to those of the best model. In smt, for example, the predictor weights learned by the adaptive model have a ratio very similar to the global model's and does just as well. On Answers-students, however, it learns weights similar to those of the in-domain model, again approaching best results for the domain.

Figure 5: Results of adaptation to sts domains across different amounts of training data. Table shows mean\(\pm\)SD from 20 random train/test splits. While the baselines falter at extremes, the adaptive model shows consistent performance.

 

Table 4 shows the effect of this on two specific sentence pairs as examples. The first pair is from smt; the adaptive model has a much lower error than the individual model on this pair, as it learns a higher relative weight for the embedding feature in this domain (Table 3) via inductive transfer from out-of-domain annotations. The second pair, from msrpar-test, shows the opposite: in-domain annotations help the adaptive model fix the faulty output of the global model by upweighting the alignment feature and downweighting the embedding feature.

The adaptive model gains from the strengths of both in-domain (higher relevance) and out-of-domain (more training data) annotations, leading to good results even in extreme scenarios (e.g., in domains with unique parameter distributions or noisy annotations).

### Multitask Learning

We now analyze performance of our multitask learning (mtl) model in each of the three tasks: sts, sas and asr. Multitask baselines resemble da's: (1) a global model trained on all available training data (Figure 4), and (2) nineteen task-specific models, each trained on an individual dataset from one of the three tasks (Figure 1). The smallest of these datasets has only 204 pairs (ss assignment #1); therefore, we use training sets with up to 175 pairs per dataset. Because the mtl model is more complex, we use a stronger regularization for this model (\(m_{\sigma_{w}}\)=10) while keeping the number of mcmc samples unchanged. As in the da experiments, we compute average performance over twenty random train/test splits for each training set size.

Figure 6 showssts results for all models across different training set sizes. Like da, the adaptive model consistently performs well while the global and individual models have different failure modes. However, the individual model does better than in da: it overtakes the global model with fewer training examples and the differences with the adaptive model are smaller. This suggests that inductive transfer and therefore adaptation is less effective for sts in the mtl setup than in da. Later in this section, coarse-grained asr annotations (binary as opposed to real-valued) in mtl may provide an explanation for this.

The performance drop after 150 training pairs is a likely consequence of the random train/test selection process.

For ss, the adaptive model again has the best overall performance for both correlation and error (Figure 7). The correlation plot is qualitatively similar to the sts plot, but the global model has a much higher rmse across all training set sizes, indicating a parameter shift across tasks. Importantly, the adaptive model remains unaffected by this shift.

The asr results in Figure 8 show a different pattern. Contrary to all results thus far, the global model performs the best in this task. The individual model consistently has lower scores, regardless of the amount of training data. Importantly, the adaptive model stays close to the global model even with very few training examples. The asr datasets are heavily biased towards negative examples; thus, we

\begin{table}
\begin{tabular}{p{99.6pt}|p{113.8pt}} \hline \hline Now, the labor of cleaning up at the karaoke parlor is realized. & Gold=.52 \\ \cline{2-3} Up till now on the location the cleaning work is already completed. & \(\Delta\)G=**.1943** \\ \hline The Chelsea defender Marcel Desailly has been the latest to speak out. & \(\Delta\)A=.2024 \\ \hline Marcel Desailly, the France captain and Chelsea defender, believes the latter is true. & Gold=.45 \\ \hline \hline \end{tabular} 
\begin{tabular}{p{99.6pt} p{113.8pt} p{113.8pt} p{113.8pt} p{113.8pt} p{113.8pt} p{113.8pt}} \multirow{2}{*}{global} & \multicolumn{2}{c}{\(\underline{\text{d}}\)G=.2513} \\  & \multicolumn{2}{c}{\(\Delta\)G=**.2222**} \\  & \multicolumn{2}{c}{\(\Delta\)H=**.2245**} \\ \hline \hline \end{tabular}
\end{table}
Table 4: Sentence pairs from smt and msrpar-test with gold similarity scores and model errors (Global, Individual and Adaptive). The adaptive model error is very close to the best model error in each case.

Figure 6: Multitask learning for sts: mean\(\pm\)sd from twenty random train/test splits. The adaptive model consistently performs well while the baselines have different failure modes.

 

* [25] M. C. Bader, A. K. He, and J. S. Kim, "A new approach to the classification of neural networks for large-scale image recognition," in _IEEE transactions on pattern analysis and machine intelligence_, vol. 109(3): IEEE, 2014, pp. 768-772.
* [26] D. R. Kingma and P. Ba, "Adam: a method for stochastic optimization," in _Proceedings of the twenty-second international conference on Machine learning_, volume 13, pages 131-140. PMLR, 2014.
* [27] L. Li, X. Wang, Z. Zhang, and Y. Sun, "Deep residual learning with deep convolutional neural networks," in _NIPS Conference Proceedings_, volume 13, pages 1318-1320. PMLR, 2014.
* [28] H. Liu, W. Chen, and Q. Yu, "Deep residual learning with deep convolutional neural networks," in _CVPR International Conference on Computer Vision and Pattern Recognition_, volume 13, pages 1318-1320. PMLR, 2014.
* [29] G. Lu, T. Lin, and J. S. Lee, "Learning multiple layers by jointly optimizing the gradient descent algorithm," _arXiv preprint arXiv:1512.01558_, 2015.
* [30] E. Maaten, N. Hinton, and I. Sutskever, "Batch normalization: Accelerating deep network training using deep networks," in _Advances in Neural Information Processing Systems_, volume 1, pages 109-116, 2016.

[MISSING_PAGE_POST]

 

use stratified sampling to ensure each asr training set has balanced examples.

A reason for the global model's strength at asr may lie in the finer granularity of the real-valued sts and sas scores compared to binary asr annotations. If a fine granularity is indeed desirable in training data, as a model that ignores in-domain and out-of-domain distinction, the global model would be affected the least by coarse-grained an annotations. To test this hypothesis, we train a linear model on all sts examples from SemEval 2012-2015 and apply it to the ast test set via a logistic transformation. This model indeed demonstrates better results (map=.766, mrr=.839) than our base model trained on asr annotations (Table 1). This is an unusual scenario where in-domain training examples matter less than out-of-domain ones, hurting domain-specific and adaptive models.

Going back to sts, this finding also offers an explanation of why adaptation might have been less useful in multitask learning than in domain adaptation, as only the former has astr annotations.

## 6 Discussion and Related Work

For a variety of short text similarity tasks, domain adaptation improves average performance across different domains, tasks, and training set sizes. Our adaptive model is also by far the least affected by adverse factors such as noisy training data and scarcity or coarse granularity of in-domain examples. This combination of excellent average-case and very reliable worst-case performance makes it the model of choice for new sts domains and applications.

Although sts is a useful task with sparse data, few domain adaptation studies have been reported. Among those is the supervised model of Heilman and Madnani (2013a; 2013b) based on the multilevel model of Daume III (2007). Gella et al. (2013) report using a two-level stacked regressor, where the second level combines predictions from \(n\) level 1 models, each trained on data from a separate domain. Unsupervised models use techniques such as tagging examples with their source datasets [4, 1] and computing vocabulary similarity between source and target domains [1]. To the best of our knowledge, ours is the first systematic study of supervised da and mtl techniques for sts with detailed comparisons with comparable non-adaptive baselines.

## 7 Conclusions and Future Work

We present hierarchical Bayesian models for supervised domain adaptation and multitask learning of short text similarity models. In our experiments, these models show improved overall performance across different domains and tasks. We intend to explore adaptation to other sts applications and with additional sts features (e.g., word and character \(n\)-gram overlap) in future. Unsupervised and semi-supervised domain adaptation techniques that do not assume the availability of in-domain annotations or that learn effective domains splits [10] provide another avenue for future research.

## Acknowledgments

This material is based in part upon work supported by the NSF under grants ehr/0835393 and ehr/0835381. Boyd-Graber is supported by NSF grants iis/1320538, iiis/1409287, and nscse/1422492. Any opinions, findings, conclusions, or recommendations expressed here are those of the authors and do not necessarily reflect the view of the sponsor.

## References

* [Agirre et al.2012] Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. 2012. SemEval-2012 task 6: A pilot on semantic textual similarity. In _SemEval_.
* [Agirre et al.2013] Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, and Weiwei Guo. 2013. "SEM 2013 shared task: Semantic textual similarity. In *_SEM_.
* [Agirre et al.2014] Eneko Agirre, Carmen Banea, Claire Cardiie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2014. SemEval-2014 Task 10: Multilingual semantic textual similarity. In _SemEval_.
* [Agirre et al.2015] Eneko Agirre, Carmen Banea, Claire Cardiie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Iiigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, German Rigau, Larraitz Uria, and Janyce Wiebe. 2015. SemEval-2015 Task 2: Semantic textual similarity, english, spanish and pilot on interpretability. In _SemEval_.
* [Arora et al.2015] Piyush Arora, Chris Hokamp, Jennifer Foster, and Gareth J.FJones. 2015. DCU: Using distributional semantics and domain adaptation for the semantic textual similarity SemEval-2015 Task 2. In _SemEval_ 

Daniel Bar, Chris Biemann, Iryna Gurevych, and Torsten Zesch. 2012. UKP: Computing semantic textual similarity by combining multiple content similarity measures. In _SemEval_.
* Baroni et al. (2014) Marco Baroni, Georgiana Dinu, and German Kruszewski. 2014. Don't count and predict! A systematic comparison of context-counting vs. context-predicting semantic vectors. In _Proceedings of the Association for Computational Linguistics_.
* Brockett (2007) Chris Brockett. 2007. Aligning the RTE 2006 corpus. Technical Report MSR-TR-2007-77, Microsoft Research.
* Chan and Ng (2008) Yee Seng Chan and Hwee Tou Ng. 2008. MAXSIM: A maximum similarity metric for machine translation evaluation. In _Proceedings of the Association for Computational Linguistics_.
* Dasgupta et al. (2013) Anirban Dasgupta, Ravi Kumar, and Sujith Ravi. 2013. Summarization through submodularity and dispersion. In _Proceedings of the Association for Computational Linguistics_.
* Daume III (2007) Hal Daume III. 2007. Frustratingly easy domain adaptation. In _Proceedings of the Association for Computational Linguistics_.
* Finkel and Manning (2009) Jenny Rose Finkel and Christopher D. Manning. 2009. Hierarchical bayesian domain adaptation. In _Conference of the North American Chapter of the Association for Computational Linguistics_, Morristown, NJ, USA.
* Ganitkevitch et al. (2013) Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. PPDB: The paraphrase database. In _Conference of the North American Chapter of the Association for Computational Linguistics_.
* Gella et al. (2013) Spandana Gella, Bahar Salehi, Marco Lui, Karl Grieser, Paul Cook, and Timothy Baldwin. 2013. UniMelb_NLP-CORE: Integrating predictions from multiple domains and feature sets for estimating semantic textual similarity. In *SEM.
* Hanig et al. (2015) Christian Hanig, Robert Remus, and Xose de la Puente. 2015. ExB Themis: Extensive feature extraction from word alignments for semantic textual similarity. In _SemEval_.
* Heilman and Madnani (2013a) Michael Heilman and Nitin Madnani. 2013a. ETS: Domain adaptation and stacking for short answer scoring. In _SemEval_.
* Heilman and Madnani (2013b) Michael Heilman and Nitin Madnani. 2013b. HENRYCORE: Domain adaptation and stacking for text similarity. In _SemEval_.
* Hu et al. (2014) Yuening Hu, Ke Zhai, Vlad Eidelman, and Jordan Boyd-Graber. 2014. Polylingual tree-based topic models for translation domain adaptation. In _Association for Computational Linguistics_.
* Liu et al. (2011) Chang Liu, Daniel Dahlmeier, and Hwee Tou Ng. 2011. Better evaluation metrics lead to better machine translation. In _Proceedings of Empirical Methods in Natural Language Processing_.
* Lynum et al. (2014) Andre Lynum, Partha Pakray, Bjorn Gamback, and Sergio Jimenez. 2014. NTNU: Measuring semantic similarity with sublexical feature representations and soft cardinality. In _SemEval_.
* Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In _Proceedings of the International Conference on Learning Representations Workshop_.
* Mohler et al. (2011) Michael Mohler, Razvan Bunescu, and Rada Mihalcea. 2011. Learning to grade short answer questions using semantic similarity measures and dependency graph alignments. In _Proceedings of the Association for Computational Linguistics_.
* Patil et al. (2010) Anand Patil, David Huard, and Christopher J. Fonnesbeck. 2010. PyMC: Bayesian stochastic modelling in python. _Journal of Statistical Software_, 35(4).
* Ramachandran et al. (2015) Lakshmi Ramachandran, Jian Cheng, and Peter Foltz. 2015. Identifying patterns for short answer scoring using graph-based lexico-semantic text matching. In _NAACL-BEA_.
* Salvatier et al. (2015) John Salvatier, Thomas V. Wiecki, and Christopher Fonnesbeck. 2015. Probabilistic programming in python using PyMC. _arXiv:1507.08050v1_.
* Severyn and Moschitti (2015) Aliaksei Severyn and Alessandro Moschitti. 2015. Learning to rank short text pairs with convolutional deep neural networks. In _Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval_.
* Severyn et al. (2013) Aliaksei Severyn, Massimo Nicosia, and Alessandro Moschitti. 2013. Learning semantic textual similarity with structural representations. In _Proceedings of the Association for Computational Linguistics_.
* Sultan et al. (2014) Md Arafat Sultan, Steven Bethard, and Tamara Sumner. 2014. Back to basics for monolingual alignment: Exploiting word similarity and contextual evidence. _TACL_, 2.
* Sultan et al. (2015) Md Arafat Sultan, Steven Bethard, and Tamara Sumner. 2015. DLS@CU: Sentence similarity from word alignment and semantic vector composition. In _SemEval_.
* Wang et al. (2007) Mengqiu Wang, Noah A. Smith, and Teruko Mitamura. 2007. What is the Jeopardy model? a quasi-synchronous grammar for QA. In _Proceedings of Empirical Methods in Natural Language Processing_.
* Wang et al. (2013) Lu Wang, Herna Raghavan, Vittorio Castelli, Radu Florian, and Claire Cardiie. 2013. A sentence compression based framework to query-focused multi-document summarization. In _Proceedings of the Association for Computational Linguistics_.
* Yao et al. (2013) Xuchen Yao, Benjamin Van Durme, Chris Callison-Burch, and Peter Clark. 2013. Answer extraction as sequence tagging with tree edit distance. In _Conference of the North American Chapter of the Association for Computational Linguistics_