Introduction
The specificity of a sentence measures its “quality of belonging or relating uniquely to a particular subject” BIBREF0 . It is often pragmatically defined as the level of detail in the sentence BIBREF1 , BIBREF2 . When communicating, specificity is adjusted to serve the intentions of the writer or speaker BIBREF3 . In the examples below, the second sentence is clearly more specific than the first one:
Ex1: This brand is very popular and many people use its products regularly.
Ex2: Mascara is the most commonly worn cosmetic, and women will spend an average of $4,000 on it in their lifetimes.
Studies have demonstrated the important role of sentence specificity in reading comprehension BIBREF4 and in establishing common ground in dialog BIBREF5 . It has also been shown to be a key property in analyses and applications, such as summarization BIBREF6 , argumentation mining BIBREF7 , political discourse analysis BIBREF8 , student discussion assessment BIBREF9 , BIBREF0 , deception detection BIBREF10 , and dialogue generation BIBREF11 .
Despite their usefulness, prior sentence specificity predictors BIBREF1 , BIBREF2 , BIBREF0 are trained with sentences from specific domains (news or classroom discussions), and have been found to fall short when applied to other domains BIBREF10 , BIBREF0 . They are also trained to label a sentence as either general or specific BIBREF1 , BIBREF2 , or low/medium/high specificity BIBREF0 , though in practice specificity has been analyzed as a continuous value BIBREF6 , BIBREF12 , BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , as it should be BIBREF13 .
In this work, we present an unsupervised domain adaptation system for sentence specificity prediction, specifically designed to output real-valued estimates. It effectively generalizes sentence specificity analysis to domains where no labeled data is available, and outputs values that are close to the real-world distribution of sentence specificity.
Our main framework is an unsupervised domain adaptation system based on Self-Ensembling BIBREF14 , BIBREF15 that simultaneously reduces source prediction errors and generates feature representations that are robust against noise and across domains. Past applications of this technique have focused on computer vision problems; to make it effective for text processing tasks, we modify the network to better utilize labeled data in the source domain and explore several data augmentation methods for text.
We further propose a posterior regularization technique BIBREF16 that generally applies to the scenario where it is easy to get coarse-grained categories of labels, but fine-grained predictions are needed. Specifically, our regularization term seeks to move the distribution of the classifier posterior probabilities closer to that of a pre-specified target distribution, which in our case is a specificity distribution derived from the source domain.
Experimental results show that our system generates more accurate real-valued sentence specificity predictions that correlate well with human judgment, across three domains that are vastly different from the source domain (news): Twitter, Yelp reviews and movie reviews. Compared to a state-of-the-art system trained on news data BIBREF2 , our best setting achieves a 50%-68% reduction in mean absolute error and increases Kendall's Tau and Spearman correlations by 0.07-0.10 and 0.12-0.13, respectively.
Finally, we conduct a task-based evaluation that demonstrates the usefulness of sentence specificity prediction in open-domain dialogue generation. Prior work showed that the quality of responses from dialog generation systems can be significantly improved if short examples are removed from training BIBREF17 , potentially preventing the system from overly favoring generic responses BIBREF18 , BIBREF19 . We show that predicted specificity works more effectively than length, and enables the system to generate more diverse and informative responses with better quality.
In sum, the paper's contributions are as follows:
Task setup
With unsupervised domain adaptation, one has access to labeled sentence specificity in one source domain, and unlabeled sentences in all target domains. The goal is to predict the specificity of target domain data. Our source domain is news, the only domain with publicly available labeled data for training BIBREF1 . We crowdsource sentence specificity for evaluation for three target domains: Twitter, Yelp reviews and movie reviews. The data is described in Section SECREF4 .
Existing sentence specificity labels in news are binary, i.e., a sentence is either general or specific. However, in practice, real-valued estimates of sentence specificity are widely adopted BIBREF6 , BIBREF12 , BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 . Most of these work directly uses the classifier posterior distributions, although we will later show that such distributions do not follow the true distribution of sentence specificity (see Figure FIGREF29 , Speciteller vs. real). We aim to produce accurate real-valued specificity estimates despite the binary training data. Specifically, the test sentences have real-valued labels between 0 and 1. We evaluate our system using mean absolute error, Kendall's Tau and Spearman correlation.
Architecture
Our core technique is the Self-Ensembling BIBREF14 , BIBREF15 of a single base classification model that utilizes data augmentation, with a distribution regularization term to generate accurate real-valued predictions from binary training labels.
Base model
Figure FIGREF5 depicts the base model for sentence specificity prediction. The overall structure is similar to that in BIBREF0 . Each word in the sentence is encoded into an embedding vector and the sentence is passed through a bidirectional Long-Short Term Memory Network (LSTM) BIBREF20 to generate a representation of the sentence. This representation is concatenated with a series of hand crafted features, and then passed through a multilayer perceptron to generate specificity predictions INLINEFORM0 . Training treats these as probabilities of the positive class, but we will show in Section SECREF13 how they can be adapted to make real-valued predictions.
Our hand crafted features are taken from BIBREF2 , including: the number of tokens in the sentence; the number of numbers, capital letters and punctuations in the sentence, normalized by sentence length; the average number of characters in each word; the fraction of stop words; the number of words that can serve as explicit discourse connectives; the fraction of words that have sentiment polarity or are strongly subjective; the average familiarity and imageability of words; and the minimum, maximum, and average inverse document frequency (idf) over the words in the sentence. In preliminary experiments, we found that combining hand-crafted features with the BiLSTM performed better than using either one individually.
Unsupervised domain adaptation
Our unsupervised domain adaptation framework is based on Self-Ensembling BIBREF15 ; we lay out the core ideas first and then discuss modifications to make the framework suitable for textual data.
There are two ideas acting as the driving force behind Self-Ensembling. First, adding noise to each data point INLINEFORM0 can help regularize the model by encouraging the model prediction to stay the same regardless of the noise, creating a manifold around the data point where predictions are invariant BIBREF22 . This can be achieved by minimizing a consistency loss INLINEFORM1 between the predictions. Second, temporal ensembling—ensemble the same model trained at different time steps—is shown to be beneficial to prediction especially in semi-supervised cases BIBREF23 ; in particular, we average model parameters from each time step BIBREF14 .
These two ideas are realized by using a student network and a teacher network. The parameters of the teacher network are the exponential average of that of the student network, making the teacher a temporal ensemble of the student. Distinct noise augmentations are applied to the input of each network, hence the consistency loss INLINEFORM0 is applied between student and teacher predictions. The student learns from labeled source data and minimizes the supervised cross-entropy loss INLINEFORM1 . Domain adaptation is achieved by minimizing the consistency loss between the two networks, which can be done with unlabeled target data. The overall loss function is the weighted sum of INLINEFORM2 and INLINEFORM3 . Figure FIGREF6 depicts the process.
Concretely, the student and teacher networks are of identical structure following the base model (Section SECREF7 ), but features distinct noise augmentation. The student network learns to predict sentence specificity from labeled source domain data. The input sentences are augmented with noise INLINEFORM0 . The teacher network predicts the specificity of each sentence with a different noise augmentation INLINEFORM1 . Parameters of the teacher network INLINEFORM2 are updated each time step to be the exponential moving average of the corresponding parameters in the student network. The teacher parameter INLINEFORM3 at each time step INLINEFORM4 is DISPLAYFORM0
where INLINEFORM0 is the degree of weighting decay, a constant between 0 and 1. INLINEFORM1 denotes parameters for the student network.
The consistency loss is defined as the squared difference between the predictions of the student and teacher networks BIBREF14 : DISPLAYFORM0
where INLINEFORM0 denotes the base network and x denotes the input sentence. The teacher network is not involved when minimizing the supervised loss INLINEFORM1 .
An important difference between our work and BIBREF15 is that in their work only unlabeled target data contributes to the consistency loss INLINEFORM0 . Instead, we use both source and target sentences, bringing the predictions of the two networks close to each other not only on the target domain, but also the source domain. Unlike many vision tasks where predictions intuitively stay the same with different types of image augmentation (e.g., transformation and scaling), text is more sensitive to noise. Self-Ensembling relies on heavy augmentation on both student and teacher networks, and our experiments revealed that incorporating source data in the consistency loss term mitigates additional biases from noise augmentation.
At training time, the teacher network's parameters are fixed during gradient descent, and the gradient only propagates through the student network. After each update of the student network, we recalculate the weights of the teacher network using exponential moving average. At testing time, we use the teacher network for prediction.
An important factor contributing to the effectiveness of Self-Ensembling is applying noise to the inputs to make them more robust against domain shifts. For computer vision tasks, augmentation techniques including affine transformation, scaling, flipping and cropping could be used BIBREF15 . However these operations could not be used on text. We designed several noise augmentations for sentences, including: adding Gaussian noise to both the word embeddings and shallow features; randomly removing words in a sentence; substituting word embeddings with a random vector, or a zero vector like applying dropout. To produce enough variations of data, augmentation is applied to INLINEFORM0 half of the words in a sentence.
Regularizing the posterior distribution
Sentence specificity prediction is a task where the existing training data have binary labels, while real-valued outputs are desirable. Prior work has directly used classifier posterior probabilities. However, the posterior distribution and the true specificity distribution are quite different (see Figure FIGREF29 , Speciteller vs. real). We propose a regularization term to bridge the gap between the two.
Specifically, we view the posterior probability distribution as a latent distribution, which allows us to apply a variant of posterior regularization BIBREF16 , previously used to apply pre-specified constraints to latent variables in structured prediction. Here, we apply a distance penalty between the latent distribution and a pre-specified reference distribution (which, in our work, is from the source domain). BIBREF13 found that in news, the distribution of sentence specificity is bell shaped, similar to that of a Gaussian. Our analysis on sentence specificity for three target domains yields the same insights (Figure FIGREF18 ). We explored two regularization formulations, with and without assuming that the two distributions are Gaussian. Both were successful and achieved similar performance.
Let INLINEFORM0 and INLINEFORM1 be the mean and standard deviation of the predictions (posterior probabilities) in a batch. The first formulation assumes that the predictions and reference distributions are Gaussian. It uses the KL divergence between the predicted distribution INLINEFORM2 and the reference Gaussian distribution INLINEFORM3 . The distribution regularization loss can be written as: DISPLAYFORM0
The second formulation does not assume Gaussian distributions and only compares the mean and standard deviation of the two distributions using a weighting term INLINEFORM0 : DISPLAYFORM0
Combining the regularization term INLINEFORM0 into a single objective, the total loss is thus: DISPLAYFORM0
where INLINEFORM0 is the cross entropy loss for the source domain predictions, INLINEFORM1 is the consistency loss. INLINEFORM2 and INLINEFORM3 are weighting hyperparameters.
In practice, this regularization term serves a second purpose. After adding the consistency loss INLINEFORM0 , we observed that the predictions are mostly close to each other with values between 0.4 and 0.6, and their distribution resembles a Gaussian with very small variance (c.f. Figure FIGREF29 line SE+A). This might be due to the consistency loss pulling all the predictions together, since when all predictions are identical, the loss term will be zero. This regularization can be used to counter this effect, and avoid the condensation of predicted values. Finally, this regularization is distinct from class imbalance loss terms such as that used in BIBREF15 , which we found early on to hurt performance rather than helping.
Source domain
The source domain for sentence specificity is news, for which we use three publicly available labeled datasets: (1) training sentences from BIBREF1 and BIBREF2 , which consists of 1.4K general and 1.4K specific sentences from the Wall Street Journal. (2) 900 news sentences crowdsourced for binary general/specific labels BIBREF24 ; 55% of them are specific. (3) 543 news sentences from BIBREF13 . These sentences are rated on a scale of INLINEFORM0 , so for consistency with the rest of the training labels, we pick sentences with average rating INLINEFORM1 as general examples and those with average rating INLINEFORM2 as specific. In total, we have 4.3K sentences with binary labels in the source domain.
Target domains
We evaluate on three target domains: Twitter, Yelp and movie reviews. Since no annotated data exist for these domains, we crowdsource specificity for sentences sampled from each domain using Amazon Mechanical Turk.
We follow the context-independent annotation instructions from BIBREF13 . Initially, 9 workers labeled specificity for 1000 sentences in each domain on a scale of 1 (very general) to 5 (very specific), which we rescaled to 0, 0.25, 0.5, 0.75, and 1. Inter-annotator agreement (IAA) is calculated using average Cronbach's alpha BIBREF25 values for each worker. For quality control, we exclude workers with IAA below 0.3, and include the remaining sentences that have at least 5 raters. Our final IAA values fall between 0.68-0.70, compatible with the reported 0.72 from expert annotators in BIBREF13 . The final specificity value is aggregated to be the average of the rescaled ratings. We also use large sets of unlabeled data in each domain:
Twitter: 984 tweets annotated, 50K unlabeled, sampled from BIBREF26 .
Yelp: 845 sentences annotated, 95K unlabeled, sampled from the Yelp Dataset Challenge 2015 BIBREF27 .
Movie: 920 sentences annotated, 12K unlabeled, sampled from BIBREF28 .
Figure FIGREF18 shows the distribution of ratings for the annotated data. We also plot a fitted Gaussian distribution for comparison. Clearly, most sentences have mid-range specificity values, consistent with news sentences BIBREF13 . Interestingly the mean and variance for the three distributions are similar to each other and to those from BIBREF13 , as show in Table TABREF23 . Therefore, we use the source distribution (news, BIBREF13 ) as the reference distribution for posterior distribution regularization, and set INLINEFORM0 to 0.417 and 0.227 accordingly.
Experiments
We now evaluate our framework on predicting sentence specificity for the three target domains. We report experimental results on a series of settings to evaluate the performance of different components.
Systems
Length baseline This simple baseline predicts specificity proportionally to the number of words in the sentence. Shorter sentences are predicted as more general and longer sentences are predicted as more specific.
Speciteller baseline Speciteller BIBREF2 is a semi-supervised system trained on news data with binary labels. The posterior probabilities of the classifier are used directly as specificity values.
Self-ensembling baseline (SE) Our system with the teacher network, but only using exponential moving average (without the consistency loss or distribution regularization).
Distribution only (SE+D) Our system with distribution regularization INLINEFORM0 using the mean and standard deviation (Eq. EQREF15 ), but without the consistency loss INLINEFORM1 .
Adaptation only (SE+A) Our system with the consistency loss INLINEFORM0 , but without distribution regularization INLINEFORM1 .
SE+AD (KL) Our system with both INLINEFORM0 and INLINEFORM1 using KL divergence (Eq. EQREF14 ).
SE+AD (mean-std) Our system with both INLINEFORM0 and INLINEFORM1 using mean and standard deviation (Eq. EQREF15 ).
SE+AD (no augmentation) We also show the importance of noise augmentation, by benchmarking the same setup as SE+AD (mean-std) without data augmentation.
Training details
Hyperparameters are tuned on a validation set of 200 tweets that doesn't overlap with the test set. We then use this set of parameters for all testing domains. The LSTM encoder generates 100-dimensional representations. For the multilayer perceptron, we use 3 fully connected 100-dimensional layers. We use ReLU activation with batch normalization. For the Gaussian noise in data augmentation, we use standard deviation 0.1 for word embeddings and 0.2 for shallow features. The probabilities of deleting a word and replacing a word vector are 0.15. The exponential moving average decay INLINEFORM0 is 0.999. Dropout rate is 0.5 for all layers. The batch size is 32. INLINEFORM1 , INLINEFORM2 for KL loss and 100 for mean and std.dev loss. INLINEFORM3 . We fix the number of training to be 30 epochs for SE+A and SE+AD, 10 epochs for SE, and 15 epochs for SE+D. We use the Adam optimizer with learning rate 0.0001, INLINEFORM4 , INLINEFORM5 . As discussed in Section SECREF4 , posterior distribution regularization parameters INLINEFORM6 are set to be those from BIBREF13 .
Evaluation metrics
We use 3 metrics to evaluate real-valued predictions: (1) the Spearman correlation between the labeled and predicted specificity values, higher is better; (2) the pairwise Kendall's Tau correlation, higher is better; (3) mean absolute error (MAE): INLINEFORM0 , lower is better.
Results and analysis
Table TABREF25 shows the full results for the baselines and each configuration of our framework. For analysis, we also plot in Figure FIGREF29 the true specificity distributions in Twitter test set, predicted distributions for Speciteller, the Self-Ensembling baseline (SE), SE with adaptation (SE+A) and also with distribution regularization (SE+AD).
Speciteller, which is trained on news sentences, cannot generalize well to other domains, as it performs worse than just using sentence length for two of the three domains (Yelp and Movie). From Figure FIGREF29 , we can see that the prediction mass of Speciteller is near the extrema values 0 and 1, and the rest of the predictions falls uniformly in between. These findings confirm the need of a generalizable system.
Across all domains and all metrics, the best performing system is our full system with domain adaptation and distribution regularization (SE+AD with mean-std or KL), showing that the system generalizes well across different domains. Using a paired Wilcoxon test, it significantly ( INLINEFORM0 ) outperforms Speciteller in terms of MAE; it also achieved higher Spearman and Kendall's Tau correlations than both Length and Speciteller.
Component-wise, the Self-Ensembling baseline (SE) achieves significantly lower MAE than Speciteller, and higher correlations than either baseline. Figure FIGREF29 shows that unlike Speciteller, the SE baseline does not have most of its prediction mass near 0 and 1, demonstrating the effectiveness of temporal ensembling. Using both the consistency loss INLINEFORM0 and the distribution regularization INLINEFORM1 achieves the best results on all three domains; however adding only INLINEFORM2 (SE+A) or INLINEFORM3 (SE+D) improves for some measures or domains but not all. This shows that both terms are crucial to make the system robust across domains.
The improvements from distribution regularization are visualized in Figure FIGREF29 . With SE+A, most of the predicted labels are between 0.4 and 0.6. Applying distribution regularization (SE+AD) makes them much closer to the real distribution of specificity. With respect to the two formulations of regularization (KL and mean-std), both are effective in generating more accurate real-valued estimates. Their performances are comparable, hence using only the mean and standard deviation values, without explicitly modeling the reference Gaussian distribution, works equally well.
Finally, without data augmentation (column no aug.), the correlations are clearly lower than our full model, stressing the importance of data augmentation in our framework.
Specificity in dialogue generation
We also evaluate our framework in open-domain dialogue. This experiment also presents a case for the usefulness of an effective sentence specificity system in dialogue generation.
With seq2seq dialogue generation models, BIBREF17 observed significant quality improvement by removing training examples with short responses during preprocessing; this is potentially related to this type of models favor generating non-informative, generic responses BIBREF18 , BIBREF19 . We show that filtering training data by predicted specificity results in responses of higher quality and informativeness than filtering by length.
Task and settings
We implemented a seq2seq question-answering bot with attention using OpenNMT BIBREF29 . The bot is trained on OpenSubtitles BIBREF30 following prior work BIBREF17 . We restrict the instances to question-answer pairs by selecting consecutive sentences with the first sentence ending with question mark, the second sentence without question mark and follows the first sentence by less than 20 seconds, resulting in a 14M subcorpus.
The model uses two hidden layers of size 2048, optimized with Adam with learning rate 0.001. The batch size is 64. While decoding, we use beam size 5; we block repeating n-grams, and constrain the minimum prediction length to be 5. These parameters are tuned on a development set. We compare two ways to filter training data during preprocessing:
Remove Short: Following BIBREF17 , remove training examples with length of responses shorter than a threshold of 5. About half of the data are removed.
Remove General: Remove predicted general responses from training examples using our system. We use the responses in the OpenSubtitles training set as the unlabeled target domain data during training. We remove the least specific responses such that the resulting number of examples is the same as Remove Short.
For fair comparison, at test time, we adjust the length penalty described in BIBREF31 for both models, so the average response lengths are the same among both models.
Evaluation
We use automatic measures and human evaluation as in BIBREF32 and BIBREF17 .
Table TABREF31 shows the diversity and perplexity of the responses. Diversity is calculated as the type-token ratio of unigrams and bigrams. The test set for these two metrics is a random sample of 10K instances from OpenSubtitles that doesn't overlap with the training set. Clearly, filtering training data according to specificity results in more diverse responses with lower perplexity than filtering by length.
We also crowdsource human evaluation for quality; in addition, we evaluate the systems for response informativeness. Note that in our instructions, informativeness means the usefulness of information and is a distinct measure from specificity. The original training data of specificity are linguistically annotated and involves only the change in the level of details BIBREF1 . Separate experiments are conducted to avoid priming. We use a test set of 388 instances, including questions randomly sampled from OpenSubtitles that doesn't overlap with the training set, and 188 example questions from previous dialogue generation papers, including BIBREF33 . We use Amazon MechenicalTurk for crowdsourcing. 7 workers chose between the two responses to the same question.
Table TABREF32 shows the human evaluation comparing Remove Short vs. Remove General. Removing predicted general responses performs better than removing short sentences, on both informativeness and quality and on both test sets. This shows that sentence specificity is a superior measure for training data preprocessing than sentence length.
Related Work
Sentence specificity prediction as a task is proposed by BIBREF1 , who repurposed discourse relation annotations from WSJ articles BIBREF34 for sentence specificity training. BIBREF2 incorporated more news sentences as unlabeled data. BIBREF0 developed a system to predict sentence specificity for classroom discussions, however the data is not publicly available. All these systems are classifiers trained with categorical data (2 or 3 classes).
We use Self-Ensembling BIBREF15 as our underlying framework. Self-Ensembling builds on top of Temporal Ensembling BIBREF23 and the Mean-Teacher network BIBREF14 , both of which originally proposed for semi-supervised learning. In visual domain adaptation, Self-Ensembling shows superior performance than many recently proposed approaches BIBREF35 , BIBREF36 , BIBREF37 , BIBREF38 , BIBREF39 , BIBREF40 , including GAN-based approaches. To the best of our knowledge, this approach has not been used on language data.
Conclusion
We present a new model for predicting sentence specificity. We augment the Self-Ensembling method BIBREF15 for unsupervised domain adaptation on text data. We also regularize the distribution of predictions to match a reference distribution. Using only binary labeled sentences from news articles as source domain, our system could generate real-valued specificity predictions on different target domains, significantly outperforming previous work on sentence specificity prediction. Finally, we show that sentence specificity prediction can potentially be beneficial in improving the quality and informativeness of dialogue generation systems.
Acknowledgments
This research was partly supported by the Amazon Alexa Graduate Fellowship. We thank the anonymous reviewers for their helpful comments.

Question: What domains do they experiment with?
Answer: Twitter, Yelp reviews and movie reviews