Task setup	With unsupervised domain adaptation, one has access to labeled sentence specificity in one source domain, and unlabeled sentences in all target domains. The goal is to predict the specificity of target domain data. Our source domain is news, the only domain with publicly available labeled data for training  . We crowdsource sentence specificity for evaluation for three target domains: Twitter, Yelp reviews and movie reviews. The data is described in Section SECREF4 . Existing sentence specificity labels in news are binary, i.e., a sentence is either general or specific. However, in practice, real-valued estimates of sentence specificity are widely adopted  ,  ,  ,  ,  ,  . Most of these work directly uses the classifier posterior distributions, although we will later show that such distributions do not follow the true distribution of sentence specificity (see Figure FIGREF29 , Speciteller vs. real). We aim to produce accurate real-valued specificity estimates despite the binary training data. Specifically, the test sentences have real-valued labels between 0 and 1. We evaluate our system using mean absolute error, Kendall's Tau and Spearman correlation.

Experiments	We now evaluate our framework on predicting sentence specificity for the three target domains. We report experimental results on a series of settings to evaluate the performance of different components.

In preliminary experiments, we found that combining hand-crafted features with the BiLSTM performed better than using either one individually.

Specificity in dialogue generation	We also evaluate our framework in open-domain dialogue. This experiment also presents a case for the usefulness of an effective sentence specificity system in dialogue generation. With seq2seq dialogue generation models,  observed significant quality improvement by removing training examples with short responses during preprocessing; this is potentially related to this type of models favor generating non-informative, generic responses  ,  . We show that filtering training data by predicted specificity results in responses of higher quality and informativeness than filtering by length.

The consistency loss is defined as the squared difference between the predictions of the student and teacher networks  : DISPLAYFORM0  where INLINEFORM0 denotes the base network and x denotes the input sentence. The teacher network is not involved when minimizing the supervised loss INLINEFORM1 . An important difference between our work and  is that in their work only unlabeled target data contributes to the consistency loss INLINEFORM0 . Instead, we use both source and target sentences, bringing the predictions of the two networks close to each other not only on the target domain, but also the source domain. Unlike many vision tasks where predictions intuitively stay the same with different types of image augmentation (e.g., transformation and scaling), text is more sensitive to noise. Self-Ensembling relies on heavy augmentation on both student and teacher networks, and our experiments revealed that incorporating source data in the consistency loss term mitigates additional biases from noise augmentation. At training time, the teacher network's parameters are fixed during gradient descent, and the gradient only propagates through the student network. After each update of the student network, we recalculate the weights of the teacher network using exponential moving average. At testing time, we use the teacher network for prediction.

Evaluation	We use automatic measures and human evaluation as in  and  . Table TABREF31 shows the diversity and perplexity of the responses. Diversity is calculated as the type-token ratio of unigrams and bigrams. The test set for these two metrics is a random sample of 10K instances from OpenSubtitles that doesn't overlap with the training set. Clearly, filtering training data according to specificity results in more diverse responses with lower perplexity than filtering by length. We also crowdsource human evaluation for quality; in addition, we evaluate the systems for response informativeness. Note that in our instructions, informativeness means the usefulness of information and is a distinct measure from specificity. The original training data of specificity are linguistically annotated and involves only the change in the level of details  . Separate experiments are conducted to avoid priming. We use a test set of 388 instances, including questions randomly sampled from OpenSubtitles that doesn't overlap with the training set, and 188 example questions from previous dialogue generation papers, including  . We use Amazon MechenicalTurk for crowdsourcing. 7 workers chose between the two responses to the same question.

Target domains	We evaluate on three target domains: Twitter, Yelp and movie reviews. Since no annotated data exist for these domains, we crowdsource specificity for sentences sampled from each domain using Amazon Mechanical Turk. We follow the context-independent annotation instructions from  . Initially, 9 workers labeled specificity for 1000 sentences in each domain on a scale of 1 (very general) to 5 (very specific), which we rescaled to 0, 0.25, 0.5, 0.75, and 1. Inter-annotator agreement (IAA) is calculated using average Cronbach's alpha  values for each worker. For quality control, we exclude workers with IAA below 0.3, and include the remaining sentences that have at least 5 raters. Our final IAA values fall between 0.68-0.70, compatible with the reported 0.72 from expert annotators in  . The final specificity value is aggregated to be the average of the rescaled ratings. We also use large sets of unlabeled data in each domain: Twitter: 984 tweets annotated, 50K unlabeled, sampled from  .

Results and analysis	Table TABREF25 shows the full results for the baselines and each configuration of our framework. For analysis, we also plot in Figure FIGREF29 the true specificity distributions in Twitter test set, predicted distributions for Speciteller, the Self-Ensembling baseline (SE), SE with adaptation (SE+A) and also with distribution regularization (SE+AD). Speciteller, which is trained on news sentences, cannot generalize well to other domains, as it performs worse than just using sentence length for two of the three domains (Yelp and Movie). From Figure FIGREF29 , we can see that the prediction mass of Speciteller is near the extrema values 0 and 1, and the rest of the predictions falls uniformly in between. These findings confirm the need of a generalizable system. Across all domains and all metrics, the best performing system is our full system with domain adaptation and distribution regularization (SE+AD with mean-std or KL), showing that the system generalizes well across different domains.

Domain Agnostic Real-Valued Specificity Prediction	Sentence specificity quantifies the level of detail in a sentence, characterizing the organization of information in discourse. While this information is useful for many downstream applications, specificity prediction systems predict very coarse labels (binary or ternary) and are trained on and tailored toward specific domains (e.g., news). The goal of this work is to generalize specificity prediction to domains where no labeled data is available and output more nuanced real-valued specificity ratings. We present an unsupervised domain adaptation system for sentence specificity prediction, specifically designed to output real-valued estimates from binary training labels. To calibrate the values of these predictions appropriately, we regularize the posterior distribution of the labels towards a reference distribution. We show that our framework generalizes well to three different domains with 50%~68% mean absolute error reduction than the current state-of-the-art system trained for news sentence specificity. We also demonstrate the potential of our work in improving the quality and informativeness of dialogue generation systems.

They are also trained to label a sentence as either general or specific  ,  , or low/medium/high specificity  , though in practice specificity has been analyzed as a continuous value  ,  ,  ,  ,  ,  , as it should be  . In this work, we present an unsupervised domain adaptation system for sentence specificity prediction, specifically designed to output real-valued estimates. It effectively generalizes sentence specificity analysis to domains where no labeled data is available, and outputs values that are close to the real-world distribution of sentence specificity. Our main framework is an unsupervised domain adaptation system based on Self-Ensembling  ,  that simultaneously reduces source prediction errors and generates feature representations that are robust against noise and across domains. Past applications of this technique have focused on computer vision problems; to make it effective for text processing tasks, we modify the network to better utilize labeled data in the source domain and explore several data augmentation methods for text. We further propose a posterior regularization technique  that generally applies to the scenario where it is easy to get coarse-grained categories of labels, but fine-grained predictions are needed.

Conclusion	We present a new model for predicting sentence specificity. We augment the Self-Ensembling method  for unsupervised domain adaptation on text data. We also regularize the distribution of predictions to match a reference distribution. Using only binary labeled sentences from news articles as source domain, our system could generate real-valued specificity predictions on different target domains, significantly outperforming previous work on sentence specificity prediction. Finally, we show that sentence specificity prediction can potentially be beneficial in improving the quality and informativeness of dialogue generation systems.

Source domain	The source domain for sentence specificity is news, for which we use three publicly available labeled datasets: (1) training sentences from  and  , which consists of 1.4K general and 1.4K specific sentences from the Wall Street Journal. (2) 900 news sentences crowdsourced for binary general/specific labels  ; 55% of them are specific. (3) 543 news sentences from  . These sentences are rated on a scale of INLINEFORM0 , so for consistency with the rest of the training labels, we pick sentences with average rating INLINEFORM1 as general examples and those with average rating INLINEFORM2 as specific. In total, we have 4.3K sentences with binary labels in the source domain.

Specifically, our regularization term seeks to move the distribution of the classifier posterior probabilities closer to that of a pre-specified target distribution, which in our case is a specificity distribution derived from the source domain. Experimental results show that our system generates more accurate real-valued sentence specificity predictions that correlate well with human judgment, across three domains that are vastly different from the source domain (news): Twitter, Yelp reviews and movie reviews. Compared to a state-of-the-art system trained on news data  , our best setting achieves a 50%-68% reduction in mean absolute error and increases Kendall's Tau and Spearman correlations by 0.07-0.10 and 0.12-0.13, respectively. Finally, we conduct a task-based evaluation that demonstrates the usefulness of sentence specificity prediction in open-domain dialogue generation. Prior work showed that the quality of responses from dialog generation systems can be significantly improved if short examples are removed from training  , potentially preventing the system from overly favoring generic responses  ,  . We show that predicted specificity works more effectively than length, and enables the system to generate more diverse and informative responses with better quality. In sum, the paper's contributions are as follows:

Using a paired Wilcoxon test, it significantly ( INLINEFORM0 ) outperforms Speciteller in terms of MAE; it also achieved higher Spearman and Kendall's Tau correlations than both Length and Speciteller. Component-wise, the Self-Ensembling baseline (SE) achieves significantly lower MAE than Speciteller, and higher correlations than either baseline. Figure FIGREF29 shows that unlike Speciteller, the SE baseline does not have most of its prediction mass near 0 and 1, demonstrating the effectiveness of temporal ensembling. Using both the consistency loss INLINEFORM0 and the distribution regularization INLINEFORM1 achieves the best results on all three domains; however adding only INLINEFORM2 (SE+A) or INLINEFORM3 (SE+D) improves for some measures or domains but not all. This shows that both terms are crucial to make the system robust across domains. The improvements from distribution regularization are visualized in Figure FIGREF29 . With SE+A, most of the predicted labels are between 0.4 and 0.6. Applying distribution regularization (SE+AD) makes them much closer to the real distribution of specificity.

Introduction	The specificity of a sentence measures its “quality of belonging or relating uniquely to a particular subject”  . It is often pragmatically defined as the level of detail in the sentence  ,  . When communicating, specificity is adjusted to serve the intentions of the writer or speaker  . In the examples below, the second sentence is clearly more specific than the first one: Ex1: This brand is very popular and many people use its products regularly. Ex2: Mascara is the most commonly worn cosmetic, and women will spend an average of $4,000 on it in their lifetimes. Studies have demonstrated the important role of sentence specificity in reading comprehension  and in establishing common ground in dialog  . It has also been shown to be a key property in analyses and applications, such as summarization  , argumentation mining  , political discourse analysis  , student discussion assessment  ,  , deception detection  , and dialogue generation  . Despite their usefulness, prior sentence specificity predictors  ,  ,  are trained with sentences from specific domains (news or classroom discussions), and have been found to fall short when applied to other domains  ,  .

The student learns from labeled source data and minimizes the supervised cross-entropy loss INLINEFORM1 . Domain adaptation is achieved by minimizing the consistency loss between the two networks, which can be done with unlabeled target data. The overall loss function is the weighted sum of INLINEFORM2 and INLINEFORM3 . Figure FIGREF6 depicts the process. Concretely, the student and teacher networks are of identical structure following the base model (Section SECREF7 ), but features distinct noise augmentation. The student network learns to predict sentence specificity from labeled source domain data. The input sentences are augmented with noise INLINEFORM0 . The teacher network predicts the specificity of each sentence with a different noise augmentation INLINEFORM1 . Parameters of the teacher network INLINEFORM2 are updated each time step to be the exponential moving average of the corresponding parameters in the student network. The teacher parameter INLINEFORM3 at each time step INLINEFORM4 is DISPLAYFORM0  where INLINEFORM0 is the degree of weighting decay, a constant between 0 and 1. INLINEFORM1 denotes parameters for the student network.

Unsupervised domain adaptation	Our unsupervised domain adaptation framework is based on Self-Ensembling  ; we lay out the core ideas first and then discuss modifications to make the framework suitable for textual data. There are two ideas acting as the driving force behind Self-Ensembling. First, adding noise to each data point INLINEFORM0 can help regularize the model by encouraging the model prediction to stay the same regardless of the noise, creating a manifold around the data point where predictions are invariant  . This can be achieved by minimizing a consistency loss INLINEFORM1 between the predictions. Second, temporal ensembling—ensemble the same model trained at different time steps—is shown to be beneficial to prediction especially in semi-supervised cases  ; in particular, we average model parameters from each time step  . These two ideas are realized by using a student network and a teacher network. The parameters of the teacher network are the exponential average of that of the student network, making the teacher a temporal ensemble of the student. Distinct noise augmentations are applied to the input of each network, hence the consistency loss INLINEFORM0 is applied between student and teacher predictions.

Regularizing the posterior distribution	Sentence specificity prediction is a task where the existing training data have binary labels, while real-valued outputs are desirable. Prior work has directly used classifier posterior probabilities. However, the posterior distribution and the true specificity distribution are quite different (see Figure FIGREF29 , Speciteller vs. real). We propose a regularization term to bridge the gap between the two. Specifically, we view the posterior probability distribution as a latent distribution, which allows us to apply a variant of posterior regularization  , previously used to apply pre-specified constraints to latent variables in structured prediction. Here, we apply a distance penalty between the latent distribution and a pre-specified reference distribution (which, in our work, is from the source domain).  found that in news, the distribution of sentence specificity is bell shaped, similar to that of a Gaussian. Our analysis on sentence specificity for three target domains yields the same insights (Figure FIGREF18 ). We explored two regularization formulations, with and without assuming that the two distributions are Gaussian. Both were successful and achieved similar performance. Let INLINEFORM0 and INLINEFORM1 be the mean and standard deviation of the predictions (posterior probabilities) in a batch.

An important factor contributing to the effectiveness of Self-Ensembling is applying noise to the inputs to make them more robust against domain shifts. For computer vision tasks, augmentation techniques including affine transformation, scaling, flipping and cropping could be used  . However these operations could not be used on text. We designed several noise augmentations for sentences, including: adding Gaussian noise to both the word embeddings and shallow features; randomly removing words in a sentence; substituting word embeddings with a random vector, or a zero vector like applying dropout. To produce enough variations of data, augmentation is applied to INLINEFORM0 half of the words in a sentence.

Related Work	Sentence specificity prediction as a task is proposed by  , who repurposed discourse relation annotations from WSJ articles  for sentence specificity training.  incorporated more news sentences as unlabeled data.  developed a system to predict sentence specificity for classroom discussions, however the data is not publicly available. All these systems are classifiers trained with categorical data (2 or 3 classes). We use Self-Ensembling  as our underlying framework. Self-Ensembling builds on top of Temporal Ensembling  and the Mean-Teacher network  , both of which originally proposed for semi-supervised learning. In visual domain adaptation, Self-Ensembling shows superior performance than many recently proposed approaches  ,  ,  ,  ,  ,  , including GAN-based approaches. To the best of our knowledge, this approach has not been used on language data.

The first formulation assumes that the predictions and reference distributions are Gaussian. It uses the KL divergence between the predicted distribution INLINEFORM2 and the reference Gaussian distribution INLINEFORM3 . The distribution regularization loss can be written as: DISPLAYFORM0  The second formulation does not assume Gaussian distributions and only compares the mean and standard deviation of the two distributions using a weighting term INLINEFORM0 : DISPLAYFORM0  Combining the regularization term INLINEFORM0 into a single objective, the total loss is thus: DISPLAYFORM0  where INLINEFORM0 is the cross entropy loss for the source domain predictions, INLINEFORM1 is the consistency loss. INLINEFORM2 and INLINEFORM3 are weighting hyperparameters. In practice, this regularization term serves a second purpose. After adding the consistency loss INLINEFORM0 , we observed that the predictions are mostly close to each other with values between 0.4 and 0.6, and their distribution resembles a Gaussian with very small variance (c.f. Figure FIGREF29 line SE+A). This might be due to the consistency loss pulling all the predictions together, since when all predictions are identical, the loss term will be zero.

Training details	Hyperparameters are tuned on a validation set of 200 tweets that doesn't overlap with the test set. We then use this set of parameters for all testing domains. The LSTM encoder generates 100-dimensional representations. For the multilayer perceptron, we use 3 fully connected 100-dimensional layers. We use ReLU activation with batch normalization. For the Gaussian noise in data augmentation, we use standard deviation 0.1 for word embeddings and 0.2 for shallow features. The probabilities of deleting a word and replacing a word vector are 0.15. The exponential moving average decay INLINEFORM0 is 0.999. Dropout rate is 0.5 for all layers. The batch size is 32. INLINEFORM1 , INLINEFORM2 for KL loss and 100 for mean and std.dev loss. INLINEFORM3 . We fix the number of training to be 30 epochs for SE+A and SE+AD, 10 epochs for SE, and 15 epochs for SE+D. We use the Adam optimizer with learning rate 0.0001, INLINEFORM4 , INLINEFORM5 . As discussed in Section SECREF4 , posterior distribution regularization parameters INLINEFORM6 are set to be those from  .

Task and settings	We implemented a seq2seq question-answering bot with attention using OpenNMT  . The bot is trained on OpenSubtitles  following prior work  . We restrict the instances to question-answer pairs by selecting consecutive sentences with the first sentence ending with question mark, the second sentence without question mark and follows the first sentence by less than 20 seconds, resulting in a 14M subcorpus. The model uses two hidden layers of size 2048, optimized with Adam with learning rate 0.001. The batch size is 64. While decoding, we use beam size 5; we block repeating n-grams, and constrain the minimum prediction length to be 5. These parameters are tuned on a development set. We compare two ways to filter training data during preprocessing: Remove Short: Following  , remove training examples with length of responses shorter than a threshold of 5. About half of the data are removed. Remove General: Remove predicted general responses from training examples using our system. We use the responses in the OpenSubtitles training set as the unlabeled target domain data during training.

Base model	Figure FIGREF5 depicts the base model for sentence specificity prediction. The overall structure is similar to that in  . Each word in the sentence is encoded into an embedding vector and the sentence is passed through a bidirectional Long-Short Term Memory Network (LSTM)  to generate a representation of the sentence. This representation is concatenated with a series of hand crafted features, and then passed through a multilayer perceptron to generate specificity predictions INLINEFORM0 . Training treats these as probabilities of the positive class, but we will show in Section SECREF13 how they can be adapted to make real-valued predictions. Our hand crafted features are taken from  , including: the number of tokens in the sentence; the number of numbers, capital letters and punctuations in the sentence, normalized by sentence length; the average number of characters in each word; the fraction of stop words; the number of words that can serve as explicit discourse connectives; the fraction of words that have sentiment polarity or are strongly subjective; the average familiarity and imageability of words; and the minimum, maximum, and average inverse document frequency (idf) over the words in the sentence.

Acknowledgments	This research was partly supported by the Amazon Alexa Graduate Fellowship. We thank the anonymous reviewers for their helpful comments.

This regularization can be used to counter this effect, and avoid the condensation of predicted values. Finally, this regularization is distinct from class imbalance loss terms such as that used in  , which we found early on to hurt performance rather than helping.

Yelp: 845 sentences annotated, 95K unlabeled, sampled from the Yelp Dataset Challenge 2015  . Movie: 920 sentences annotated, 12K unlabeled, sampled from  . Figure FIGREF18 shows the distribution of ratings for the annotated data. We also plot a fitted Gaussian distribution for comparison. Clearly, most sentences have mid-range specificity values, consistent with news sentences  . Interestingly the mean and variance for the three distributions are similar to each other and to those from  , as show in Table TABREF23 . Therefore, we use the source distribution (news,  ) as the reference distribution for posterior distribution regularization, and set INLINEFORM0 to 0.417 and 0.227 accordingly.

Systems	Length baseline This simple baseline predicts specificity proportionally to the number of words in the sentence. Shorter sentences are predicted as more general and longer sentences are predicted as more specific. Speciteller baseline Speciteller  is a semi-supervised system trained on news data with binary labels. The posterior probabilities of the classifier are used directly as specificity values. Self-ensembling baseline (SE) Our system with the teacher network, but only using exponential moving average (without the consistency loss or distribution regularization). Distribution only (SE+D) Our system with distribution regularization INLINEFORM0 using the mean and standard deviation (Eq. EQREF15 ), but without the consistency loss INLINEFORM1 . Adaptation only (SE+A) Our system with the consistency loss INLINEFORM0 , but without distribution regularization INLINEFORM1 . SE+AD (KL) Our system with both INLINEFORM0 and INLINEFORM1 using KL divergence (Eq. EQREF14 ). SE+AD (mean-std) Our system with both INLINEFORM0 and INLINEFORM1 using mean and standard deviation (Eq. EQREF15 ).

SE+AD (no augmentation) We also show the importance of noise augmentation, by benchmarking the same setup as SE+AD (mean-std) without data augmentation.

Evaluation metrics	We use 3 metrics to evaluate real-valued predictions: (1) the Spearman correlation between the labeled and predicted specificity values, higher is better; (2) the pairwise Kendall's Tau correlation, higher is better; (3) mean absolute error (MAE): INLINEFORM0 , lower is better.

With respect to the two formulations of regularization (KL and mean-std), both are effective in generating more accurate real-valued estimates. Their performances are comparable, hence using only the mean and standard deviation values, without explicitly modeling the reference Gaussian distribution, works equally well. Finally, without data augmentation (column no aug.), the correlations are clearly lower than our full model, stressing the importance of data augmentation in our framework.

We remove the least specific responses such that the resulting number of examples is the same as Remove Short. For fair comparison, at test time, we adjust the length penalty described in  for both models, so the average response lengths are the same among both models.

Table TABREF32 shows the human evaluation comparing Remove Short vs. Remove General. Removing predicted general responses performs better than removing short sentences, on both informativeness and quality and on both test sets. This shows that sentence specificity is a superior measure for training data preprocessing than sentence length.

Architecture	Our core technique is the Self-Ensembling  ,  of a single base classification model that utilizes data augmentation, with a distribution regularization term to generate accurate real-valued predictions from binary training labels.

Question: What domains do they experiment with?
Answer: Twitter, Yelp reviews and movie reviews