Transfer Learning Between Related Tasks Using Expected Label Proportions

Abstract: Deep learning systems thrive on abundance of labeled training data but such data is not always available, calling for alternative methods of supervision. One such method is expectation regularization (XR) (Mann and McCallum, 2007), where models are trained based on expected label proportions. We propose a novel application of the XR framework for transfer learning between related tasks, where knowing the labels of task A provides an estimation of the label proportion of task B. We then use a model trained for A to label a large corpus, and use this corpus with an XR loss to train a model for task B. To make the XR framework applicable to large-scale deep-learning setups, we propose a stochastic batched approximation procedure. We demonstrate the approach on the task of Aspect-based Sentiment classification, where we effectively use a sentence-level sentiment predictor to train accurate aspect-based predictor. The method improves upon fully supervised neural system trained on aspect-level data, and is also cumulative with LM-based pretraining, as we demonstrate by improving a BERT-based Aspect-based Sentiment model.

Data annotation is a key bottleneck in many data driven algorithms. Specifically, deep learning models, which became a prominent tool in many data driven tasks in recent years, require large datasets to work well. However, many tasks require manual annotations which are relatively hard to obtain at scale. An attractive alternative is lightly supervised learning BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 , in which the objective function is supplemented by a set of domain-specific soft-constraints over the model's predictions on unlabeled data. For example, in label regularization BIBREF0 the model is trained to fit the true label proportions of an unlabeled dataset. Label regularization is special case of expectation regularization (XR) BIBREF0 , in which the model is trained to fit the conditional probabilities of labels given features.
In this work we consider the case of correlated tasks, in the sense that knowing the labels for task A provides information on the expected label composition of task B. We demonstrate the approach using sentence-level and aspect-level sentiment analysis, which we use as a running example: knowing that a sentence has positive sentiment label (task A), we can expect that most aspects within this sentence (task B) will also have positive label. While this expectation may be noisy on the individual example level, it holds well in aggregate: given a set of positively-labeled sentences, we can robustly estimate the proportion of positively-labeled aspects within this set. For example, in a random set of positive sentences, we expect to find 90% positive aspects, while in a set of negative sentences, we expect to find 70% negative aspects. These proportions can be easily either guessed or estimated from a small set.
We propose a novel application of the XR framework for transfer learning in this setup. We present an algorithm (Sec SECREF12 ) that, given a corpus labeled for task A (sentence-level sentiment), learns a classifier for performing task B (aspect-level sentiment) instead, without a direct supervision signal for task B. We note that the label information for task A is only used at training time. Furthermore, due to the stochastic nature of the estimation, the task A labels need not be fully accurate, allowing us to make use of noisy predictions which are assigned by an automatic classifier (Sections SECREF12 and SECREF4 ). In other words, given a medium-sized sentiment corpus with sentence-level labels, and a large collection of un-annotated text from the same distribution, we can train an accurate aspect-level sentiment classifier.
The XR loss allows us to use task A labels for training task B predictors. This ability seamlessly integrates into other semi-supervised schemes: we can use the XR loss on top of a pre-trained model to fine-tune the pre-trained representation to the target task, and we can also take the model trained using XR loss and plentiful data and fine-tune it to the target task using the available small-scale annotated data. In Section SECREF56 we explore these options and show that our XR framework improves the results also when applied on top of a pre-trained Bert-based model BIBREF9 .
Finally, to make the XR framework applicable to large-scale deep-learning setups, we propose a stochastic batched approximation procedure (Section SECREF19 ). Source code is available at https://github.com/MatanBN/XRTransfer.
An effective way to supplement small annotated datasets is to use lightly supervised learning, in which the objective function is supplemented by a set of domain-specific soft-constraints over the model's predictions on unlabeled data. Previous work in lightly-supervised learning focused on training classifiers by using prior knowledge of label proportions BIBREF2 , BIBREF3 , BIBREF10 , BIBREF0 , BIBREF11 , BIBREF12 , BIBREF7 , BIBREF13 , BIBREF14 , BIBREF15 , BIBREF16 , BIBREF8 or prior knowledge of features label associations BIBREF1 , BIBREF17 , BIBREF18 , BIBREF19 , BIBREF20 . In the context of NLP, BIBREF17 suggested to use distributional similarities of words to train sequence models for part-of-speech tagging and a classified ads information extraction task. BIBREF19 used background lexical information in terms of word-class associations to train a sentiment classifier. BIBREF21 , BIBREF22 suggested to exploit the bilingual correlations between a resource rich language and a resource poor language to train a classifier for the resource poor language in a lightly supervised manner.
Expectation Regularization (XR) BIBREF0 is a lightly supervised learning method, in which the model is trained to fit the conditional probabilities of labels given features. In the context of NLP, XR was used by BIBREF20 to train twitter-user attribute prediction using hundreds of noisy distributional expectations based on census demographics. Here, we suggest using XR to train a target task (aspect-level sentiment) based on the output of a related source-task classifier (sentence-level sentiment).
The main idea of XR is moving from a fully supervised situation in which each data-point INLINEFORM0 has an associated label INLINEFORM1 , to a setup in which sets of data points INLINEFORM2 are associated with corresponding label proportions INLINEFORM3 over that set.
Formally, let INLINEFORM0 be a set of data points, INLINEFORM1 be a set of INLINEFORM2 class labels, INLINEFORM3 be a set of sets where INLINEFORM4 for every INLINEFORM5 , and let INLINEFORM6 be the label distribution of set INLINEFORM7 . For example, INLINEFORM8 would indicate that 70% of data points in INLINEFORM9 are expected to have class 0, 20% are expected to have class 1 and 10% are expected to have class 2. Let INLINEFORM10 be a parameterized function with parameters INLINEFORM11 from INLINEFORM12 to a vector of conditional probabilities over labels in INLINEFORM13 . We write INLINEFORM14 to denote the probability assigned to the INLINEFORM15 th event (the conditional probability of INLINEFORM16 given INLINEFORM17 ).
A typically objective when training on fully labeled data of INLINEFORM0 pairs is to maximize likelihood of labeled data using the cross entropy loss, INLINEFORM1 
Instead, in XR our data comes in the form of pairs INLINEFORM0 of sets and their corresponding expected label proportions, and we aim to optimize INLINEFORM1 to fit the label distribution INLINEFORM2 over INLINEFORM3 , for all INLINEFORM4 .
As counting the number of predicted class labels over a set INLINEFORM0 leads to a non-differentiable objective, BIBREF0 suggest to relax it and use instead the model's posterior distribution INLINEFORM1 over the set: DISPLAYFORM0 DISPLAYFORM1 
where INLINEFORM0 indicates the INLINEFORM1 th entry in INLINEFORM2 . Then, we would like to set INLINEFORM3 such that INLINEFORM4 and INLINEFORM5 are close. BIBREF0 suggest to use KL-divergence for this. KL-divergence is composed of two parts: INLINEFORM6 INLINEFORM7 
Since INLINEFORM0 is constant, we only need to minimize INLINEFORM1 , therefore the loss function becomes: DISPLAYFORM0 
Notice that computing INLINEFORM0 requires summation over INLINEFORM1 for the entire set INLINEFORM2 , which can be prohibitive. We present batched approximation (Section SECREF19 ) to overcome this.
 BIBREF0 find that XR might find a degenerate solution. For example, in a three class classification task, where INLINEFORM0 , it might find a solution such that INLINEFORM1 for every instance, as a result, every instance will be classified the same. To avoid this, BIBREF0 suggest to penalize flat distributions by using a temperature coefficient T likewise: DISPLAYFORM0 
Where z is a feature vector and W and b are the linear classifier parameters.
In the aspect-based sentiment classification (ABSC) task, we are given a sentence and an aspect, and need to determine the sentiment that is expressed towards the aspect. For example the sentence “Excellent food, although the interior could use some help.“ has two aspects: food and interior, a positive sentiment is expressed about the food, but a negative sentiment is expressed about the interior. A sentence INLINEFORM0 , may contain 0 or more aspects INLINEFORM1 , where each aspect corresponds to a sub-sequence of the original sentence, and has an associated sentiment label (Neg, Pos, or Neu). Concretely, we follow the task definition in the SemEval-2015 and SemEval-2016 shared tasks BIBREF23 , BIBREF24 , in which the relevant aspects are given and the task focuses on finding the sentiment label of the aspects.
While sentence-level sentiment labels are relatively easy to obtain, aspect-level annotation are much more scarce, as demonstrated in the small datasets of the SemEval shared tasks.
[t!] Inputs: A dataset INLINEFORM0 , batch size INLINEFORM1 , differentiable classifier INLINEFORM2 [H] not converged INLINEFORM3 random( INLINEFORM4 ) INLINEFORM5 random-choice( INLINEFORM6 , INLINEFORM7 ) INLINEFORM8 INLINEFORM9 INLINEFORM10 INLINEFORM11 Compute loss INLINEFORM12 (eq (4)) Compute gradients and update INLINEFORM13 INLINEFORM14 Stochastic Batched XR
Consider two classification tasks over a shared input space, a source task INLINEFORM0 from INLINEFORM1 to INLINEFORM2 and a target task INLINEFORM3 from INLINEFORM4 to INLINEFORM5 , which are related through a conditional distribution INLINEFORM6 . In other words, a labeling decision for task INLINEFORM7 induces an expected label distribution over the task INLINEFORM8 . For a set of datapoints INLINEFORM9 that share a source label INLINEFORM10 , we expect to see a target label distribution of INLINEFORM11 .
Given a large unlabeled dataset INLINEFORM0 , a small labeled dataset for the target task INLINEFORM1 , classifier INLINEFORM2 (or sufficient training data to train one) for the source task, we wish to use INLINEFORM3 and INLINEFORM4 to train a good classifier INLINEFORM5 for the target task. This can be achieved using the following procedure.
Apply INLINEFORM0 to INLINEFORM1 , resulting in a noisy source-side labels INLINEFORM2 for the target task.
Estimate the conditional probability INLINEFORM0 table using MLE estimates over INLINEFORM1 INLINEFORM2 
where INLINEFORM0 is a counting function over INLINEFORM1 .
Apply INLINEFORM0 to the unlabeled data INLINEFORM1 resulting in labels INLINEFORM2 . Split INLINEFORM3 into INLINEFORM4 sets INLINEFORM5 according to the labeling induced by INLINEFORM6 : INLINEFORM7 
Use Algorithm SECREF12 to train a classifier for the target task using input pairs INLINEFORM0 and the XR loss.
In words, by using XR training, we use the expected label proportions over the target task given predicted labels of the source task, to train a target-class classifier.
 BIBREF0 and following work take the base classifier INLINEFORM0 to be a logistic regression classifier, for which they manually derive gradients for the XR loss and train with LBFGs BIBREF25 . However, nothing precludes us from using an arbitrary neural network instead, as long as it culminates in a softmax layer.
One complicating factor is that the computation of INLINEFORM0 in equation ( EQREF5 ) requires a summation over INLINEFORM1 for the entire set INLINEFORM2 , which in our setup may contain hundreds of thousands of examples, making gradient computation and optimization impractical. We instead proposed a stochastic batched approximation in which, instead of requiring that the full constraint set INLINEFORM3 will match the expected label posterior distribution, we require that sufficiently large random subsets of it will match the distribution. At each training step we compute the loss and update the gradient with respect to a different random subset. Specifically, in each training step we sample a random pair INLINEFORM4 , sample a random subset INLINEFORM5 of INLINEFORM6 of size INLINEFORM7 , and compute the local XR loss of set INLINEFORM8 : DISPLAYFORM0 
where INLINEFORM0 is computed by summing over the elements of INLINEFORM1 rather than of INLINEFORM2 in equations ( EQREF5 –2). The stochastic batched XR training algorithm is given in Algorithm SECREF12 . For large enough INLINEFORM3 , the expected label distribution of the subset is the same as that of the complete set.
We demonstrate the procedure given above by training Aspect-based Sentiment Classifier (ABSC) using sentence-level sentiment signals.
We observe that while the sentence-level sentiment does not determine the sentiment of individual aspects (a positive sentence may contain negative remarks about some aspects), it is very predictive of the proportion of sentiment labels of the fragments within a sentence. Positively labeled sentences are likely to have more positive aspects and fewer negative ones, and vice-versa for negatively-labeled sentences. While these proportions may vary on the individual sentence level, we expect them to be stable when aggregating fragments from several sentences: when considering a large enough sample of fragments that all come from positively labeled sentences, we expect the different samples to have roughly similar label proportions to each other. This situation is idealy suited for performing XR training, as described in section SECREF12 .
The application to ABSC is almost straightforward, but is complicated a bit by the decomposition of sentences into fragments: each sentence level decision now corresponds to multiple fragment-level decisions. Thus, we apply the sentence-level (task A) classifier INLINEFORM0 on the aspect-level corpus INLINEFORM1 by applying it on the sentence level and then associating the predicted sentence labels with each of the fragments, resulting in fragment-level labeling. Similarly, when we apply INLINEFORM2 to the unlabeled data INLINEFORM3 we again do it at the sentence level, but the sets INLINEFORM4 are composed of fragments, not sentences: INLINEFORM5 
We then apply algorithm SECREF12 as is: at each step of training we sample a source label INLINEFORM0 Pos,Neg,Neu INLINEFORM1 , sample INLINEFORM2 fragments from INLINEFORM3 , and use the XR loss to fit the expected fragment-label proportions over these INLINEFORM4 fragments to INLINEFORM5 . Figure FIGREF21 illustrates the procedure.
We model the ABSC problem by associating each (sentence,aspect) pair with a sentence-fragment, and constructing a neural classifier from fragments to sentiment labels. We heuristically decompose a sentence into fragments. We use the same BiLSTM based neural architecture for both sentence classification and fragment classification.
We now describe the procedure we use to associate a sentence fragment with each (sentence,aspect) pairs. The shared tasks data associates each aspect with a pivot-phrase INLINEFORM0 , where pivot phrase INLINEFORM1 is defined as a pre-determined sequence of words that is contained within the sentence. For a sentence INLINEFORM2 , a set of pivot phrases INLINEFORM3 and a specific pivot phrase INLINEFORM4 , we consult the constituency parse tree of INLINEFORM5 and look for tree nodes that satisfy the following conditions:
The node governs the desired pivot phrase INLINEFORM0 .
The node governs either a verb (VB, VBD, VBN, VBG, VBP, VBZ) or an adjective (JJ, JJR, JJS), which is different than any INLINEFORM0 .
The node governs a minimal number of pivot phrases from INLINEFORM0 , ideally only INLINEFORM1 .
We then select the highest node in the tree that satisfies all conditions. The span governed by this node is taken as the fragment associated with aspect INLINEFORM0 . The decomposition procedure is demonstrated in Figure FIGREF22 .
When aspect-level information is given, we take the pivot-phrases to be the requested aspects. When aspect-level information is not available, we take each noun in the sentence to be a pivot-phrase.
Our classification model is a simple 1-layer BiLSTM encoder (a concatenation of the last states of a forward and a backward running LSTMs) followed by a linear-predictor. The encoder is fed either a complete sentence or a sentence fragment.
Table TABREF44 compares these baselines to three XR conditions.
The first condition, BiLSTM-XR-Dev, performs XR training on the automatically-labeled sentence-level dataset. The only access it has to aspect-level annotation is for estimating the proportions of labels for each sentence-level label, which is done based on the validation set of SemEval-2015 (i.e., 20% of the train set). The XR setting is very effective: without using any in-task data, this model already surpasses all other models, both supervised and semi-supervised, except for the BIBREF35 , BIBREF34 models which achieve higher F1 scores. We note that in contrast to XR, the competing models have complete access to the supervised aspect-based labels. The second condition, BiLSTM-XR, is similar but now the model is allowed to estimate the conditional label proportions based on the entire aspect-based training set (the classifier still does not have direct access to the labels beyond the aggregate proportion information). This improves results further, showing the importance of accurately estimating the proportions. Finally, in BiLSTM-XR+Finetuning, we follow the XR training with fully supervised fine-tuning on the small labeled dataset, using the attention-based model of BIBREF35 . This achieves the best results, and surpasses also the semi-supervised BIBREF35 baseline on accuracy, and matching it on F1.
We report significance tests for the robustness of the method under random parameter initialization. Our reported numbers are averaged over five random initialization. Since the datasets are unbalanced w.r.t the label distribution, we report both accuracy and macro-F1.
The XR training is also more stable than the other semi-supervised baselines, achieving substantially lower standard deviations across different runs.
In each experiment in this section we estimate the proportions using the SemEval-2015 train set.
How does the XR training scale with the amount of unlabeled data? Figure FIGREF54 a shows the macro-F1 scores on the entire SemEval-2016 dataset, with different unlabeled corpus sizes (measured in number of sentences). An unannotated corpus of INLINEFORM0 sentences is sufficient to surpass the results of the INLINEFORM1 sentence-level trained classifier, and more unannotated data further improves the results.
Our method requires a sentence level classifier INLINEFORM0 to label both the target-task corpus and the unlabeled corpus. How does the quality of this classifier affect the overall XR training? We vary the amount of supervision used to train INLINEFORM1 from 0 sentences (assigning the same label to all sentences), to 100, 1000, 5000 and 10000 sentences. We again measure macro-F1 on the entire SemEval 2016 corpus.
The results in Figure FIGREF54 b show that when using the prior distributions of aspects (0), the model struggles to learn from this signal, it learns mostly to predict the majority class, and hence reaches very low F1 scores of 35.28. The more data given to the sentence level classifier, the better the potential results will be when training with our method using the classifier labels, with a classifiers trained on 100,1000,5000 and 10000 labeled sentences, we get a F1 scores of 53.81, 58.84, 61.81, 65.58 respectively. Improvements in the source task classifier's quality clearly contribute to the target task accuracy.
The Stochastic Batched XR algorithm (Algorithm SECREF12 ) samples a batch of INLINEFORM0 examples at each step to estimate the posterior label distribution used in the loss computation. How does the size of INLINEFORM1 affect the results? We use INLINEFORM2 fragments in our main experiments, but smaller values of INLINEFORM3 reduce GPU memory load and may train better in practice. We tested our method with varying values of INLINEFORM4 on a sample of INLINEFORM5 , using batches that are composed of fragments of 5, 25, 100, 450, 1000 and 4500 sentences. The results are shown in Figure FIGREF54 c. Setting INLINEFORM6 result in low scores. Setting INLINEFORM7 yields better F1 score but with high variance across runs. For INLINEFORM8 fragments the results begin to stabilize, we also see a slight decrease in F1-scores with larger batch sizes. We attribute this drop despite having better estimation of the gradients to the general trend of larger batch sizes being harder to train with stochastic gradient methods.
The XR training can be performed also over pre-trained representations. We experiment with two pre-training methods: (1) pre-training by training the BiLSTM model to predict the noisy sentence-level predictions. (2) Using the pre-trained Bert representation BIBREF9 . For (1), we compare the effect of pre-train on unlabeled corpora of sizes of INLINEFORM0 , INLINEFORM1 and INLINEFORM2 sentences. Results in Figure FIGREF54 d show that this form of pre-training is effective for smaller unlabeled corpora but evens out for larger ones.
For the Bert experiments, we experiment with the Bert-base model with INLINEFORM1 sets, 30 epochs for XR training or sentence level fine-tuning and 15 epochs for aspect based fine-tuning, on each training method we evaluated the model on the dev set after each epoch and the best model was chosen. We compare the following setups:
-Bert INLINEFORM0 Aspect Based Finetuning: pretrained bert model finetuned to the aspect based task.
-Bert INLINEFORM0 : A pretrained bert model finetuned to the sentence level task on the INLINEFORM1 sentences, and tested by predicting fragment-level sentiment.
-Bert INLINEFORM0 INLINEFORM1 INLINEFORM2 Aspect Based Finetuning: pretrained bert model finetuned to the sentence level task, and finetuned again to the aspect based one.
-Bert INLINEFORM0 XR: pretrained bert model followed by XR training using our method.
-Bert INLINEFORM0 XR INLINEFORM1 Aspect Based Finetuning: pretrained bert followed by XR training and then fine-tuned to the aspect level task.
The results are presented in Table TABREF55 . As before, aspect-based fine-tuning is beneficial for both SemEval-16 and SemEval-15. Training a BiLSTM with XR surpasses pre-trained bert models and using XR training on top of the pre-trained Bert models substantially increases the results even further.
We presented a transfer learning method based on expectation regularization (XR), and demonstrated its effectiveness for training aspect-based sentiment classifiers using sentence-level supervision. The method achieves state-of-the-art results for the task, and is also effective for improving on top of a strong pre-trained Bert model. The proposed method provides an additional data-efficient tool in the modeling arsenal, which can be applied on its own or together with another training method, in situations where there is a conditional relations between the labels of a source task for which we have supervision, and a target task for which we don't.
While we demonstrated the approach on the sentiment domain, the required conditional dependence between task labels is present in many situations. Other possible application of the method includes training language identification of tweets given geo-location supervision (knowing the geographical region gives a prior on languages spoken), training predictors for renal failure from textual medical records given classifier for diabetes (there is a strong correlation between the two conditions), training a political affiliation classifier from social media tweets based on age-group classifiers, zip-code information, or social-status classifiers (there are known correlations between all of these to political affiliation), training hate-speech detection based on emotion detection, and so on.
The work was supported in part by The Israeli Science Foundation (grant number 1555/15).

Figure and table captions:
Figure 1: Illustration of the algorithm. Cs is applied to Du resulting in ỹ for each sentence, Uj is built according with the fragments of the same labelled sentences, the probabilities for each fragment in Uj are summed and normalized, the XR loss in equation (4) is calculated and the network is updated.
Figure 2: Illustration of the decomposition procedure, when given a1=“duck confit“ and a2= “foie gras terrine with figs“ as the pivot phrases.
Table 1: Average accuracies and Macro-F1 scores over five runs with random initialization along with their standard deviations. Bold: best results or within std of them. ∗ indicates that the method’s result is significantly better than all baseline methods, † indicates that the method’s result is significantly better than all baselines methods that use the aspect-based data only, with p < 0.05 according to a one-tailed unpaired t-test. The data annotations S, N and A indicate training with Sentence-level, Noisy sentence-level and Aspect-level data respectively. Numbers for TDLSTM+Att,ATAE-LSTM,MM,RAM and LSTM+SynATT+TarRep are from (He et al., 2018a). Numbers for Semisupervised are from (He et al., 2018b).
Figure 3: Macro-F1 scores for the entire SemEval-2016 dataset of the different analyses. (a) the contribution of unlabeled data. (b) the effect of sentence classifier quality. (c) the effect of k. (d) the effect of sentence-level pretraining vs. corpus size.
Table 2: BERT pre-training: average accuracies and Macro-F1 scores from five runs and their stdev. ∗ indicates that the method’s result is significantly better than all baseline methods, † indicates that the method’s result is significantly better than all non XR baseline methods, with p < 0.05 according to a one-tailed unpaired t-test. The data annotations S, N and A indicate training with Sentence-level, Noisy sentence-level and Aspect-level data respectively.

Questions about this paper:
How accurate is the aspect based sentiment classifier trained only using the XR loss?

Answers:
BiLSTM-XR-Dev Estimation accuracy is 83.31 for SemEval-15 and 87.68 for SemEval-16.
BiLSTM-XR accuracy is 83.31 for SemEval-15 and 88.12 for SemEval-16.