Introduction
In this paper, we explore the effectiveness of methods designed to improve sentiment classification (positive vs. negative) of sentences that contain complex syntactic structures. While simple bag-of-words or lexicon-based methods BIBREF1 , BIBREF2 , BIBREF3 achieve good performance on this task, they are unequipped to deal with syntactic structures that affect sentiment, such as contrastive conjunctions (i.e., sentences of the form “A-but-B”) or negations. Neural models that explicitly encode word order BIBREF4 , syntax BIBREF5 , BIBREF6 and semantic features BIBREF7 have been proposed with the aim of improving performance on these more complicated sentences. Recently, hu2016harnessing incorporate logical rules into a neural model and show that these rules increase the model's accuracy on sentences containing contrastive conjunctions, while PetersELMo2018 demonstrate increased overall accuracy on sentiment analysis by initializing a model with representations from a language model trained on millions of sentences.
In this work, we carry out an in-depth study of the effectiveness of the techniques in hu2016harnessing and PetersELMo2018 for sentiment classification of complex sentences. Part of our contribution is to identify an important gap in the methodology used in hu2016harnessing for performance measurement, which is addressed by averaging the experiments over several executions. With the averaging in place, we obtain three key findings: (1) the improvements in hu2016harnessing can almost entirely be attributed to just one of their two proposed mechanisms and are also less pronounced than previously reported; (2) contextualized word embeddings BIBREF0 incorporate the “A-but-B” rules more effectively without explicitly programming for them; and (3) an analysis using crowdsourcing reveals a bigger picture where the errors in the automated systems have a striking correlation with the inherent sentiment-ambiguity in the data.
Logic Rules in Sentiment Classification
Here we briefly review background from hu2016harnessing to provide a foundation for our reanalysis in the next section. We focus on a logic rule for sentences containing an “A-but-B” structure (the only rule for which hu2016harnessing provide experimental results). Intuitively, the logic rule for such sentences is that the sentiment associated with the whole sentence should be the same as the sentiment associated with phrase “B”.
More formally, let $p_\theta (y|x)$ denote the probability assigned to the label $y\in \lbrace +,-\rbrace $ for an input $x$ by the baseline model using parameters $\theta $ . A logic rule is (softly) encoded as a variable $r_\theta (x,y)\in [0,1]$ indicating how well labeling $x$ with $y$ satisfies the rule. For the case of A-but-B sentences, $r_\theta (x,y)=p_\theta (y|B)$ if $x$ has the structure A-but-B (and 1 otherwise). Next, we discuss the two techniques from hu2016harnessing for incorporating rules into models: projection, which directly alters a trained model, and distillation, which progressively adjusts the loss function during training.
A Reanalysis
In this section we reanalyze the effectiveness of the techniques of hu2016harnessing and find that most of the performance gain is due to projection and not knowledge distillation. The discrepancy with the original analysis can be attributed to the relatively small dataset and the resulting variance across random initializations. We start by analyzing the baseline CNN by kim2014convolutional to point out the need for an averaged analysis.
Importance of Averaging
We run the baseline CNN by kim2014convolutional across 100 random seeds, training on sentence-level labels. We observe a large amount of variation from run-to-run, which is unsurprising given the small dataset size. The inset density plot in [fig:variation]Figure fig:variation shows the range of accuracies (83.47 to 87.20) along with 25, 50 and 75 percentiles. The figure also shows how the variance persists even after the average converges: the accuracies of 100 models trained for 20 epochs each are plotted in gray, and their average is shown in red.
We conclude that, to be reproducible, only averaged accuracies should be reported in this task and dataset. This mirrors the conclusion from a detailed analysis by reimers2017reporting in the context of named entity recognition.
Performance of hu2016harnessing
We carry out an averaged analysis of the publicly available implementation of hu2016harnessing. Our analysis reveals that the reported performance of their two mechanisms (projection and distillation) is in fact affected by the high variability across random seeds. Our more robust averaged analysis yields a somewhat different conclusion of their effectiveness.
In [fig:hu-performance]Figure fig:hu-performance, the first two columns show the reported accuracies in hu2016harnessing for models trained with and without distillation (corresponding to using values $\pi =1$ and $\pi =0.95^t$ in the $t^\text{th}$ epoch, respectively). The two rows show the results for models with and without a final projection into the rule-regularized space. We keep our hyper-parameters identical to hu2016harnessing.
The baseline system (no-project, no-distill) is identical to the system of kim2014convolutional. All the systems are trained on the phrase-level SST2 dataset with early stopping on the development set. The number inside each arrow indicates the improvement in accuracy by adding either the projection or the distillation component to the training algorithm. Note that the reported figures suggest that while both components help in improving accuracy, the distillation component is much more helpful than the projection component.
The next two columns, which show the results of repeating the above analysis after averaging over 100 random seeds, contradict this claim. The averaged figures show lower overall accuracy increases, and, more importantly, they attribute these improvements almost entirely to the projection component rather than the distillation component. To confirm this result, we repeat our averaged analysis restricted to only “A-but-B” sentences targeted by the rule (shown in the last two columns). We again observe that the effect of projection is pronounced, while distillation offers little or no advantage in comparison.
Contextualized Word Embeddings
Traditional context-independent word embeddings like word2vec BIBREF8 or GloVe BIBREF9 are fixed vectors for every word in the vocabulary. In contrast, contextualized embeddings are dynamic representations, dependent on the current context of the word. We hypothesize that contextualized word embeddings might inherently capture these logic rules due to increasing the effective context size for the CNN layer in kim2014convolutional. Following the recent success of ELMo BIBREF0 in sentiment analysis, we utilize the TensorFlow Hub implementation of ELMo and feed these contextualized embeddings into our CNN model. We fine-tune the ELMo LSTM weights along with the CNN weights on the downstream CNN task. As in [sec:hu]Section sec:hu, we check performance with and without the final projection into the rule-regularized space.
We present our results in [tab:elmo]Table tab:elmo.
Switching to ELMo word embeddings improves performance by 2.9 percentage points on an average, corresponding to about 53 test sentences. Of these, about 32 sentences (60% of the improvement) correspond to A-but-B and negation style sentences, which is substantial when considering that only 24.5% of test sentences include these discourse relations ([tab:sst2]Table tab:sst2). As further evidence that ELMo helps on these specific constructions, the non-ELMo baseline model (no-project, no-distill) gets 255 sentences wrong in the test corpus on average, only 89 (34.8%) of which are A-but-B style or negations.
Crowdsourced Experiments
We conduct a crowdsourced analysis that reveals that SST2 data has significant levels of ambiguity even for human labelers. We discover that ELMo's performance improvements over the baseline are robust across varying levels of ambiguity, whereas the advantage of hu2016harnessing is reversed in sentences of low ambiguity (restricting to A-but-B style sentences).
Our crowdsourced experiment was conducted on Figure Eight. Nine workers scored the sentiment of each A-but-B and negation sentence in the test SST2 split as 0 (negative), 0.5 (neutral) or 1 (positive). (SST originally had three crowdworkers choose a sentiment rating from 1 to 25 for every phrase.) More details regarding the crowd experiment's parameters have been provided in [appendix:appcrowd]Appendix appendix:appcrowd.
We average the scores across all users for each sentence. Sentences with a score in the range $(x, 1]$ are marked as positive (where $x\in [0.5,1)$ ), sentences in $[0, 1-x)$ marked as negative, and sentences in $[1-x, x]$ are marked as neutral. For instance, “flat , but with a revelatory performance by michelle williams” (score=0.56) is neutral when $x=0.6$ . We present statistics of our dataset in [tab:crowdall]Table tab:crowdall. Inter-annotator agreement was computed using Fleiss' Kappa ( $\kappa $ ). As expected, inter-annotator agreement is higher for higher thresholds (less ambiguous sentences). According to landis1977measurement, $\kappa \in (0.2, 0.4]$ corresponds to “fair agreement”, whereas $\kappa \in (0.4, 0.6]$ corresponds to “moderate agreement”.
We next compute the accuracy of our model for each threshold by removing the corresponding neutral sentences. Higher thresholds correspond to sets of less ambiguous sentences. [tab:crowdall]Table tab:crowdall shows that ELMo's performance gains in [tab:elmo]Table tab:elmo extends across all thresholds. In [fig:crowd]Figure fig:crowd we compare all the models on the A-but-B sentences in this set. Across all thresholds, we notice trends similar to previous sections: 1) ELMo performs the best among all models on A-but-B style sentences, and projection results in only a slight improvement; 2) models in hu2016harnessing (with and without distillation) benefit considerably from projection; but 3) distillation offers little improvement (with or without projection). Also, as the ambiguity threshold increases, we see decreasing gains from projection on all models. In fact, beyond the 0.85 threshold, projection degrades the average performance, indicating that projection is useful for more ambiguous sentences.
Conclusion
We present an analysis comparing techniques for incorporating logic rules into sentiment classification systems. Our analysis included a meta-study highlighting the issue of stochasticity in performance across runs and the inherent ambiguity in the sentiment classification task itself, which was tackled using an averaged analysis and a crowdsourced experiment identifying ambiguous sentences. We present evidence that a recently proposed contextualized word embedding model (ELMo) BIBREF0 implicitly learns logic rules for sentiment classification of complex sentences like A-but-B sentences. Future work includes a fine-grained quantitative study of ELMo word vectors for logically complex sentences along the lines of peters2018dissecting.
Crowdsourcing Details
Crowd workers residing in five English-speaking countries (United States, United Kingdom, New Zealand, Australia and Canada) were hired. Each crowd worker had a Level 2 or higher rating on Figure Eight, which corresponds to a “group of more experienced, higher accuracy contributors”. Each contributor had to pass a test questionnaire to be eligible to take part in the experiment. Test questions were also hidden throughout the task and untrusted contributions were removed from the final dataset. For greater quality control, an upper limit of 75 judgments per contributor was enforced.
Crowd workers were paid a total of $1 for 50 judgments. An internal unpaid workforce (including the first and second author of the paper) of 7 contributors was used to speed up data collection.

Question: What logic rules can be learned using ELMo?
Answer: 1).But   2).Eng  3). A-But-B

A-but-B and negation

Question: Does Elmo learn all possible logic rules?
Answer: Yes

No