Text Sentiment Analysis in NLP. Problems, use-cases, and methods: from... | by Arun Jagota | Towards Data Science

People like expressing sentiment. Happy or unhappy. Like or dislike. Praise or complain. Good or bad. That is, positive or negative.
Sentiment analysis in NLP is about deciphering such sentiment from text. Is it positive, negative, both, or neither? If there is sentiment, which objects in the text the sentiment is referring to and the actual sentiment phrase such as poor, blurry, inexpensive, ... (Not just positive or negative.) This is also called aspect-based analysis [1].
As a technique, sentiment analysis is both interesting and useful.
First, to the interesting part. It’s not always easy to tell, at least not for a computer algorithm, whether a text’s sentiment is positive, negative, both, or neither. The cues can be subtle. Overall sentiment aside, it’s even harder to tell which objects in the text are the subject of which sentiment, especially when both positive and negative sentiments are involved.
Next, to the useful part. This is easy to explain. People who sell things want to know about how people feel about these things. It is called customer feedback 😊. Ignoring it is bad for business.
There are other uses as well. Such as opinion mining, i.e. trying to figure out who holds (or held) what opinions. Such as, according to John Smith, the coronavirus will simply go away within six months. This task may be formalized as seeking (source, target, opinion) triples. In our example, source = John Smith, target = coronavirus, opinion = will simply go away within six months.
(Many) Examples
Sentiment analysis is what you might call a long-tail problem. Lots of varying scenarios and subtleties. Such problems are often best described by examples.
First, let’s see some easy positives.
Amazing customer service.Love it.Good price.
Next, some positives and negatives a bit harder to discriminate.
Positives:   What is not to like about this product.   Not bad.   Not an issue.   Not buggy.Negatives:   Not happy.   Not user-friendly.   Not good.Definitely not positive:   Is it any good?
The positives in the above list are not the strongest ones. That said, they are especially good for training ML algorithms to make key distinctions, as we definitely don’t want these positives to be predicted as negatives.
Positives:  Low price.Negatives:  Low quality.
These instances are especially good for training ML algorithms to make key distinctions.
Positives:   Quick turn-around.Negatives:   Quick to fail.
The same point applies here.
Positives:  Inexpensive.Negatives:  cheap.
The same point applies here.
Finally, some negatives which are a bit harder to decipher.
I didn’t get what I was promised.My package hasn’t arrived yet.No one in your team has been able to solve my problem.I was put on hold for an hour!Why is this so hard to use?Why does it fail so easily?
Use Cases
It’s easy to imagine many. Here are some of the main specific ones.
Discover negative reviews of your product or service. On blog posts or eCommerce sites or social media. More broadly anywhere on the web.Aggregate sentiment on financial instruments. Such as specific stocks. What is the recent market sentiment on stock xyz? Also, aspect-based variants. Such as according to analysts at financial company xyz, stock abc is likely to grow 20% in the coming year. Discerning who’s opinion it is provides more information, which may be used to assess credibility or lack thereof.Identify which components of your product or service are people complaining about? Especially strongly. For prioritizing tactical or long-term improvements.Track changes to customer sentiment over time for a specific product or service (or a line of these). To check if things have been getting better ...Track shifting opinions of politicians over time. Individuals or groups such as political parties. News media love to do this. To fuel nagging questions such as you said that then but now this!.
Discover negative reviews of your product or service. On blog posts or eCommerce sites or social media. More broadly anywhere on the web.
Aggregate sentiment on financial instruments. Such as specific stocks. What is the recent market sentiment on stock xyz? Also, aspect-based variants. Such as according to analysts at financial company xyz, stock abc is likely to grow 20% in the coming year. Discerning who’s opinion it is provides more information, which may be used to assess credibility or lack thereof.
Identify which components of your product or service are people complaining about? Especially strongly. For prioritizing tactical or long-term improvements.
Track changes to customer sentiment over time for a specific product or service (or a line of these). To check if things have been getting better ...
Track shifting opinions of politicians over time. Individuals or groups such as political parties. News media love to do this. To fuel nagging questions such as you said that then but now this!.
Computational Problems
What we’ve discussed thus far may be crystallized into two distinct computational problems.
What is the text’s overall sentiment: positive, negative, both, or neither?Which sentiment applies to which portions of the text. This is also called aspect-based sentiment analysis.
What is the text’s overall sentiment: positive, negative, both, or neither?
Which sentiment applies to which portions of the text. This is also called aspect-based sentiment analysis.
Let’s start with the first problem, which we will call sentiment classification.
Sentiment Classification Problem
The input is text. The output we seek is whether the sentiment is positive, negative, both or neither. In a variant of this problem, which we will not address here, we are interested in additionally predicting the strengths of the positive and negative sentiments. You can imagine why. xyz phone really sucks is way more negative than I’m a little disappointed with xyz phone.
Dictionary-based Approach
The simplest approach is to create two dictionaries, of terms carrying positive and negative sentiment respectively. By term, we mean a word or a phrase. A text is classified as positive or negative based on hits of the terms in the text to these two dictionaries. A text is classified as neutral if it hits neither dictionary. A text is classified as both positive and negative if it hits in both dictionaries.
This approach is worth considering when one wishes to quickly get a somewhat effective sentiment classifier off-the-ground and one doesn’t have a rich-enough data set of text labeled with the sentiment. Simplicity is one reason. The more important reason is that the machine learning alternative has its own obstacles to be overcome. We’ll delve into these in detail when we discuss that topic.
Machine-learning obstacles notwithstanding, a dictionary-based approach will run into quality issues sooner or later. So if high precision and high recall of the various sentiment classes are important in your use case, you should consider biting the bullet upfront and investing in ML. Your task will become much easier if you can find a rich-enough labeled data set or come up with some creative ways to get one, possibly after some additional lightweight NLP (discussed in an upcoming section).
Supervised Learning Challenges
The first challenge is the necessity of having a large and diverse data set of texts labeled with their sentiment classes: positive, negative, both, or neither.
The issue is this. Think of the text as being represented by a vector. For now in the usual vector space model, i.e. as a bag of words. That said, the challenge applies, albeit to a somewhat lesser extent, even to word embeddings.
The vector space is huge. Each word in the lexicon has a dimension.
The vast majority of the words in this space carry no sentiment. To train a machine learning classifier would require a huge training set. Much of what it would be doing is learning which words are “nuisance” words. That is, unlearning biases it collected along the way (see example below).
Let’s see an example from which the classifier can learn to wrongly associate neutral words with positive or negative sentiment.
xyz phone sucks → negative
It will learn to associate the word phone with the sentiment negative. Obviously we don’t want this. Unlearning this will require training set instances with the word phone in them that are labeled neither (i.e., neutral).
That being said, breaking up a large and diverse corpus (such as Wikipedia) into sentences and labeling each neutral might alleviate this problem. The intuition here is this. All words will initially learn to be neutral. Words such as sucks that repeatedly occur in text labeled negative will eventually ‘escape’ from their neutral label.
Beyond Bag-of-words Features
From the labeled examples we saw in an earlier section, it seems that a ‘?’ is a predictor of sentiment. This makes sense intuitively. Skeptics ask questions. Not true believers.
Leveraging Dictionaries as Features
If we already have dictionaries of phrases correlated with positive or negative sentiment (or find them easy to construct), why not use them as additional features. They don’t have to be complete. Just curated. So we can take advantage of their quality.
In more detail, here’s how. Say not good is in the dictionary of negatives. We would create a boolean feature for this entry. This feature’s value is 1 if not good appears in text and 0 if not. We might also add the entry (not good, negative) to our training set. Note that here we are thinking of not good as the full text.
Use Part-of-speech?
Sentiment-rich words are often adjectives. This makes one wonder whether using information about the part-of-speech of each word in the text might be useful? As additional features or for pruning features.
Let’s start by looking at the parts-of-speech of the words in our various examples. This analysis was done using the online pos-tagger at [2].
What thoughts does this trigger? The POS-tag adjective seems significantly correlated with sentiment polarity (positive or negative). The POS-tag adverb also. Determiners, prepositions, and pronouns seem to predict the neutral class.
How might we take advantage of this? We could gate bag-of-words features on their parts-of-speech. For example, filter out all words whose POS-tag is determiner, preposition, or pronoun. This may be viewed as an elaborate form of stop-words removal.
Feature Engineering: Some Observations
Whereas these observations are general, they especially apply to our problem (sentiment classification).
First, we don’t need strong evidence before we add a new feature. Merely a weak belief that it might help. The machine learning algorithm will figure out how predictive this feature is, possibly in conjunction with other features. As the training set gets richer over time, the ML will automatically learn to use this feature more effectively if this is possible. Weak features can add up.
The only downside to this is that if we go overboard, i.e. add too many features, the feature space explosion may come back to haunt us. More on that later.
Let’s expand on “weak belief that it might help”. Here, ‘help’ just means that the feature is predictive of some sentiment class. We don’t need to know which. The ML will figure this out. That is, which feature value predicts which sentiment class. By contrast, when setting up a rule-based system (of which dictionaries are a special case) one has to specify which combinations of feature values predict which sentiment class.
Does This Risk Feature Space Explosion?
We have already accepted that using bag-of-words features will explode our feature space. For reasons discussed earlier, we have decided to bite the bullet on this front. The question is, will the additional features mentioned in this section make the matter worse?
Actually they will make it better. Let’s reason through this. First the question-mark feature. It is boolean-valued. No explosion here. Next, the dictionary-based features. These in fact reduce the noise in the space of word vectors as they surface sentiment-rich words and phrases. Finally, the part-of-speech features. Using them as suggested, for filtering (i.e. removing words), prunes the feature space.
Word k-gram Features?
We deliberately put this after the previous section because this does run a greater risk of exploding the feature space if not done right. The space of word k-grams even with k = 2 is huge. That said, pruning this space sensibly can potentially increase the benefit-to-cost ratio from these features.
Below are some plausible ideas to consider. In the discussion, we limit ourselves to k=2, i.e. to bigrams, although it applies more generally.
Prune away bigrams from the model that don’t have sufficient support in the training set. (By the support of a bigram we mean the number of times it occurs in the training set.)For additional pruning, consider parts-of-speech as well.
Prune away bigrams from the model that don’t have sufficient support in the training set. (By the support of a bigram we mean the number of times it occurs in the training set.)
For additional pruning, consider parts-of-speech as well.
Taking Stock ...
We’ll close this section by taking stock of what we have discussed here and its implications. First, we see that the ML approach can be empowered with a variety of features. We simply throw features into the mix. So long as there is a plausible case for each inclusion. We don’t worry about correlations among features. Too complicated to analyze. Let the ML sort it out. The end justifies the means. So long as we have a rich enough labeled data set which we can partition to train-and-test splits and reliably measure the quality of what we are referring to as ‘end’.
We do need to think about the feature space explosion. We already did.
Now a few words about the learning algorithm. We have lots of choices. Naive Bayes. Logistic Regression. Decision Tree. Random Forest. Gradient Boosting. Maybe even Deep Learning. The key point to bring to the surface is that these choices span varying levels of sophistication. Some can automatically discover multivariate features that are especially predictive of sentiment. The risk here is that many of the multivariate features they discover are also noisy.
Okay so now we have lots of feature choices and lots of learning algorithm choices. Potentially very powerful. But also risky. As mentioned earlier, we can mitigate the risk by keeping in mind the feature-space explosion.
Ultimately though we should focus on building as rich of a labeled data set, even if only incrementally. Longer-term this has more value than tactically optimizing features to compensate for not having a great training set. This is the single most important aspect of this problem. Invest in this.
Target Variants
To this point, we’ve been thinking of sentiment classification as a 4-class problem: positive, negative, both, neither. In some settings, the class both can be ignored. In such settings, we interpret neither as neutral.
In most use cases, we only care about the first two. So neutral is a nuisance class. ‘Nuisance’ means it needs to be accounted for, even though it’s not what we seek. Why does it need to be accounted for? Well, we don’t want text that is neutral to get classified as positive or negative. Said another way, including the neutral class (backed by a sufficiently rich training set for it), improves the precision of the positives and negatives.
This is easy to illustrate with an example. Remember the instance
xyz phone sucks → negative
We wouldn’t want the inference phone → sucks. Meaning that every phone sucks. By adding the neutral class, along with a suitably rich training set for it, the risk of this type of unwarranted inference reduces greatly.
Probabilistic Classification
Regardless of which learning algorithm we end up choosing — Naive Bayes, Logistic Regression, Decision Tree, Random Forest, Gradient Boosting, ... — we should consider leveraging the predicted probabilities of the various classes. For example, if the predicted probabilities on an input are roughly 50% (positive), 50% (negative), 0% (0) then we can interpret the text as having both positive and negative sentiments.
How to build a training set efficiently
Okay, so it’s clear that the ML approach is powerful. Let’s now look to “feeding the beast”, i.e. building a rich training set.
Here’s an idea of how to quickly assemble a large set of texts that can be manually labeled efficiently.
Pick a suitable source of unstructured text. Such as product reviews at an e-commerce site.Create two columns in a spreadsheet, one for text, one for label.Put each document (e.g. each product review) in its own cell in the column labeled text.Manually add the labels.
Pick a suitable source of unstructured text. Such as product reviews at an e-commerce site.
Create two columns in a spreadsheet, one for text, one for label.
Put each document (e.g. each product review) in its own cell in the column labeled text.
Manually add the labels.
Let’s elaborate on step 4. Consider crowd-sourcing it. Or at least dividing up the work among team members. Plus adopt a convention that an empty cell in the label column denotes a specific label. A good choice is neither, i.e. neutral.
You might be surprised at how quickly you can build up a rich training set using this process.
If your product reviews data set comes with a star-rating attached to each review, you can use this rating to auto-label the positive and negative instances. This can speed up the labeling process. That said, you should make a manual pass after the auto-labeling to review it and correct those labels that are wrong.
The assumption underlying this auto-labeling is that its quality is reasonably good. So that only a small proportion of the labels need fixing. You do have to look at them all. Still, visually scanning all labels has a much higher throughput than editing individual ones.
Training Instance Granularity
Generally speaking, to the extent possible, input instances should be more granular than coarser. Customer product reviews are generally granular enough. Especially if they are already tagged with the ratings, from which we might auto-derive the sentiment target. In this case, breaking longer reviews down to individual sentences and manually tagging them with an appropriate sentiment label might be too much work, whereas its benefit unclear.
Next, consider starting points being longer documents. Such as full-length review articles of product classes. For example, The Best 10 Phones for 2020 or The Best 10 Stocks for 2020.
The case for breaking these down into finer granularity such as paragraphs or even sentences is stronger. Clearly, if we can restrict the text to the region to which a specific sentiment is applicable, it can help improve the learning algorithm’s accuracy.
As an extreme example, say you have a 20-page document, all of it neutral, except one sentence which has a strong sentiment. It makes sense to label this sentence with the sentiment and the rest of the text as neutral. That’ll likely work better than labeling the 20-page document with the sentiment in that one sentence.
Granularity Of Instances In The Field
As discussed above, for the training set, finer-grained instances in the training set are generally better than coarser-grained ones. This preference does not apply to classification time, i.e. the use of the classifier in the field. We should go ahead and predict the sentiment of whatever text we are given, be it a sentence or a chapter.
Unlike during training, there is no downside to predicting the sentiment of a long document. It's just a question of expectations. If a user seeks a sentiment of a document longer than a paragraph, what she really means is she wants the overall general sentiment across the text. Is it positive overall, negative overall, both, or neither (neutral)?
This is fine, sometimes that is what you want. And once you have discovered documents that carry some sentiment, you can always drill down to run the sentiment classifier on their individual sentences or paragraphs.
In view of this, we should keep in mind that evaluation on a test set held-out from the labeled data set will not yield an accurate assessment of how well the classifier works in the field. The held-out test set is derived from the labeled data set, which is composed of granular instances for reasons discussed earlier. The field’s inputs are not necessarily always that granular.
Aspect-based Sentiment Analysis
Here, in addition to deciphering the various sentiments in the text we also seek to figure out which of them applies to what.
Clearly such analysis can be very useful, as illustrated by the example below.
The camera on my <xyz-brand> phone sucks. Apart from that, I’m happy.
You clearly want to know what is being complained about and what is being liked.
Often, we also care to extract the actual sentiment phrases. Consider the example below from a made-up holistic review of a new TV.
Good price. Sharp image. Vivid colors. Static in Audio. Motion lags a bit.
Ideally, we’d like to extract (aspect, sentiment-phrase, polarity) triples from it.
Aspect:           price    image   colors   audio   motionSentiment-phrase: good     sharp   vivid    static  lags a bitPolarity:          +         +       +        —        -
The polarities may help derive an overall quality score (e.g., here 3 out of 5). May have other uses as well.
Extracting Aspects And Sentiment Phrases
Let’s run this text through the POS-tagger at [2].
Recall that the POS-tag legend is
What jumps out at you? As a first attempt, splitting the text into sentences, running a POS-tagger on each sentence, and if the tag sequence is
deeming adjective to be the sentiment-phrase and noun to be the aspect works surprisingly well. In precision terms, that is. Not recall because this pattern is too-specific. For example, it doesn’t detect the aspect-sentiment phrase in Motion lags a bit.
So how can we try to extend the idea of the previous paragraph to try to improve recall? Formulate this as a sequence labeling problem. See [3] for a detailed sequence-labeling formulation of a similar problem, named entity recognition.
The text is tokenized as a sequence of words. Associated with this sequence is a label sequence, which indicates what is the aspect and what the sentiment-phrase.
Below is our earlier example, reformulated in this convention, with A denoting aspect, S denoting sentiment-phrase, and N denoting neither. We’ve split the pair into two as it won’t fit in a horizontal line.
words   Good price. Sharp image. Vivid colors. Static in Audio.labels   S    A      S     A      S     A        S    N    A words   Motion lags a bit.labels    A     S   S  S
In [3] we focused on Hidden Markov models for sequence labeling. Here, it is more natural to work with conditional Markov models [4], for reasons we explain below.
First, what is a conditional Markov model? Recall that our inference problem is to input a sequence of words and find the most likely sequence of labels for it. For the token sequence [Motion, lags, a, bit] we would expect the best label sequence to be [A, S, S, S].
A conditional Markov model (CMM) models this inference problem as one of finding the label sequence L that maximizes the conditional probability P(L|T) for the given token sequence T. The Markov model makes certain assumptions which make this inference problem tractable. Specifically, P(L|T) is assumed to be factorable as
P(L|T) = P(L1|L0,T1)*P(L2|L1,T2)*...*P(Ln|L_{n-1},Tn)
Rather than explain it, let’s illustrate it with our example. We have added a label B denoting begin.
P( [B,A,S,S,S] | [B, Motion, lags, a, bit] ) = P(A|B, Motion)*P(S|A, lags)*P(S|S, a)*P(S|S, bit)
We won’t describe the inference algorithm. It is too complex for this post. Besides, this is not our focus. However, we will explain the individual probabilities in the above example qualitatively. Equipped with such an explanation, we can imagine trying out all possible label sequences, computing the probability of each, and finding the one that has the highest probability.
Let’s start with P(A|B, Motion). This is influenced by two factors and their interaction. First, the likelihood that the first word is part of the aspect. Second, the likelihood that Motion is an aspect word.
The first factor’s likelihood is significantly greater than 0. We can imagine many real examples in which the first word is an aspect word. Such as camera is low-resolution.
It is the second factor’s likelihood that we’d like to dwell more on. Consider P(A|Motion), ignoring the influence of the previous state B. The CMM allows us to model this probability as being influenced by any features of our choice derived from the combination of A and Motion. Possibly overlapping. The HMM, by contrast, would work in terms of P(Motion|A) instead. It would treat Motion and A as symbols, not letting us exploit any features we may deem useful.
In effect, we can think of P(A|Motion) as a supervised learning problem in which (A, Motion) is the input and P(A|Motion) the output. The power of this approach lies in its ability to learn complex mappings P(Li|Ti) in which we can use whatever features from the pair (Li, Ti) that we deem fit.
Two features especially come to mind. The word’s part-of-speech and whether the word is labeled as being in a recognized named entity. (See [3] which covers named entity recognition in NLP with many real-world use cases and methods.)
The part-of-speech feature has already been suggested by the examples we saw, in which the POS-tag noun seemed a predictor of the label aspect and adjective a predictor of sentiment-phrase. The named entity feature is motivated by the intuition that aspects are often objects of specific types. For instance, retail products. A NER that can recognize retail products and associated product features can be very useful to pick these out as aspects from sentiment-laden reviews.
Typically we set up NER to recognize fine-grained entities. Such as product names. Not noun phrases. While in principle we could, noun phrases are too varied to model as NER. POS-tag is coarser-grained. In view of this, we can think of the benefit of combining the two features as follows. NER gives us precision. The POS feature helps with recall.