# Consumer Satisfaction Rating System Using Sentiment Analysis

Kumar Gaurav, Prabhat Kumar

► To cite this version:

Kumar Gaurav, Prabhat Kumar. Consumer Satisfaction Rating System Using Sentiment Analysis. 16th Conference on e-Business, e-Services and e-Society (I3E), Nov 2017, Delhi, India. pp.400-411, 10.1007/978-3-319-68557-1_35 . hal-01768521

HAL Id: hal-01768521

https://inria.hal.science/hal-01768521

Submitted on 17 Apr 2018

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Consumer Satisfaction Rating System using Sentiment Analysis

Kumar Gaurav, Prabhat Kumar

Department of Computer Science and Engineering,
National Institute of Technology Patna, India
kumargaurav.nitp@gmail.com
prabhat@nitp.ac.in

**Abstract.** Owing to the inclination towards e-Commerce, the importance of consumer reviews has evolved significantly. Potential consumers exhibit sincere intents in seeking opinions of other consumers who have already had a usage experience of the products they are intending to make a purchase decision on. The underlying businesses also deem it fit to ascertain common public opinions regarding the quality of their products as well as services. However, the consumer reviews have bulked over time to such an extent that it has become a highly challenging task to read all the reviews, even if limiting to only the top ones, to reach an informed purchase decision or have an insight regarding how satisfied or not the consumers of a particular product are. Since most of the reviews are either unstructured or semi-structured, information classification is employed to derive knowledge from the reviews. However, most classification methods based on sentiment orientation of reviews do not detect which features of a product were specifically liked or disliked by a reviewer. This constitutes itself into another problem since most consumers are on the lookout for certain prerequisite features while viewing the products. The paper proposes a Consumer Satisfaction Rating System (CSRS) based on sentiment analysis of consumer reviews in context of the features of a product. The system aims at providing a summary that represents the extent to which the consumers were satisfied or unsatisfied with the specific features of a product.

**Keywords:** e-Commerce, Online Reviews, Product Features, Sentiment Analysis, Opinion Mining

## 1 Introduction

The Internet has paved several alternate ways for tasks and transactions which could be conducted only through the levels of physical contact or communication in the yesteryears. With the recent advances in e-Commerce, there has been a catastrophic surge in the number of people conducting their commercial transactions over the Internet. They shop, book, rent, sell, etc. on various e-Commerce platforms that have taken the reigns of marketing from their offline counterparts. However, contrary to
traditional shopping in which consumers purchase from physical stores and gain an
actual experience of the product such as how a cloth material actually feels especially
in cases when they are looking for good quality fabric, online marketing is received
with much skepticism since there is absence of any such tangible or physical contact
during initial product showcasing. One cannot have an insight into how a product
actually feels or performs like unless one receives it after waiting for a requisite peri-
od of time post placing the order on an online shopping platform. The same stands
true for services such as booking a hotel online or buying tickets for a bus ride. And
this is when consultation from reviews is sought. The most cardinal objective of the
reviews is to bridge the gap between product experience and potential consumers [1].
While surfing through the product details listed on the online e-Commerce websites,
the one prominent element which the consumers are mostly looking for is assurance
in terms of product quality and usage experience indicial of consumer satisfaction.
Consumers tend to read reviews posted by other consumers who have had an experi-
ence of the product or have something to share regarding it. These attempts are pri-
marily focused on gaining insight on product and its features and whether the product
actually succeeded in delivering the level of performance it claimed. The businesses
underlying e-Commerce platform also consider such reviews as an effective infor-
mation source for ascertaining whether the consumers of its products or services are
actually satisfied with them or not. However, maneuvering through the large quanti-
ties of information provided in the product reviews poses as a very tricky as well as a
tiresome task for consumers seeking to make an informed purchase decision. Fig. 1
depicts the huge bulk of consumer reviews that are available on an e-Commerce web-
site such as Amazon [2]. It can be further inferred from the aforementioned figure that
even if the consumers narrow their review-reading to encompass only the top reviews,
they would still have to read a considerably large number of reviews before deducing
information catalytic in reaching to their decision regarding the purchase of that prod-
uct or service.

It is therefore a necessary improvisation to seek ways through which information
provided in the consumer reviews can be classified or the most vital elements of a
review that help in the decision making can be extracted and effectively represented
in the form of a summary. Since most of the reviews are generally in the form of un-
structured or semi-structured text, it becomes but a necessity to involve Natural Lan-
guage Processing techniques to extract usable information.

Various machine learning algorithms such as Naïve Bayes, Decision Tree and
Support Vector Machines are employed along with the Natural Language Processing
techniques to detect the sentimental orientation of the reviews and thereby provide an
overview depicting the consumers liking or disliking the product or service [3]. How-
ever, most of such review summarization methods reflect the fraction of consumers
which have liked or not liked the product as a whole. They do not delve into answer-
ing questions such as what were those salient features which the consumers particu-
larly liked or disliked in the product.
<figure><img src="image_2.png" /><figcaption>Fig. 1. Availability of Consumer Reviews in Bulk</figcaption></figure>

Detection of sentiment orientation of reviews is conducted on document level, sentence level and feature/aspect level [3]. Most of the research works have considered searching opinions on the former two levels [4][5]. Less focus has been laid on the feature/aspect level of sentiment analysis. This paper proposes a Consumer Satisfaction Rating System based on sentiment analysis of consumer reviews in context of the aspects/features of a product or a service. The objective is to mine opinions of consumers from their reviews in regard of the features of a product or a service. This will help the consumers as well as the manufacturers in conceiving which features or aspects of a product or a service were particularly liked or disliked. It reflects how many consumers were satisfied with which particular features of the product or service.

The paper has been organized in the following manner. Section 2 provides an overview of the related work in this domain. Section 3 explains the proposed work. Section 4 sheds light on the results obtained and related analysis. Lastly, conclusion and future scope have been provided in Section 5.

## 2 Related Work

Sentiment Analysis integrates an intersection of various disciplines of study such as Data Mining, Computational Linguists and Natural Language Processing. As previously mentioned, sentiment analysis can be processed at document, sentence and feature levels [3]. The characteristic feature of sentiment analysis at document level is that, here, the aggregate opinion orientation of the document, as a whole, is considered for classifying it. This is different from conducting sentiment analysis at sentence level where the sentimental orientation of each sentence is considered for its classification into being negative, neutral or positive. Further, aspect/feature level sentiment analysis incorporates determination of opinion orientation regarding the aspects/features of the product [4].
Most of the previous research works based on detecting sentiment orientation of
consumer reviews have concentrated on document and sentence levels. The most
notable contribution which also happens to be the pioneering research work in the
domain of application of sentiment analysis and opinion mining techniques is that of
Hu and Liu [6]. They proposed a technique for conducting summarization of reviews.
Their work was mainly focused on searching the frequent features from the consumer
reviews and detecting opinion orientation in their regard. The works succeeding it
subjected subjective phrases for determining opinion polarity of reviews.

In [7], OPINE, an information extraction system using unsupervised learning, has been proposed by Popescu and Etzioni. It mines the reviews and builds a model of the important features. Further, it checks for how these features are evaluated by the reviewers and also compares their quality with other products. However, OPINE's major focus is limited to finding the features rather than detecting opinion orientation of the reviews corresponding to those features.

Pang et al. [8] attempted to discern whether determination of sentiment and its
classification could be simply categorized into two categories of positive sentiment
and negative sentiment. The work does not extend to consideration of how positive or
negative a sentiment was in its true orientation.

Peñalver-Martinez et al. [9] proposed an improvement in the contemporary method of sentiment analysis. The proposed method involves ontology in selection of features. Further, it uses a sentiment analysis method based on vector analysis. Several prominent works have followed in the recent years [10-14].

It can be concluded from the literature survey that major emphasis has been laid on feature extraction and, therefore, development of a system for reflecting summarized feedback of the consumers regarding the features of a product is a domain in need of further exploration.

# 3 Proposed Work

Several instances have been evident where a customer seeks only for a particular fea-
ture or set of features while making a purchase decision. As of now, a customer has to
read a bulk of reviews in order to ascertain the satisfaction level of the customers who
have submitted their reviews. Even in cases where one is looking only for a selected
feature such as battery life or performance of a mobile phone, one has no alternative
to reading all the reviews related to the aforementioned mobile phone, regardless of
whether the review has anything to mention about that particular feature or not. None
of the popular e-Commerce platforms have introduced any provision for reducing the
efforts invested by the customers while looking to make a purchase decision after
finding the satisfaction levels of the reviewers with a particular feature or feature set.

Our proposed method consists of the following steps, as represented in Fig. 2.
First, the system scrapes all of the customer reviews available online for a particular
product at a specified point of time. It applies various preprocessing techniques on the
customer review sentences and subsequently adds them in a review database created
by the system. Then the system scrapes the product description provided for the given
product. It preprocesses the data and prepares a filtered description table containing the aspects/features of the product. After constructing the review database and the filtered description table, the system searches the reviews for obtaining a match with the aspects/features listed in the filtered description. Once matches are obtained, opinion phrases/words are detected for those matched features. Each opinion phrase/word carries a sentiment polarity value between the range [-1, 1]. The system involves a dictionary-based approach. Therefore, it uses a lexical database for obtaining sentiment polarity scores.

<figure><img src="image_3.png" /><figcaption>Fig. 2. Proposed System Framework</figcaption></figure>

After obtaining a cumulative sentiment score for each aspect/feature listed in the filtered description table, it is utilized to rate the customer satisfaction level for the
specific aspect/feature. Lastly, these ratings are represented along the product description on the e-Commerce webpage.

## 3.1 Data Extraction

The product reviews are extracted from the e-Commerce website using a web scraper. A web scraper can be a crawler or an HTML parser. In either case, the webpages containing the reviews are fetched and parsed in order to extract targeted content. Since the system is primarily focused at obtaining customer satisfaction levels, it scrapes reviewer name and review content. Further, a review database is created and the extracted reviews are saved in it.

## 3.2 Preprocessing and Review Database Creation

The data collected from the e-Commerce website contains noise. Therefore, it becomes important that such unnecessary data be removed. Also, data that do not contribute in the sentiment analysis detection needs to be removed. Preprocessing involves steps related to detection of stop word, stemming and tokenization.

Stemming refers to reduction of a word into its root form. Stop word demarcates the end of a sentence. Tokenization refers conversion of words into tokens. The aforementioned preprocessing methods alleviate convenience in effective implementation of Parts of Speech (POS) tagging [15], discussed in the following section, and sentiment analysis processes. The data obtained after conducting preprocessing is saved in a review database featuring mainly two columns, one containing the name of the reviewers and the other containing the review text.

## 3.3 POS Tagging and Feature Selection

The objective of the proposed work necessitates detection of product features and opinions regarding them. Product features are generally nouns and immediate pronouns and their opinions are contained in adjectives, adverbs or related phrases. Therefore, tagging of the words dominant in the reviews becomes a necessary step in determination of feature opinion pairs. Table 1 represents a Feature Opinion pair.

<table><caption>Table 1. Review Sentence and Feature Opinion Pair</caption><thead><tr><th colspan="2">Review</th></tr></thead><tbody><tr><td>Sentence</td><td>The display is bright.</td></tr><tr><td>Feature Opinion Pair</td><td>(display, bright)</td></tr></tbody></table>

In POS tagging, words are classified and labeled according to their parts of speech [15]. The associated tags have been represented with their parts of speech counterpart in Table 2.
<table><caption>Table 2. POS Tagging</caption><thead><tr><th>Tag</th><th>Description</th><th>Tag</th><th>Description</th></tr></thead><tbody><tr><td>CC</td><td>Coordinating Conjunction</td><td>PRP$</td><td>Possessive Pronoun</td></tr><tr><td>CD</td><td>Cardinal Number</td><td>RB</td><td>Adverb</td></tr><tr><td>DT</td><td>Determiner Adverb</td><td>RBR</td><td>Adverb, Comparative</td></tr><tr><td>EX</td><td>Existential There</td><td>RBS</td><td>Adverb, Superlative</td></tr><tr><td>FW</td><td>Foreign Word</td><td>RP</td><td>Particle</td></tr><tr><td>IN</td><td>Preposition/Subordinating Conjunction</td><td>SYM</td><td>Symbol</td></tr><tr><td>JJ</td><td>Adjective</td><td>TO</td><td>To</td></tr><tr><td>JJR</td><td>Adjective, Comparative</td><td>UH</td><td>Interjection</td></tr><tr><td>JJS</td><td>Adjective, Superlative</td><td>VB</td><td>Verb, Base Form</td></tr><tr><td>LS</td><td>List Item Marker</td><td>VBD</td><td>Verb, Past Tense</td></tr><tr><td>MD</td><td>Modal</td><td>VBG</td><td>Verb, Gerund/Present Participle</td></tr><tr><td>NN</td><td>Noun, Singular/Mass</td><td>VBN</td><td>Verb, Past Participle</td></tr><tr><td>NNS</td><td>Noun, Plural</td><td>VBP</td><td>Verb, Non 3<sup>rd</sup> Person Singular Present</td></tr><tr><td>NNP</td><td>Proper Noun, Singular</td><td>VBZ</td><td>Verb, 3<sup>rd</sup> Person Singular Present</td></tr><tr><td>NNPS</td><td>Proper Noun, Plural</td><td>WDT</td><td>Wh-Determiner</td></tr><tr><td>PDT</td><td>Predeterminer</td><td>WP</td><td>Wh-Pronoun</td></tr><tr><td>POS</td><td>Possessive Ending</td><td>WP$</td><td>Possessive Wh-Pronoun</td></tr><tr><td>PRP</td><td>Personal Pronoun</td><td>WRB</td><td>Wh-Adverb</td></tr></tbody></table>

The POS tagging method is utilized in mining features of product from the product description provided on the webpage of the e-Commerce platform. These features are saved in a separate table post which these features are searched for matches in the review text stored in the review database.

## 3.4 Sentiment Analysis and Representation

Sentiment analysis integrates an intersection of computational linguistics, natural language processing and text analysis with the objective of subjectivity determination and information extraction [3]. It detects the opinion polarity of a natural language element to ascertain whether it reflects a negative, neutral or positive opinion [16]. Regular opinions are elements of expression that exhibit opinion on a particular entity, here, for products or services [4]. As for example, “The processor of this palmtop is sluggish”. They are mostly subjective in nature. On the other hand, opinions involving comparisons include depictions of two or more entities and how they fare against each other on certain parameters. As for example, “X processor is faster than Y processor.” Such opinions can be either objective or subjective. The basic components of an opinion are as follows:
* Opinion holder: It refers to the individual/organization that has an opinion regarding an entity.

* Entity: It refers to the subject regarding which the opinion is.

* Opinion: It is the assumption, conclusion, notion or view of an opinion holder regarding an entity.

As stated in Equation 1, an opinion has been defined as a quintuple or an expression formed through combination of five essential elements:

$$ (e_x, a_{xy}, so_{wxyz}, h_w, t_z) \qquad (1) $$

where

* $e_x$ represents target entity.

* $a_y$ represents aspect of entity $e_x$.

* $so_{wxyz}$ represents sentiment value of the opinion of the opinion holder $h_w$ on aspect $a_{xy}$ of entity $e_x$ at time $t_z$. $so_{wxyz}$ is +ve, -ve, or neutral, or a more granular rating.

* $h_w$ represents opinion holder.

* $t_z$ represents time of opinion expression.

After finding a match for the extracted feature in the review text, its immediate adjective/adverb phrase is considered as the opinion phrase. Since the proposed work employs a dictionary-based approach, we relate the text considered as opinion phrase with the synonym/antonym sets provided in the dictionary. The dictionary stores sets of synonyms and antonyms and values depicting the levels of positivity/negativity. Thus, sentimental orientation of the opinion phrases for the product features are determined as neutral, positive or negative along with their numerical representations.

<table><caption>**Algorithm 1: Proposed Algorithm for CSRS**</caption><thead><tr><th>Input</th><td>List of all reviews (review[]),<br/>List of all features present in description (fil_desc[])</td></tr><tr><th>Output</th><td>List of sentiment values for features</td></tr></thead><tbody><tr><td>Initialization;</td><td></td></tr><tr><td>1.</td><td>Pre-processing and subsequent POS tagging</td></tr><tr><td>2.</td><td>desc_sent[]=[0]*length(fil_desc)</td></tr><tr><td>3.</td><td>for i=1 to length(review)</td></tr><tr><td>4.</td><td>for j=1 to lengthsentences_in_review)</td></tr><tr><td>5.</td><td>for k=1 to length(fil_desc)</td></tr><tr><td>6.</td><td>if fil_desc[k] is in sentences_in_review[j]</td></tr><tr><td>7.</td><td>desc_sent[k]=sentimentsentences_in_review[j])</td></tr></tbody></table>

Now, opinions for each product feature contained in the product feature table, discussed in Section 3.3, have been obtained. The proposed method for developing the
Consumer Satisfaction Rating System has been provided in Algorithm 1. The con-
cluding objective of the proposed work is to represent the results obtained for these
product features in the product description section provided on the webpage of the e-
Commerce portal.

4 Implementation and Results

The experiments were conducted on the Linux platform with a system configuration
of i5-2430M CPU and 3 GB RAM. The proposed system was implemented on Python
for detecting sentiment orientation of opinion phrases associated with specific product
features provided in the product description. Over 53,291 reviews were extracted
from Amazon during the period of October 2016 – January 2017. The reviews be-
longed to the Electronics category of the website and were concentrated around 4
products, namely two smartphones (Moto G4 Plus and Redmi Note 3) and two lap-
tops (Apple Macbook Air and Dell Inspiron 3558 Notebook). Further, a review data-
base was created and the aforementioned reviews were appended after initial prepro-
cessing. The review database consisted of fields containing reviewer's name and re-
view text.

In succession to the aforementioned steps of preprocessing, product features were
extracted from the product description text scraped from the webpage and saved in a
CSV file. The product description text was mined to obtain product features from it
by first tagging the text according to their parts of speech. The concept that product
features are most likely to be nouns and the more detailed ones are likely to be proper
nouns was exploited. For example, in case of a smartphone, processor is a noun
whereas a detail such as it being a Snapdragon is a proper noun. Thus, a final feature
database containing a table of product features mined from the product description
text was obtained. Further, a search was conducted in the review database for a match
with the features dominant in our final feature database.

After obtaining a match, the related opinion phrases were marked and the senti-
ments involved were analyzed using a pattern library. The module PatternAnalyzer
was utilized for pattern based processing. For POS tagging, wherever required,
NLTK's standard TreeBank tagger has been utilized [17]. The aforementioned tagger
requires NumPy [18], another module aiding in scientific processing. Since the pro-
posed method uses a dictionary-based approach for text analysis, we rely on the
synsets and sentiment scores provided in publicly available lexical databases named
WordNet [19] and SentiWordNet [20]. Fig. 3 and 4 represent the results obtained
through sentiment analysis of reviews based on the product features of smartphones
and laptops, respectively.

The most significant aspects of the results obtained are that they are concentrated
around conducting sentiment analysis at the aspect/feature level and the results are
cumulative in nature i.e. the results show that the features which have higher instances
of dominance in the reviews have higher values in comparison to the features which
have been mentioned minimally. Besides being informative for consumers, the results
also aim at benefitting business forerunners since they will be able to acknowledge
which features of their product have been more prominently liked by the consumers.

<figure><img src="image_4.png" /><figcaption>Fig. 3. Sentiment results for features of smartphones</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Fig. 4. Sentiment results for features of laptops</figcaption></figure>

Also, they will also be able to get insights into how their product fares in comparison
to their competitors and that how they have been received by the consumers.
## 4.1 Validation

The accuracy of the proposed system can be evaluated by computing a ratio between the summation of truly positive (TP) and truly negative (TN) sentiment for feature f and the summation of truly positive, falsely positive (FP), truly negative and falsely negative (FN) sentiment for feature f, as shown in Equation 2.

$$ Accuracy(f) = (TP + TN)/(TP + FP + TN + FN) \qquad (2) $$

The accuracy is represented between 0 and 1, where 0 represents minimum accuracy i.e. 0% and 1 represents maximum accuracy i.e. 100%. On an average, an accuracy of 0.908 was obtained for the proposed system.

## 5 Conclusion and Future Work

It can be inferred from the results mentioned in Section 4 that a Consumer Satisfaction Rating System, which detects how satisfied or not the consumers are with respect to the individual features of a product, has been developed. This, indeed, is helpful for consumers who are intending to make decisions regarding purchasing a product based on whether it actually contains that one feature they have been looking for and that it has satisfied its previous users or not. Besides the fact that the proposed work processes sentiment analysis targeted at the specific features of a product, the unique aspect of the work is that the results obtained through it are cumulative and not simply comparative. This means that a few instances of a particular feature such as camera of a smartphone being mentioned as good does not necessitate its association with a higher satisfaction value. If many users are satisfied with a particular feature then that particular feature is given a higher value even if the satisfaction value hadn't been one of the soaring ones individually.

Research efforts in future may be levied at improving mining of features and integrating the application of features provided by the manufacturers on the webpage as well as those mentioned by the consumers in the reviews. Also, research may be extended further to detect deceptive reviews, which are aimed at conducting socioeconomic attacks [21], and eventually nullify their contribution in the average consumer satisfaction rating. Efforts can also be levied at developing a visualization method that would be an improvement in the comprehension levels offered by the contemporary alternatives.

## References

1. Bollen, J., Mao, H.: Twitter Mood Predicts The Stock Market. 2, 1–8 (2011)

2. Amazon, www.amazon.com

3. Pang, B., Lee, L.: Opinion Mining and Sentiment Analysis. Found. Trends® Inf. Retr. 1, 91–231 (2006)
4. Liu, B.: Sentiment Analysis and Opinion Mining. Synthesis Lectures on Human Language Technologies. 5, 1–167 (2012)

5. Srivastava, A., Singh, M.P., Kumar, P.: Supervised Semantic Analysis of Product Reviews Using Weighted k-NN Classifier. In: 11th International Conference on Information Technology: New Generations (ITNG), pp. 502-507. IEEE, Las Vegas (2014)

6. Hu, M., Liu, B.: Mining and summarizing customer reviews. In: ACM SIGKDD International Conference on Knowledge Discovery and Data Mining KDD 04, pp.168-177 (2004)

7. Popescu, A.-M., Etzioni, O.: Extracting Product Features and Opinion from Reviews. Hum. Lang. Technol. Empir. Methods Nat. Lang. Process. Vancouver, Br. Columbia. 339–346 (2005)

8. Pang, B., Lee, L., Vaithyanathan, S.: Thumbs up?: sentiment classification using machine learning techniques. In: ACL-02 conference on Empirical methods in natural language processing - EMNLP'02. (2002)

9. Peñalver-Martinez, I., Garcia-Sanchez, F., Valencia-Garcia, R., Rodríguez-García, M.Á., Moreno, V., Fraga, A., Sánchez-Cervantes, J.L.: Feature-based opinion mining through ontologies. Expert Systems with Applications. 41, 5995–6008 (2014)

10. Fan, Z.-P., Che, Y.-J., Chen, Z.-Y.: Product sales forecasting using online reviews and historical sales data: A method combining the Bass model and sentiment analysis. Journal of Business Research. 74, 90–100 (2017)

11. Tiwari, P., Mishra, B.K., Kumar, S., Kumar, V.: Implementation of n-gram Methodology for Rotten Tomatoes Review Dataset Sentiment Analysis. International Journal of Knowledge Discovery in Bioinformatics. 7, 30–41 (2017)

12. Liu, Y., Bi, J.-W., Fan, Z.-P.: Ranking products through online reviews: A method based on sentiment analysis technique and intuitionistic fuzzy set theory. Information Fusion. 36, 149–161 (2017)

13. Villaroel Ordenes, F., Ludwig, S., Grewal, D., de Ruyter, K., Wetzels, M.: Analyzing Online Reviews Through the Lens of Speech Act Theory: Implications for Consumer Sentiment Analysis. Journal of Consumer Research. 189, 227-246 (2016)

14. Qiu, G., Liu, B., Chen, C.: Opinion Word Expansion and Target Extraction through Double Propagation. Association for Computational Linguistics. 37, 9-27 (2011)

15. Santorini, B.: Part-of-speech tagging guidelines for the Penn Treebank Project (3rd revision). (1990)

16. Kumar, S., Kumar, P., Singh, M.P.: A Generalized Procedure of Opinion Mining and Sentiment Analysis. In: International Conference on Recent Trends in Communication and Computer Networks (ComNet), pp. 105-108. Hyderabad (2013)

17. Bird, S., Loper, E., Klein, E.: Natural Language Processing with Python. O’Reilly Media Inc. (2009)

18. Walt, S., Colbert, S. C., Varoquaux, G.: The NumPy Array: A Structure for Efficient Numerical Computation. Computing in Science & Engineering. 13, 22-30 (2011)

19. WordNet. http://wordnet.princeton.edu

20. Baccianella, S., Esuli, A., Sebastiani, F.: SentiWordNet 3.0: An Enhanced Lexical Resource for Sentiment Analysis and Opinion Mining. In: 7th Conference on International Language Resources and Evaluation (LREC’10), pp. 2200-2204 (2010)

21. Kumar, P., Dasari, Y., Nath, S., Sinha, A.: Controlling and Mitigating Targeted Socio-Economic Attacks. In: 15th IFIP WG 6.11 Conference on e-Business, e-Services, and e-Society (I3E 2016), pp. 471-476. Springer International Publishing, Swansea (2016)