# Sentiment Analysis of Short Text - Sentence and User Representation Learning

Revant Kumar, Parminder Bhatia, Christopher Chow

<figure><img src="image_1.png" /></figure>

## To cite this version:

Revant Kumar, Parminder Bhatia, Christopher Chow. Sentiment Analysis of Short Text - Sentence and User Representation Learning. Georgia Institute of Technology, USA. 2015, pp.10. hal-04322917

HAL Id: hal-04322917

https://hal.science/hal-04322917v1

Submitted on 5 Dec 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Sentiment Analysis of Short Text - Sentence and User Representation Learning

**Revant Kumar**

MS in Computer Science
College of Computing, Georgia Institute of Technology, USA

**Parminder Bhatia**

MS in Computational Science & Engineering
College of Computing, Georgia Institute of Technology, USA

**Christopher Chow**

MS in Electrical Engineering
Georgia Institute of Technology, USA

## 1 Problem Statement & Motivation

The majority of existing approaches employ machine learning algorithms to build classifiers from tweets with manually annotated sentiment polarity. Most studies focus on designing effective features to obtain better classification performance.

Current methods work well for long paragraphs but they do not perform as well in sentences or short text. The objective of this project is to combine multiple existing word embedding schemes and neural network architectures to develop more powerful sentence models for sentiment classification.

The aim of a sentence model is to analyze and represent the semantic content of a sentence for purposes of classification or generation. The core of a sentence model involves a feature function that defines the process by which the features of the sentence are extracted from the features of the words or n-grams. Neural sentence models have a number of advantages. They can be trained to obtain generic vectors for words and phrases by predicting, for instance, the contexts in which the words and phrases occur. Through supervised training, neural sentence models can fine-tune these vectors to information that is specific to a certain task.

Existing work requires complicated networks which are at times tough to train and reproduce. In this paper, we demonstrate a sentiment analysis system that works on short texts such as Tweets. We also aim to incorporate Users Information in the Tweets to further enhance the system and make the tweets more personalized.

## 2 Related Work

Recently, there has been lot of work on sentence modeling and its application in other NLP tasks such as sentiment analysis, question answering, machine translation etc.

### 2.1 Neural Networks

* Neural Bag-of-Words (NBoW) Models: Projection layer that maps words, sub-word units or n-grams to high dimensional embeddings (vectors). The embeddings then combined component-wise with an operation such as summation. The resulting combined vector is classified through one or more fully connected layers.
* Recursive Neural Networks (RecNN): A model that adopts a more general structure provided by an external parse tree. At every node in the tree the contexts at the left and right children of the node are combined by a classical layer. The layer computed at the top node gives a representation for the sentence.

<figure><img src="image_3.png" /><figcaption>Figure 2: Narrow and wide types of convolution.
The filter m has size m = 5.</figcaption></figure>

<caption>Figure 1: Time Delay Neural Networks</caption>

## 2.2 Time Delay Neural Networks

$$c_j = m^T s_{j-m+1:j}$$

* The trained weights in the filter m correspond to a linguistic feature detector that learns to recognize a specific class of n-grams. These n-grams have size n ≤ m, where m is the width of the filter.

* Narrow type of convolution requires that s > m.

* A wide convolution ensures that all weights in the filter reach the entire sentence, including the words at the beginning and end of the sentence.

## 2.3 Dynamic Convolutional Neural Networks (DCNN)

* 1-D wide convolutions ensure that all weights in the filter reach the entire sentence, including the words at the beginnings and ends of the sentences.

* Dynamic k-max pooling - k is a function both of layer depth and sentence length

* Small filters at higher layers can capture syntactic or semantic relations between non-continuous phrases that are far apart in the input sentence.

* Folding operation to make it fully dependent across rows.

Recent work at Google Mind on Dynamic Convolution Neural Networks explores learning convolution filters at both the sentence and document level, hierarchically learning to capture and compose low level lexical features into high level semantic concepts. As DCNN models are based on the convolution operation, they are able to preserve ordering information between words in a sentence and between sentences in a document.

DCNNs are similar to the convolutional networks used in computer vision – with cascades of convolution, pooling and nonlinear transformations. However, DCNNs are specially adapted for some of the specifics of the sentence modeling task. They are particularly useful in **Discourse Analysis**, where relation between different sentences is also taken into account and also in tasks such as **Text Summarization**.

# 3 Approach and Techniques

## 3.1 Architecture (new)

The major objective of our network design is to incorporate the above advantages of DCNN with a simpler architecture that is easier to train and more scalable.
<figure><img src="image_4.png" /><figcaption>Figure 2: DCNN Architecture</figcaption></figure>

In our present work, we train a simple CNN with one layer of convolution on top of word vectors obtained from an unsupervised neural language model – word2vec. These embedding scheme was trained by Mikolov et al. (2013) on 100 billion words from Google News, and is publicly available.

Despite little tuning of hyper parameters, this simple model achieves excellent results on multiple benchmarks, suggesting that the pre-trained vectors are universal feature extractors that can be utilized for various classification tasks. Learning task-specific vectors through fine-tuning results in further improvements.

### 3.1.1 Sentence Representation

* We currently utilize *filter-sizes* of 3, 4, and 5, each with a length 100 feature map.

* The sentence representation is obtained by concatenating these feature maps to yield 300 total features.

* These features are fed into an MLP which consists of 300 inputs and hidden units and 2 outputs corresponding to sentiments.

* Dropout (p=0.5) is also added at the penultimate fully connected layer.

## 3.2 Existing Algorithms and Optimizations

We have tried to incorporate various types of losses in our algorithm.

### 3.2.1 Cross Entropy: Word - Sentiment

We express the cross-entropy loss associated with the true label of the document:

$$Lw_i = y_k \log(y^k) + (1 - y_k) \log(1 - y^k) + \lambda_w (\phi^2)$$
<figure><img src="image_5.png" /><figcaption>Figure 3: Our Architecture</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>Figure 4: Architecture Diagram for Sentence Modeling</figcaption></figure>
Optimizing against the above loss function maximizes the log probability according to the true label. We then backpropagate the error all the way through the word embeddings. Thus, the word representation achieved will also incorporate sentiment information which is not present in word2vec embeddings.

### 3.2.2 Negative Sampling: User - Word

We also express a loss function incorporating user modelling, where *u* is the author of a given tweet *i*, and *k* is a random other user.

$$Lu_{i,u} = \log(\sigma(user_u * tweet_i)) + \sum_k \log(\sigma(-user_k * tweet_i))$$

The idea is that similar users will have similar values of sentiment. Two words may mean slightly different things for different users. Thus, **great** could really positive for some users while others might use **great** sarcastically or more freely in a variety of contexts. We have tried to use negative sampling here between the user and the tweet. Thus, if two users write similar tweets then their negative sampling will be low.

### 3.2.3 Negative Sampling: User - User

Finally, we can express a loss using user-user information. Let *i* be a user who follows user *u* and *k* be a random user who doesn't follow user *u*. The loss is:

$$Lu_i u_u = \log(\sigma(user_u * follower_i)) + \sum_k \log(\sigma(-user_u * follower_k))$$

The idea is that similar users will have similar representations for the same word. We assume that a user will have similar representations as his followers which can help in determining the sentiment for a user according to loss 2. This loss helps in understanding user representations similarly to word2vec.

Thus the overall loss is given by:

$$L = Lw_i + Lu_{i,u} + Lu_i u_u$$

## 4 Experimental Design

A sentence (*n* words) is represented as a matrix of dimension *n* × *d* where *d* is the dimensionality for each word from word2vec.

A convolution operation involves a filter *w* ∈ $R^{h×k}$, which is applied to a window of *h* words to produce a new feature. This filter is applied to each possible window of words in the sentence {$x_{1:h}, x_{2:h+1}, ..., x_{n-h+1:n}$} to produce a feature map *c* = [*c*₁, *c*₂, ..., *c*_{n-h+1}], with *c* ∈ $R_{n-h+1}$.
We then apply a max-overtime pooling operation (Collobert et al., 2011) over the feature map and take the maximum value $\hat{c} = \max\{c\}$ as the feature corresponding to this particular filter. The idea is to capture the most important feature, one with the highest value for each feature map. This pooling scheme naturally deals with variable sentence lengths.

We have described the process by which one feature is extracted from one filter. The model uses multiple filters (with varying window sizes) to obtain multiple features. These features form the penultimate layer and are passed to a fully connected softmax layer whose output is the probability distribution over labels.

We summarize the above process as follows:

* Input 4D tensor (MiniBatch, 1, n, d)

* Weight 4D tensor (Number of features, 1, filter_size, d)
* Max Pooling - one value per feature

* Output of one convolution after max-pooling is (Number of features × 1).

* In this project, we use filters of length 3, 4 and 5, each with 100 features. Thus, each sentence is represented by a 300 element vector right before the fully connected softmax layer.

## 4.1 Model Variations

We experiment with several variants of the model.

* CNN-rand: Our baseline model where all words are randomly initialized and then modified during training. This should learn word embeddings like word2vec.

* CNN-static: A model with pretrained vectors from word2vec. All words – including the unknown ones that are randomly initialized – are kept static and only the other parameters of the model are learned.

* CNN-finetuned: Same as above but the pretrained vectors are fine-tuned for each task.

# 5 Results and Analysis

Using the above models, we have achieved various results and findings which we would like to discuss in the implementation order.

## 5.1 Data

Data used for evaluation is the SemEval 2013 Dataset for Twitter. The dataset consists of many tweets about many different topics. The dataset ground truth has three classes - positive, negative, and neutral. SemEval also provides unique user ID's per tweet, allowing us to employ user-dependent learning as described in the loss functions above. We first acquire and clean the data and hash it into a dictionary for efficient lookup and training.

## 5.2 Framework

We used Theano which is implemented in python, as it gives freedom to design your model in details where we can modify and change our loss functions. Theano allowed us to rapidly prototype network flow graphs and play with their parameters. Another advantage is that it calculates the derivatives automatically thus we don't have to implement back propagation every time.

## 5.3 Optimization Algorithm

We use stochastic gradient descent with a mini batch size of 50 units and we perform around 30 epochs. It takes around 8 to 10 hrs on a 16 CPU Cluster.

The specific type of SGD used is a kind of SGD with momentum called Adadelta, as recent papers have shown good results with this algorithm in language-related tasks.

## 5.4 Results and Evaluation

### 5.4.1 Sentiment Loss

Using the negative log likelihood for the sentiment, we generated 300 feature values for each sentence. We also test the classification power of the representations with an SVM. Using the above representations, we obtain around 83-84% F measure on Twitter sentiment data with the best CNN. We used the following activation functions:

* ReLU

* Tanh
* Sigmoid

* Identity

Out of the above, Tanh and Rectified Linear Units gave the best results.

### 5.4.2 Dropout

Using a dropout layer in the MLP gave us 2 to 4 percent improvement. Dropout proved to be such a good regularizer that it was fine to use a larger than necessary network and simply let dropout regularize it. We were able to achieve about 86-87% accuracy using dropout.

### 5.4.3 Combined Loss

After combining all the losses associated with tweets and user-associated losses, as mentioned in the equation

$$L = Lw_i + Lu_{i,u} + Lu_iu_u$$

We observed that results we achieved were around 85-86%. Using all the losses could not improve the results which we achieved after dropouts. We have various hypotheses for why this might be.

* It may be that even though we are calculating per-user loss though negative sampling, one of the major problem in data is that most of the users had just 1 or 2 tweets. Thus it would have been difficult for the $Lu_{i,u}$ and $Lu_iu_u$ loss to capture useful losses.

* Second reason is that even though we are given followers for a user, there may not be tweets for all of those followers, making inference about those followers difficult.

### 5.4.4 Static vs. Non-Static Representations

As we had inferred in the beginning we observed that using non-static representation of words gave us better results as compared to static representations.

The reason for this result is that word2vec representation are based on context where good and bad have similar representations. By backpropagating the losses till the words we are able to differentiate between those words more finely.

<figure><img src="image_7.png" /><figcaption>Figure 5: Using User Loss</figcaption></figure>
<figure><img src="image_8.png" /><figcaption>Figure 6: Using Dropout</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Figure 7: Combining Dropout and User Loss</figcaption></figure>
<figure><img src="image_10.png" /><figcaption>Figure 8: Dynamic Word Embeddings</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>Figure 9: Comparison with Popular Classification ML Algorithms</figcaption></figure>
# 6 Discussion

* **Sentiment Loss:** Obtained around 83-84 percent F-measure. Tanh and ReLU units performed the best.

* **Dropouts:** Achieved about 86-87 percent accuracy using dropout.

* **Combined loss:** Achieved around 85-86 percent F-measure. Using all the losses could not improve the results which we achieved after dropout because each User had 1-2 tweets and some followers had no tweets.

* **Static vs Non-static:** The reason for this result is that word2vec representation are based on context where good and bad have similar representations.

* **User Embedding-** Similarly we backpropagated the derivatives for the User embeddings which in itself can be used for many Social Computing Applications.

* **AdaDelta** gave similar results but in less number of epochs.

# 7 Conclusions and Future Work

* Achieved great results with a simple architecture.

* Learned sentence representations can be used for further NLP tasks such as machine translation and question answering.

* Learned user embeddings which can be used on Social Data graphs.

* Results could be improved from User losses if distribution for User-Tweet ratio is higher.

# References

[1] Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, Jeffrey Dean: Distributed Representations of Words and Phrases and their Compositionality. NIPS 2013: 3111-3119

[2] Nal Kalchbrenner and Edward Grefenstette and Phil Blunsom. "A Convolutional Neural Network for Modelling Sentences". Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics. June 2014

[3] Duyu Tang, Furu Wei, Nan Yang, Ming Zhou, Ting Liu, Bing Qin. Learning SentimentSpecific Word Embedding for Twitter Sentiment Classification. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 1555-1565, Baltimore, Maryland, USA, June 23-25 2014

[4] Yoon Kim. Convolutional Neural Networks for Sentence Classification. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746-1751, October 25-29, 2014, Doha, Qatar.

[5] D. Tang, F. Wei, B. Qin, T. Liu, and M. Zhou, "Coooolll: A Deep Learning System for Twitter Sentiment Classification," SemEval 2014, p. 208, 2014

[6] R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu, and P. Kuksa, "Natural language processing (almost) from scratch," The Journal of Machine Learning Research, vol. 12, pp. 2493-2537, 2011.