Title: TOWARDS BUILDING AFFECT SENSITIVE WORD DIS-TRIBUTIONS

Abstract: Learning word representations from large available corpora relies on the distributional hypothesis that words present in similar contexts tend to have similar meanings. Recent work has shown that word representations learnt in this manner lack sentiment information which, fortunately, can be leveraged using external knowledge. Our work addresses the question: Can affect lexica improve the word representations learnt from a corpus ? In this work, we propose techniques to incorporate affect lexica, which capture fine-grained information about a word's psycholinguistic and emotional orientation, into the training process of Word2Vec SkipGram, Word2Vec CBOW, and GloVe methods using a Joint Learning approach. We use affect scores from Warriner's affect lexicon to regularize the vector representations learnt from an unlabeled corpus. Our proposed method outperforms previously proposed methods on standard tasks for word similarity detection, outlier detection, and sentiment detection. We also show the usefulness of our approach for a new task related to the prediction of formality, frustration, and politeness in corporate communication.

1. INTRODUCTION:

In natural language research, words, sentences and paragraphs are considered in context through vector space representations, rather than as atomic units with no relational information among them. Although n-gram based methods trained on large volumes of data have been found to outperform more complex approaches both on computational cost and accuracy, the techniques do not scale well in cases where the corpus size is limited(for example, for labeled speech or affect corpora with a size of a few millions of words). Recent work has attempted to improve the performance of word distributions for downstream tasks such as sentiment analysis (Sedoc et al., 2017) and knowledge base completion (Kumar & Araki, 2016) using lexical knowledge to enrich word embeddings, by performing methods such as regularization or introducing a loss term in the learning objective.

Sentiment relationships between words can be considered transitive, where 'good' < 'better' < 'best' implies that 'good' < 'best'. However, word representations based on traditional approaches such as Word2Vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014) are agnostic to the associated sentiments, emotions, or more generally affects (Bradley & Lang, 1999) . Furthermore, although words such as delighted and disappointed share similar vector representations given their similar contexts, these words are associated with opposite reactions (or sentiments) as well as have a fairly different interpreted meaning. The challenge in using syntactic relational information for sentiment detection, is that sentiment relations are transitive and symmetric (i.e., if 'delighted' is the opposite of 'disappointed', then 'disappointed' is the opposite of 'delighted'.) Ignoring the bipolar nature of words could lead to spurious results, especially in predictive tasks related to synonyms and antonyms and sentiment analysis. On the other hand, incorporating affect-related information would make word distributions homogeneous and suitable for speech and text generation tasks that aim at capturing author or reader reactions. Furthermore, by using a small sentiment lexicon, it is possible to develop an automatic way to rate words based on their vector space representations. This could help reduce the time and cost required to gather word ratings, as well as eliminate the implicit biases that may be introduced in annotations, such as the high correlation between high valence ratings with high arousal reported by Sedoc et al. (2017) .

We present an approach to build affect-enriched word representations. In other words, we enhance word distributions by incorporating reactions and affect dimensions. The output of this work produces word distributions that capture human reactions by modeling the affect information in the words. The affective word representations distinguish between semantically similar words that have varying affective interpretations. Affect is represented as a weighted relational information between two words, following the approach used by existing work. Sedoc et al. (2017) identify words of opposite polarity by performing signed spectral clustering on pre-trained embeddings. We present an approach to incorporate external affect and reaction signals in the pre-training step, using the hand-annotated affect lexica to learn from. Our experiments are based on using the state-of-the-art Warriner's affect lexicon (Warriner et al., 2013) as the input. The proposed approach builds on the intuition that relationships between synonyms and antonyms can be characterized using semantic dictionaries and the relationship can then be deterministically captured into the training loss functions.

We evaluate the proposed enriched word distributions on standard natural language tasks. We predict formality, frustration and politeness on a labeled dataset and show improved results using the enriched word embeddings. Further, we outperform the state-of-the-art for sentiment prediction on standard datasets. The key contributions of this paper include:

• Algorithm to incorporate affect sensors in the cost functions of distributional word representations (including Word2Vec SkipGram, Word2Vec CBOW, and GloVe) during training using semantic and external affect signals.

• Establish the utility of affect enriched word-embeddings for linguistic tasks such as Sentiment and Formality prediction in text data. Our method out performs the state-of-the-art with an 20% improvement in accuracy for the outlier detection methods. Detailed results are reported in table 1.

• Introduce a workflow to incorporate affective and reaction signals to word representations during pre-training. We show the generalizability of the workflow through experiments on 3 existing embeddings; Word2Vec-CBOW, Word2Vec-SkipGram, and GloVe.

Section 2 covers the prior art in both pre-training and post-training approaches for distributional word representations. Section 3 presents the proposed approach and detailed experiments are discussed in section 4. We conclude with a discussion on the learnings and the observations through this process 5.

2. RELATED WORK:

A huge amount of research has explored how to use external resources to improve on Word2Vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014) embeddings. Research has refined embeddings for various downstream tasks such as dependency parsing (Bansal et al., 2014) , sentiment analysis (Sedoc et al., 2017) and knowledge base completion (Kumar & Araki, 2016) .

There are mainly two approaches: joint learning and post-training. The former considers the incorporation of external knowledge into the training process of learning word embeddings itself. On the other hand, post-training approaches take already trained embeddings and use additional information to modify them. Our approach falls in the first category.

To the best of our knowledge, no prior work focuses on improving word embeddings by jointly learning from a corpus and an affect lexica. Bian et al. (2014) define a new basis for word representation and explore syntactic, semantic and morphological knowledge to provide additional information.

A binary indicator function is used to define relations. Yu & Dredze (2014) propose a Relation Constrained Model(RCM) which predicts one word from another related word. They use a linear combination of the objectives in CBOW and RCM for joint learning. Kiela et al. (2015) build specialized word embeddings for either similarity or relatedness. They use a joint learning approach by using additional context words from external sources with the SkipGram loss function. Our Word2Vec approach is similar to the pre-training model used in Kiela et al. (2015) . Essentially, words in the pruned list L i pruned (section 3) can also be taken as additional context words, similar to their understanding. However, in our case, this addition is from an affect lexica and includes a strength associated with it. Xu et al. (2014) propose a general framework RC-NET to incorporate knowledge into the word distributions. They encode external relational and categorical information into regularization functions and combine them with the original objective of Word2Vec SkipGram model. Our modified loss function can also be thought of as similar to the one in "Categorical Knowledge Powered model" in Xu et al. (2014) , thinking of the strength as a similarity score, although the distance function there is just defined as the Euclidean distance. Bollegala et al. (2016) use a similar approach as ours for GloVe method to include various relations like synonyms, antonyms, hyponyms and so on. They make use of a binary function to indicate whether the relation between any two words exists or not.

For post-training, Faruqui et al. (2014) makes no assumption about input pre-trained word vectors. This work proposes an objective to further refine the input embeddings using relational information from semantic lexica. Another post training approach has been proposed in Mrkšić et al. (2016) which defines the final objective as a weighted sum of "Antonym Repel", "Synonym Attract" and "Vector Space Preservation" objectives.

3 JOINT LEARNING FROM UNLABELED CORPUS AND AFFECT LEXICA 3.1 NOTATIONS Consider a corpus C and an affect lexica L consisting of l(word, affect-score) pairs. Let us denote the i th pair in L with p i = (w i , s i ). We define a function S(i, j) which captures the strength of the relationship between any two words w i and w j in L.

3.2. OUR PROPOSED APPROACH:

Figure 1 : Workflow for the incorporation of affect information into the process of learning word embeddings.

To incorporate an affect lexica into the process of learning word embeddings, we follow a three step approach (See Figure 1 ):

Step 1: Identifying Synonyms and Antonyms -Given a pair p i in L, WordNet (Miller, 1995) is used to identify all pairs p j in L such that w j is either a synonym or an antonym of w i . Synonyms and antonyms are retrieved based on WordNet definitions. Note that, S(i, j) will be defined as non-zero only when w j is returned by WordNet as the synonym or antonym of w i . For example, consider a pair of words from the Warriner's affect lexicon (see section 4.1 for lexicon details): although 'confident' and 'funny' have similar valence scores of 7.56 and 7.59 respectively (on a scale of 1-9), they do not share a semantic relationship between them (i.e. not defined as synonyms or antonyms). Here, S(i, j) is set to 0.

Step 2: Defining strength S(i, j) for all possible (i, j) pairs -Polarity information is captured in our modeling by centering the scores around 0 instead of the 1-9 scale. As already mentioned, S(i, j) is defined as 0 if w j is not a synonym or antonym of w i . For all other cases, the strength models the difference in affect scores of the two words under consideration. If the words have scores with the same sign, we define a positive strength inversely proportional to the relative distance between them. If the words have scores with opposite signs, the strength is negative, with magnitude directly proportional to the difference in their scores. Algorithm 1 describes the approach.

Algorithm 1 Compute strength between two words 1: function GETSTRENGTH(s i , s j , smax, smin) where s i and s j -normalized scores of w i and w j , smax and smin -maximum and minimum normalized scores return strength 15: end function

Step 3: Loss function definition -We introduce a new loss function for the embedding training. The loss functions defined for Word2Vec and GloVe are described here.

Word2Vec loss function: Rong (2014) describes the back-propagation algorithm of Word2Vec:

SkipGram and CBOW techniques. We build on top of that intuition. For the SkipGram approach, the model predicts the context of an input word in the corpus. Using the same notation as introduced in Rong ( 2014), the loss function of Word2Vec model (Mikolov et al., 2013) with negative sampling is defined as the following:

EQUATION

where w O is the output word (i.e. the positive sample) and v w O is it's output vector. h is the output value of the hidden layer. For the SkipGram approach, h is simply v w I , which is the input vector corresponding to the input word w I . W neg = {w j |1, ..., K} is the set of all negative samples. The standard unigram distribution raised to the 3 4 th power for this sampling is used for all reported experiments.

Information from affect lexica is incorporated using another loss function in the following manner:

EQUATION

where v wj is the output vector as already defined, h is the hidden layer output, i is the index of the input word w I in L and S(i, j) is the relation strength. L i pruned is the pruned list of words obtained from WordNet corresponding to the input word w I (same as the word w i in L).

Note that i will only be defined if the input word w I belongs to the affect lexica L. Hence, the loss function will only matter when a word present in L appears at the input end.

The final loss function is defined as:

E = E 1 + λE 2 (3)

where λ ≥ 0 is a hyper-parameter.

This modification results in a modified form of the derivative of E with respect to (v T wj h), as given by the following equation: The weight update equations can be further obtained using equation 4 as in Rong (2014) . Intuitively, if the strength is positive for a pair of words, the back-propagation algorithm makes their corresponding embeddings closer. Alternatively, if the strength is negative, the algorithm tries to move them apart. The magnitude of the strength controls the speed of this movement.

∂E ∂v T wj h =      σ(v T wj h) -1 if w j = w O λS(i, j)(σ(v T wj h) -1) if w j ∈ L pruned σ(v T wj h) if w j ∈ W neg (4)

Inspired from negative sampling done from the vocabulary, we also try sampling the words from the entire affect lexica instead of using WordNet. This suffers from the semantic relationship problems discussed in step one, the performance, hence declines.

The model used in CBOW is the opposite of the one defined for SkipGram. Given a context window around a word, the model predicts that word. Hence, the hidden layer output is defined as h = 1 C C c=1 v wc , the combination of all input word vectors corresponding to the context of the output word. The loss function is formulated similar to the one already described.

GloVe (Pennington et al., 2014) Loss Function: We achieve this by using a weighted regularized version of the original GloVe objective. Unlike Word2Vec, which considers only local cooccurrences, this approach uses global co-occurrences over our entire corpus.

We use the original GloVe implementation1 , first extracting the vocabulary and building a cooccurrence matrix X. Borrowing the notation from Pennington et al. (2014) , the objective for GloVe is as follows:

EQUATION

)

where V is the vocabulary, b p and b q are real-valued bias terms, w p is the target vector for the word w p and wq is the corresponding context vector for the word w q . f (X pq ) is a weighting function defined as:

EQUATION

)

Using the same values as in Pennington et al. (2014) , we keep α as 3 4 and x max as 100. With the strength function S(i, j)(defined in Step 2) in mind, we define a regularization term as follows:

EQUATION

where indices i and j in L correspond to p and q in V respectively. The final objective is obtained as follows:

EQUATION

using the same hyperparameter λ as before. The obtained expression is similar to the one used in Bollegala et al. (2016) , except that instead of using a strength function S(i, j), they use a binary function R(i, j), which indicates whether a relation exists between words w i and w j or not. We direct the readers to Bollegala et al. (2016) for the changes in update equations, which also result in similar values.

Figure 2 illustrates the proposed method through the t-distributed Stochastic Neighbor Embedding (t-SNE) visualization of the first two components of a subset of words. The points are colored to identify their positive (blue) or negative (red) valence in the affect lexicon. As shown in Figure 2a , 'accept', 'reject' and 'refuse' were found close together in the original representation. By adding valence information to baseline embeddings (Figure 2b ), the model is able to (a) pull words of similar sentiment closer together and (b) pull words of opposite sentiment, further apart. This shows that our method has face-validity in being able to identify and distinguish sentiment polarities within word embeddings.

4. EXPERIMENTS AND RESULTS:

In this section, we conduct experiments to examine whether incorporating affect information into learning continuous word representations can improve the intrinsic quality of word embeddings and the performance of trained models on downstream tasks. First, we introduce the dataset and then describe the evaluation framework.

4.1. DATA:

ukWaC corpus: We use the ukWaC (Baroni et al., 2009) corpus for building our word embeddings.

It is large enough to obtain embeddings of a good quality, while still being tractable in terms of time and space resources. We flattened the dependency-parsed format of this corpus, resulting in 2.2 billion tokens and a vocabulary size of 569,574 words after removing the words having frequency count of less than 20.

We experimented with different values for λ and found that the highest similarity score on RG word similarity dataset (Rubenstein & Goodenough, 1965) was with a λ = 2. The details of this experiment are provided in the Appendix. Accordingly, we have used this value for λ throughout. We use a window size of 10 and keep the word embedding dimensions as 300. For Word2Vec, we use negative sampling of 15 words for optimization. For GloVe, we found that running the model for 5 iterations was sufficient. On a machine with 8 core Intel 3.4GHz processor and 16GB RAM, the Word2Vec skipgram approach takes 15 hours, CBOW takes under 100 minutes while GloVe takes approximately 15 minutes per iteration.

We compare the performance of the word2vec skipgram, word2vec CBOW and GloVe models for the following settings:

• The baseline approach corresponding to setting λ as 0, i.e. only training on the unlabeled ukWaC corpus • With λ = 2, incorporating either valence, arousal and dominance scores to the original corpus one at a time. We refer to these models as '+V','+A' and '+D' respectively • Incorporating the mean weight of valence, arousal and dominance to the original corpus.

We refer to this approach as '+VAD'. • Comparison against the state of the art: We reimplemented the approach by Bollegala et al. (2016) on our dataset. The authors use a binary indicator function for the incorporation of various relations such as synonyms and antonyms in the training process. In the original paper, this approach, trained on GloVe embeddings, demonstrably improved the state of the art on standard word similarity and analogy prediction. We pick their best performing model which uses synonym pairs and train it on our ukWaC corpus with the same parameter settings.

• Comparison against post-training baseline (Append): We compare our results against a simple post-training baseline in which we concatenate the word-vectors (D-dimensional WV) with VAD-vectors (3-dimensional AV) resulting in a D + 3 dimensional vector. The considered dataset might not contain VAD ratings for stop words and proper nouns. Since they are neutral in sentiment (V), strength (D) and arousal (A), for these out-of-dictionary words the VAD vector is set to neutral ν = [5, 5, 5] in all our experiments.

1. Normalize both embeddings with their L2-Norms (Equation 9). This reduces the individual vectors to unit-length.

x i = x i D k=1 x 2 k ∀x i ∈ W V a i = a i 3 k=1 a 2 k ∀a i ∈ AV (9)

2. These regularized distributional word-embeddings are then concatenated with regularized affect scores as shown in Equation 10.

EQUATION

3. We also perform the standardization process on the resultant D + 3 dimensional embeddings. This transforms each dimension in the vector to have unit variance and zero mean so that each dimension contributes approximately proportionately.

EQUATION

where µ and σ represent the mean and std. deviation respectively.

The resultant vectors capture affect properties of the word in a distributional setting.

Warriner affect word list: We use affective information from Warriner's affect lexicon (Warriner et al., 2013) which comprises 13,915 words tagged on Valence, Arousal and Dominance scores on a scale of 1-9. Valence is the unhappy-happy scale, Arousal is the calm-excited scale and Dominance indicates the forcefulness of expressed affect. In all our experiments, we use a normalized scale from -4 to 4 to take the signed information into account, similar to Sedoc et al. (2017) .

4.2. EVALUATION FRAMEWORK:

We evaluate the proposed method on three standard tasks: predicting the similarity of words on seven benchmark datasets, detecting outliers in semantic clusters on the 8-8-8 dataset (Camacho-Collados & Navigli, 2016) and predicting sentiment on the Stanford Sentiment Treebank (Socher et al., 2013) . We then introduce a new dataset and task for formality, frustration and politeness detection for a labeled email corpus.

4.2.1. WORD SIMILARITY PREDICTION:

The task is to predict the similarity between given two words. We compute the cosine similarity between the corresponding word embeddings of the two words and assign it as the similarity score. We consider seven benchmark datasets:

• SIMLEX: the 999 word pairs list (Hill et al., 2016) • MC: 30 word pairs in Miller Charles (Miller & Charles, 1991) • MEN: 3000 pairs of words (Bruni et al., 2012) • RG: 65 word pairs by Rubenstein-Goodenough (Rubenstein & Goodenough, 1965) • RW: 2034 pairs in the Rare-Words dataset (Luong et al., 2013) ,

• SCWS: 2023 word pairs in Stanford's contextual word similarities (Huang et al., 2012) • WordSim: 353 word-pairs in the WordSim test collection (Finkelstein et al., 2001) . Each word pair in these benchmarks has a manually assigned score which we consider as gold standard ratings. We report the Spearman Correlation Coefficient between the gold standard similarity ratings and the cosine similarity scores assigned by our model.

The results are provided in Table 1 . The rows correspond to the approaches, with baselines corresponding to λ = 0. The columns correspond to the various datasets discussed above. Our proposed modifications show reasonable improvements for GloVe and word2vec CBOW embeddings. The poorest performance was observed for the word2vec skipgram approach, which did not outperform the baseline approach for most of the datasets.

In Table 1 we show that our approach showed modest improvements over the state of the art method by Bollegala et al. (2016) on word similarity for two datasets. We surmise that the nature of the word similarity task and the absence of "opposite" words in the benchmark datasets could be the reason why non-affective approaches are hard to beat for predicting word similarity. For outlier prediction, our approaches gives a performance improvement of 5% over the approach by Bollegala et al. (2016) .

4.2.2. OUTLIER DETECTION:

Word similarity tasks have been widely used for measuring the semantic coherence of vector space models. However, such tasks often suffer from low inter-annotator agreement scores of gold standard datasets. Hence, we also report our results on an outlier detection task (Camacho-Collados & Navigli, 2016) , in order to test the quality of semantic clusters in the vector space models. These results are on the 8-8-8 outlier detection dataset (Camacho-Collados & Navigli, 2016) containing 8 different topics, each made up of a cluster of 8 words and 8 possible outliers. Apart from Word2Vec SkipGram models, our approaches perform well on both OPP and Accuracy scores(higher is better). We observe that the incorporation of affect information in terms of Valence, Arousal and Dominance scores, shows improvements in an Outlier Detection task on an unrelated dataset with topics such as Football Teams, Solar Systems and Car Manufacturers.

Next, we evaluate our approach on two tasks for sentiment detection. Since our approach enriches word embeddings with affect information, we anticipate that our models would outperform the state of the art on these tasks.

4.2.3. SENTIMENT PREDICTION:

For the sentiment prediction task, we make use of an available Deep Averaging Network2 (DAN) model (Iyyer et al., 2015) with its default parameter settings along with our modified word embeddings.

We report the results on Stanford Sentiment Treebank(SST) dataset (Socher et al., 2013) which contains fine grained sentiment labels for 215,154 phrases in the parse trees of 11,855 sentences. We use the standard splits of the datasets.

Table 2 summarizes the results for sentiment prediction. We report the accuracy values for both fine and binary classification settings. For the latter, we remove all the instances with neutral labels. Our approaches show significant improvements over the baselines for both these metrics.

As evident from Table 2 , our approaches perform better on sentiment prediction task than the state of the art approach (Bollegala et al., 2016) . We attribute this improvement to our strength function which not only specifies the existence of a relation between two words, but also provides the signed strength associated with it.

4.2.4. PREDICTING FORMALITY, FRUSTRATION AND POLITENESS IN EMAILS:

Finally, we evaluate the quality our embeddings on an affect prediction task. We use a new dataset consisting of 980 emails from the ENRON dataset (Cohen, 2009), tagged with formality, frustration and politeness scores by human annotators. We use a CNN based regression model which takes our obtained embeddings as inputs and predicts formality, frustration politeness in emails.

We use a model which stacks a convolutional layer(5 filters of 10X5 size), a pooling layer(5X5 size with stride 5) and a dense layer(size 50, dropout 0.2). Rectified Linear Unit(ReLU) activation is used throughout with Stochastic Gradient Descent(SGD) optimizer and a mean squared loss.

Results are provided in Table 3 . For the affect prediction task, we report the mean and standard deviation of Mean Square Error(MSE) over five different train-test splits(test-ratio:0.2) of the dataset corresponding to Formality, Frustration and Politeness. In this case, along with GloVe and CBOW approaches, our modifications over SkipGram baseline (corpus only) show improvements with standard deviations in MSE values.

5. DISCUSSION:

We find reasonable improvements by our proposed approaches in all the task-based evaluations. SkipGram based methods perform poorly in word similarity prediction and outlier detection, but do well on sentiment and affect prediction. This difference in performance on downstream tasks, has been discussed before in Faruqui et al. (2016) and Camacho-Collados & Navigli (2016) , who point out various issues with word similarity based evaluations such as task subjectivity, low inter annotator agreements and low correlations between the performance of word vectors on word similarity and NLP tasks like text classification, parsing and sentiment analysis. Performance differences can also be attributed to corpus size, which are examined in the Appendix section. The results suggest that different embeddings perform well for different tasks. In word similarity tasks, the +V model performs well in GloVe setting but the +A model seems to perform the best for CBOW. Similar results are observed in sentiment prediction: for binary sentiment prediction, arousal scores give the best performance with CBOW embeddings but dominance and valence give the best performance with skip-gram and GloVe embeddings respectively. This suggests that the most flexible method could be an ensemble implementation that considers all these inputs before predicting a final class. Also note that given the vocabulary of our ukWaC corpus as 569, 574 words, our affect lexica with 13, 915 words is relatively small. We plan to take this work forward by further analysis in the future. At the least, we expect superior word embeddings with better quality and larger affect lexica.

5.1. AFFECT-POLYSEMY:

Words may have different affect in different contexts. This happens because their meanings change from one context to another. For example "A wicked woman came in" vs a slang "This game is wicked" should have different affect scores for both "wicked" because the meaning of the word almost turns on its head (from "devilish" to "cool/commendable"). But datasets like Warriner et al. (2013) do not capture this as the scores attributed by human annotators inherently average out the contexts (to the best of their knowledge) in which the words tend to appear. This problem (of "affect-polysemy") is related to "polysemy" which cripples today's word-level embedding models (Gladkova & Drozd (2016) , Huang et al. (2012) ). Word-level vector embeddings like SkipGram combine multiple senses of a polysemous word into single embedding vector (Liu et al. (2015) ). Although, there have been a few attempts to address this shortcoming using multi-prototype vectorspace models (Reisinger & Mooney (2010) , Huang et al. (2012 ), Tian et al. (2014) ), but they also face several challenges (Liu et al. (2015) ).

Since, single vector embeddings are still the most popular choice for NLP tasks, the proposed approach tries to improve the quality of these vanilla embeddings for tasks which can benefit by inducing affect information. Therefore, in this paper, we evaluate our approach on sentence/phrase level prediction tasks like sentiment prediction and tone of emails. The underlying state-of-theart recurrent/convolutional models try to create context-aware sentence level representations from word-embeddings to predict information like sentiment, formality etc. Our affect induced wordembeddings perform better than vanilla embeddings on these tasks thus supporting our claim.

2.. Studying the effect of corpus size::

We conduct an error analysis of the poor performance of Word2Vec Skipgram by observing the effect of varying the corpus size. We take different sized samples from the ukWaC corpus and report the word similarity performance on RG dataset in Figure 4 .

We observe irregularities in the performance of baseline approach(λ=0). Adding the affective information has a negative impact for corpora with sizes 2.5GB and 4.5GB while shows minor improvements over baseline for larger corpora. This improvement over baseline is the most for a smaller, 100MB corpus. We believe better preprocessing on the corpus should help with these non-intuitive observations. 

:

https://nlp.stanford.edu/projects/glove/

https://github.com/miyyer/dan

Figures and Tables:

if s i s j > 0 thenBoth scores have the same sign 5:if s i > 0 and s j > 0 then 6: strength = |s i -s j |/smax; |.| refers to taking the absolute value. |s i -s j |/(smax-smin)

Figure 2: t-distributed Stochastic Neighbor Embedding (t-SNE) visualizations of a subset of English words and their normalized valence scores. Incorporating valence information in word vectors separates the positive valence words from the negatives.

Figure 4: Word similarity scores(Spearman correlation coefficient) on RG dataset for a Word2Vec SkipGram model trained on corpus with different sizes using λ values as 0 and 2 with valence affect list.

Performance of proposed approaches on word similarity and outlier detection: (a) In terms of Spearman correlation coefficients for the word similarity task on 7 benchmark datasets, (b) In terms of the Outlier Position Percentage (OPP) and accuracy scores for the outlier detection task on 8-8-8 dataset, The baseline model refers to the corpus only approach, with λ = 0. λ is set to 2 for all other approaches: using Valence list(+V), Arousal(+A), Dominance(+D) and average score of valence, arousal and dominance (+VAD). (c) and (d) provides comparison with prior work.

Table content: <table><tr><td/><td/><td/><td>(a)</td><td/><td>(b)</td></tr><tr><td>Model baseline +V +A +D +VAD baseline +V +A +D +VAD baseline +V +A +D +VAD Append</td><td>SIMLEX 0.452 0.459 0.453 0.454 0.454 0.383 0.381 0.312 0.356 0.374 0.349 0.354 0.348 0.353 0.349 0.369</td><td colspan="2">Word Similarity MEN RG RW Word2vec CBOW 0.757 0.757 0.824 0.442 MC 0.791 0.756 0.827 0.438 0.794 0.755 0.839 0.44 0.767 0.756 0.837 0.442 0.784 0.759 0.841 0.433 Word2vec Skipgram 0.721 0.749 0.74 0.43 0.721 0.7 0.761 0.39 0.703 0.698 0.647 0.399 0.707 0.689 0.723 0.393 0.687 0.711 0.718 0.411 GloVe 0.676 0.613 0.775 0.298 0.734 0.614 0.823 0.3 0.703 0.612 0.778 0.291 0.721 0.615 0.817 0.297 0.736 0.613 0.812 0.297 0.682 0.534 0.723 0.239</td><td>SCWS WordSim 0.64 0.732 0.641 0.733 0.645 0.743 0.644 0.736 0.643 0.727 0.643 0.728 0.579 0.692 0.574 0.658 0.575 0.689 0.597 0.694 0.505 0.58 0.505 0.583 0.502 0.576 0.501 0.581 0.502 0.584 0.424 0.399</td><td colspan="2">Model Word2vec CBOW Outlier Detection OPP Accuracy baseline 76.172 12.5 +V 77.539 15.625 +A 78.711 18.75 +D 78.125 17.188 +VAD 78.516 18.75 Word2vec Skipgram baseline 77.93 18.75 +V 76.172 17.187 +A 76.953 15.625 +D 75.586 14.062 +VAD 76.562 14.062 GloVe baseline 76.172 25.0 +V 75.781 26.562 +A 75.781 28.125 +D 76.367 28.125 +VAD 75.976 28.125</td></tr><tr><td/><td/><td>(c)</td><td/><td/><td>(d)</td></tr><tr><td/><td>Model</td><td colspan="2">Word Similarity</td><td>Model</td><td colspan="2">Outlier Detection</td></tr><tr><td/><td/><td>MC</td><td>RG</td><td/><td>OPP</td><td>Accuracy</td></tr><tr><td/><td/><td>GloVe</td><td/><td/><td>GloVe</td></tr><tr><td/><td>baseline</td><td>0.676</td><td>0.775</td><td>baseline</td><td>76.172</td><td>25.0</td></tr><tr><td/><td>+V</td><td>0.734</td><td>0.823</td><td>+V</td><td>75.781</td><td>26.562</td></tr><tr><td/><td>+A</td><td>0.703</td><td>0.778</td><td>+A</td><td>75.781</td><td>28.125</td></tr><tr><td/><td>+D</td><td>0.721</td><td>0.817</td><td>+D</td><td>76.367</td><td>28.125</td></tr><tr><td/><td>+VAD</td><td>0.736</td><td>0.812</td><td>+VAD</td><td>75.976</td><td>28.125</td></tr><tr><td/><td>Append</td><td>0.682</td><td>0.723</td><td>Append</td><td>66.992</td><td>14.063</td></tr><tr><td colspan="3">Bollegala et al. (2016) 0.703</td><td>0.812</td><td colspan="2">Bollegala et al. (2016) 75.781</td><td>26.562</td></tr></table>

Table 1 summa-rizes the Outlier Position Percentage(OPP score) and Accuracy of our different models. We refer Camacho-Collados & Navigli (2016) to the readers for further details on the dataset and evaluation metrics.

Table content: <table/>

Performance of proposed approaches on sentiment prediction task: (a) In terms of Fine and Binary classification accuracies, using the Deep Averaging Network (DAN) model on the Stanford Sentiment Treebank, (b) Comparison with prior work. The baseline model refers to the corpus only approach, with λ = 0. λ is set to 2 for all other approaches: using Valence list(+V), Arousal(+A), Dominance(+D) and average strength(+VAD).

Table content: <table><tr><td/><td>(a)</td><td/><td/><td/><td/></tr><tr><td>Model</td><td colspan="2">Sentiment Prediction</td><td/><td/><td/></tr><tr><td/><td>Fine</td><td>Binary</td><td/><td/><td/></tr><tr><td colspan="3">Word2vec CBOW baseline 43.53 73.97</td><td/><td>(b)</td><td/></tr><tr><td>+V</td><td>43.03</td><td>74.90</td><td/><td/><td/></tr><tr><td>+A</td><td>42.89</td><td>78.58</td><td>Model</td><td colspan="2">Sentiment Prediction</td></tr><tr><td>+D</td><td>42.99</td><td>77.59</td><td/><td>Fine</td><td>Binary</td></tr><tr><td>+VAD</td><td>42.22</td><td>76.50</td><td/><td>GloVe</td><td/></tr><tr><td colspan="3">Word2vec Skipgram</td><td>baseline</td><td>41.40</td><td>80.07</td></tr><tr><td colspan="2">baseline 42.89</td><td>79.79</td><td>+V</td><td>41.45</td><td>78.09</td></tr><tr><td>+V</td><td>43.12</td><td>79.13</td><td>+A</td><td>39.32</td><td>79.47</td></tr><tr><td>+A</td><td>41.49</td><td>79.41</td><td>+D</td><td>43.26</td><td>79.21</td></tr><tr><td>+D</td><td>40.23</td><td>79.85</td><td>+VAD</td><td>41.45</td><td>80.35</td></tr><tr><td>+VAD</td><td>40.50</td><td>75.34</td><td>Append</td><td>43.10</td><td>82.76</td></tr><tr><td/><td>GloVe</td><td/><td colspan="2">Bollegala et al. (2016) 42.26</td><td>80.42</td></tr><tr><td colspan="2">baseline 41.40</td><td>80.07</td><td/><td/><td/></tr><tr><td>+V</td><td>41.45</td><td>78.09</td><td/><td/><td/></tr><tr><td>+A</td><td>39.32</td><td>79.47</td><td/><td/><td/></tr><tr><td>+D</td><td>43.26</td><td>79.21</td><td/><td/><td/></tr><tr><td>+VAD</td><td>41.45</td><td>80.35</td><td/><td/><td/></tr></table>

Performance of proposed approaches on affect prediction task: (a) In terms of Mean Square Error (MSE) values for affect prediction on a labeled email corpus, (b) Comparison with prior work.The baseline model refers to the corpus only approach, with λ = 0. λ is set to 2 for all other approaches: using Valence list(+V), Arousal(+A), Dominance(+D) and average strength(+VAD).

Table content: <table><tr><td/><td/><td/><td>(a)</td><td/><td/><td/></tr><tr><td>Model</td><td/><td colspan="5">Affect Prediction in emails(all values X 10 -2 )</td></tr><tr><td/><td colspan="2">Formality</td><td colspan="2">Frustration</td><td colspan="2">Politeness</td></tr><tr><td/><td colspan="6">Mean Std. Dev. Mean Std. Dev. Mean Std. Dev.</td></tr><tr><td/><td/><td colspan="3">Word2vec CBOW</td><td/><td/></tr><tr><td>baseline</td><td>2.93</td><td>0.35</td><td>2.40</td><td>0.34</td><td>3.09</td><td>0.14</td></tr><tr><td>+V</td><td>2.89</td><td>0.35</td><td>2.61</td><td>0.36</td><td>3.24</td><td>0.15</td></tr><tr><td>+A</td><td>3.19</td><td>0.14</td><td>2.69</td><td>0.55</td><td>3.22</td><td>0.29</td></tr><tr><td>+D</td><td>2.82</td><td>0.22</td><td>2.12</td><td>0.27</td><td>3.04</td><td>0.36</td></tr><tr><td>+VAD</td><td>2.94</td><td>0.14</td><td>2.33</td><td>0.28</td><td>3.28</td><td>0.34</td></tr><tr><td/><td/><td colspan="3">Word2vec Skipgram</td><td/><td/></tr><tr><td>baseline</td><td>2.80</td><td>0.29</td><td>2.37</td><td>0.52</td><td>3.26</td><td>0.38</td></tr><tr><td>+V</td><td>3.18</td><td>0.33</td><td>2.28</td><td>0.25</td><td>3.26</td><td>0.25</td></tr><tr><td>+A</td><td>2.79</td><td>0.36</td><td>2.42</td><td>0.23</td><td>3.22</td><td>0.28</td></tr><tr><td>+D</td><td>3.00</td><td>0.34</td><td>2.85</td><td>0.38</td><td>3.10</td><td>0.16</td></tr><tr><td>+VAD</td><td>3.26</td><td>0.43</td><td>2.62</td><td>0.40</td><td>3.25</td><td>0.29</td></tr><tr><td/><td/><td/><td>GloVe</td><td/><td/><td/></tr><tr><td>baseline</td><td>3.12</td><td>0.17</td><td>2.59</td><td>0.35</td><td>3.26</td><td>0.22</td></tr><tr><td>+V</td><td>2.93</td><td>0.28</td><td>2.32</td><td>0.37</td><td>3.19</td><td>0.11</td></tr><tr><td>+A</td><td>2.93</td><td>0.43</td><td>2.53</td><td>0.25</td><td>3.13</td><td>0.25</td></tr><tr><td>+D</td><td>3.11</td><td>0.36</td><td>2.50</td><td>0.42</td><td>3.27</td><td>0.25</td></tr><tr><td>+VAD</td><td>3.47</td><td>0.26</td><td>2.51</td><td>0.30</td><td>3.49</td><td>0.38</td></tr><tr><td/><td/><td/><td>(b)</td><td/><td/><td/></tr><tr><td>Model</td><td/><td/><td colspan="4">Affect Prediction in emails</td></tr><tr><td/><td/><td colspan="2">Formality</td><td colspan="2">Frustration</td><td colspan="2">Politeness</td></tr><tr><td/><td/><td colspan="6">Mean Std. Dev. Mean Std. Dev. Mean Std. Dev.</td></tr><tr><td/><td/><td/><td>GloVe</td><td/><td/><td/></tr><tr><td>baseline</td><td/><td>3.12</td><td>0.17</td><td>2.59</td><td>0.35</td><td>3.26</td><td>0.22</td></tr><tr><td>+V</td><td/><td>2.93</td><td>0.28</td><td>2.32</td><td>0.37</td><td>3.19</td><td>0.11</td></tr><tr><td>+A</td><td/><td>2.93</td><td>0.43</td><td>2.53</td><td>0.25</td><td>3.13</td><td>0.25</td></tr><tr><td>+D</td><td/><td>3.11</td><td>0.36</td><td>2.50</td><td>0.42</td><td>3.27</td><td>0.25</td></tr><tr><td>+VAD</td><td/><td>3.47</td><td>0.26</td><td>2.51</td><td>0.30</td><td>3.49</td><td>0.38</td></tr><tr><td>Append</td><td/><td>2.89</td><td>0.18</td><td>2.34</td><td>0.44</td><td>3.28</td><td>0.30</td></tr><tr><td colspan="2">Bollegala et al. (2016)</td><td>3.03</td><td>0.30</td><td>2.59</td><td>0.48</td><td>3.37</td><td>0.16</td></tr></table>

Fine grained predictions for GloVe based models on instances from Stanford Sentiment Treebank(SST) dataset. "baseline" refers to the corpus only approach, +Valence refers to our base-line+Valence affect lexicon model andBollegala et al. (2016) uses WordNet synonyms along with the corpus. Class labels are from +2(Very positive) to -2(Very Negative).

Table content: <table><tr><td colspan="2">Serial No. Sentence</td><td colspan="4">True label baseline +Valence (Bollegala et al., 2016)</td></tr><tr><td>1</td><td>if this movie were a book , it would be a page-turner , you ca n't wait to see what happens next .</td><td>+2</td><td>0</td><td>-1</td><td>0</td></tr><tr><td>2</td><td>the movie will reach far beyond its core demographic .</td><td>+2</td><td>-1</td><td>-1</td><td>-1</td></tr><tr><td>3</td><td>offers that rare combination of entertainment and education .</td><td>+2</td><td>+2</td><td>+2</td><td>+2</td></tr><tr><td>4</td><td>a smart , sweet and playful romantic comedy .</td><td>+2</td><td>+2</td><td>+2</td><td>+2</td></tr><tr><td>5</td><td>a dreadful live-action movie .</td><td>-2</td><td>+2</td><td>-2</td><td>+2</td></tr><tr><td>6</td><td>hubac 's script is a gem .</td><td>+2</td><td>-1</td><td>+1</td><td>+1</td></tr><tr><td>7</td><td>i hate this movie</td><td>-1</td><td>+2</td><td>-2</td><td>+2</td></tr></table>

Top 10 nearest neighbors of the word 'dreadful' in terms of cosine similarity for three GloVe based models.

Table content: <table><tr><td colspan="2">(a) baseline model</td><td colspan="2">(b) baseline+Valence</td><td colspan="2">(c) (Bollegala et al., 2016)</td></tr><tr><td>Word</td><td>Similarity</td><td>Word</td><td>Similarity</td><td>Word</td><td>Similarity</td></tr><tr><td>terrible</td><td>0.853</td><td>terrible</td><td>0.932</td><td>terrible</td><td>0.860</td></tr><tr><td>awful</td><td>0.844</td><td>awful</td><td>0.930</td><td>awful</td><td>0.848</td></tr><tr><td>horrible</td><td>0.778</td><td>horrendous</td><td>0.925</td><td>horrible</td><td>0.792</td></tr><tr><td>appalling</td><td>0.752</td><td>atrocious</td><td>0.909</td><td>appalling</td><td>0.750</td></tr><tr><td>horrendous</td><td>0.725</td><td>frightful</td><td>0.903</td><td>horrendous</td><td>0.743</td></tr><tr><td>ghastly</td><td>0.723</td><td>dread</td><td>0.885</td><td>ghastly</td><td>0.719</td></tr><tr><td>atrocious</td><td>0.672</td><td>unspeakable</td><td>0.883</td><td>frightful</td><td>0.671</td></tr><tr><td>frightful</td><td>0.671</td><td>fearful</td><td>0.870</td><td>atrocious</td><td>0.671</td></tr><tr><td>horrid</td><td>0.664</td><td>abominable</td><td>0.863</td><td>horrid</td><td>0.669</td></tr><tr><td>enfant</td><td>0.662</td><td>horrific</td><td>0.858</td><td>enfant</td><td>0.662</td></tr></table>

Peer review (iclr2018):
 This paper proposes integrating information from a semantic resource that quantifies the affect of different words into a text-based word embedding algorithm. The affect lexical seems to be a very interesting resource (although I*m not sure what it means to call it *state of the art*), and definitely support the endeavour to make language models more reflective of complex semantic and pragmatic phenomena such as affect and sentiment. The justification for why we might want to do this with word embeddings in the manner proposed seems a little unconvincing to me: - The statement that *delighted* and *disappointed* will have similar contexts is not evident to me at least (other then them both being participle / adjectives). - Affect in language seems to me to be a very contextual phenomenon. Only a tiny subset of words have intrinsic and context-free affect. Most affect seems to me to come from the use of words in (phrasal, and extra-linguistic) contexts, so a more context-dependent model, in which affect is computed over phrases or sentences, would seem to be more appropriate. Consider words like *expensive*, *wicked*, *elimination*... The model proposes several applications (sentiment prediction, predicting email tone, word similarity) where the affect-based embeddings yield small improvements. However, in different cases, taking different flavours of affect information (V, A or D) produces the best score, so it is not clear what to conclude about what sort of information is most useful. It is not surprising to me that an algorithm that uses both WordNet and running text to compute word similarity scores improves over one that uses just running text. It also not surprising that adding information about affect improves the ability to predict sentiment and the tone of emails. To understand the importance of the proposed algorithm (rather than just the addition of additional data), I would like to see comparison with various different post-processing techniques using WordNet and the affect lexicon (i.e. not just Bollelaga et al.) including some much simpler baselines. For instance, what about averaging WordNet path-based distance metrics and distance in word embedding space (for word similarity), and other ways of applying the affect data to email tone prediction?