Venue: NAACL 2018

# Word Emotion Induction for Multiple Languages as

a Deep Multi-Task Learning Problem

Sven Buechel & Udo Hahn

{sven.buechel|udo.hahn}@uni-jena.de

Jena University Language & Information Engineering (JULIE) Lab

Friedrich-Schiller-Universitat Jena, Jena, Germany

[http://www.julielab.de](http://www.julielab.de)

###### Abstract

Predicting the emotional value of lexical items is a well-known problem in sentiment analysis. While research has focused on polarity for quite a long time, meanwhile this early focus has been shifted to more expressive emotion representation models (such as Basic Emotions or Valence-Arousal-Dominance). This change resulted in a proliferation of heterogeneous formats and, in parallel, often small-sized, non-interoperable resources (lexicons and corpus annotations). In particular, the limitations in size hampered the application of deep learning methods in this area because they typically require large amounts of input data. We here present a solution to get around this language data bottleneck by rephrasing word emotion induction as a multi-task learning problem. In this approach, the prediction of each independent emotion dimension is considered as an individual task and hidden layers are _shared_ between these dimensions. We investigate whether multi-task learning is more advantageous than single-task learning for emotion prediction by comparing our model against a wide range of alternative emotion and polarity induction methods featuring 9 typologically diverse languages and a total of 15 conditions. Our model turns out to outperform each one of them. Against all odds, the proposed deep learning approach yields _the largest gain_ on _the smallest_ data sets, merely composed of one thousand samples.

## 1 Introduction

Deep Learning (DL) has radically changed the rules of the game in NLP by dramatically boosting performance figures in almost all applications areas. Yet, one of the major premises of high-performance DL engines is their dependence on huge amounts of training data. As such, DL seems ill-suited for areas where training data are scarce, such as in the field of word emotion induction.

We will use the terms _polarity_ and _emotion_ here to distinguish between research focusing on "semantic orientation" Hatzivassiloglou and McKeown (1997) (the positiveness or negativeness) of affective states, on the one hand, and approaches which provide predictions based on some of the many more elaborated representational systems for affective states, on the other hand.

Originally, research activities focused on polarity alone. In the meantime, a shift towards more expressive representation models for emotion can be observed that heavily draws inspirations from psychological theory, e.g., Basic Emotions Ekman (1992) or the Valence-Arousal-Dominance model Bradley and Lang (1994).

Though this change turned out to be really beneficial for sentiment analysis in NLP, a large variety of mutually incompatible encodings schemes for emotion and, consequently, annotation formats for emotion metadata in corpora have emerged that hinder the interoperability of these resources and their subsequent reuse, e.g., on the basis of alignments or mergers Bucchel and Hahn (2017).

As an alternative way of dealing with thus unwarranted heterogeneity, we here examine the potential of multi-task learning MTL; Caruana (1997) for word-level emotion prediction. In MTL for neural networks, a single model is fitted to solve multiple, independent tasks (in our case, to predict different emotional dimensions) which typically results in learning more robust and meaningful intermediate representations. MTL has been shown to greatly decrease the risk of overfitting Baxter (1997), work well for various NLP tasks Setiawan et al. (2015); Liu et al. (2015); Sogaard and Goldberg (2016); Cummins et al. (2016); Liu et al. (2017); Peng et al. (2017), and practically increases sample size, thus making it a natural choice for small-sized data sets typically found in the area of word emotion induction

After a discussion of related work in Section 2, we will introduce several reference methods and describe our proposed deep MTL model in Section 3. In our experiments (Section 4), we will first validate our claim that MTL is superior to single-task learning for word emotion induction. After that, we will provide a large-scale evaluation of our model featuring 9 typologically diverse languages and multiple publicly available embedding models for a total of 15 conditions. Our MTL model surpasses the current state-of-the-art for each of them, and even performs competitive relative to human reliability. Most notably however, our approach yields the largest benefit on the smallest data sets, comprising merely one thousand samples. This finding, counterintuitive as it may be, strongly suggests that MTL is particularly beneficial for solving the word emotion induction problem. Our code base as well as the resulting experimental data is freely available.1

Footnote 1: [https://github.com/JULIELab/wordEmotions](https://github.com/JULIELab/wordEmotions)

## 2 Related Work

This section introduces the emotion representation format underlying our study and describes external resources we will use for evaluation before we discuss previous methodological work.

Emotion Representation and Data Sets.Psychological models of emotion can typically be subdivided into _discrete_ (or _categorical_) and _dimensional_ ones Stevenson et al. (2007); Calvo and Mac Kim (2013). Discrete models are centered around particular sets of emotional categories considered to be fundamental. Ekman (1992), for instance, identifies six _Basic Emotions_ Joy et al. (2006).

In contrast, dimensional models consider emotions to be composed of several influencing factors (mainly two or three). These are often referred to as _Valence_ (a positive-negative scale), _Arousal_ (a calm-excited scale), and _Dominance_ (perceived degree of control over a (social) situation)--the VAD model Bradley and Lang (1994); see Figure 1 for an illustration). Many contributions though omit Dominance (the VA model) Russell (1980). For convenience, we will still use the term "VAD" to jointly refer to both variants (with and without Dominance).

VAD is the most common framework to acquire empirical emotion values for words in psychology. Over the years, a considerable number of such resources (also called "emotion lexicons") have emerged from psychological research labs (as well as some NLP labs) for diverse languages. The emotion lexicons we use in our experiments are listed in Table 1. An even more extensive list of such data sets is presented by Buechel and Hahn (2018). For illustration, we also provide three sample entries from one of those lexicons in Table 2. As can be seen, the three affective dimensions behave complementary to each other, e.g., "terrorism" and "orgasm" display similar Arousal but opposing Valence.

The task we address in this paper is to predict the values for Valence, Arousal and Dominance, given a lexical item. As is obvious from these examples, we consider emotion prediction as a regression, not as a classification problem (see arguments discussed in Buechel and Hahn (2016)).

In this paper, we focus on the VAD format for the following reasons: First, note that the Valence dimension exactly corresponds to polarity Turney and Littman (2003). Hence, with the VAD model, emotion prediction can be seen as a generalization over classical polarity prediction. Second, to the best of our knowledge, the amount and diversity of available emotion lexicons with VAD encodings is larger than for any other format (see Table 1).

Word Embeddings.Word embeddings are dense, low-dimensional vector representations of words trained on large volumes of raw text in an unsupervised manner. The following are among today's most popular embedding algorithms:

Figure 1: Affective space spanned by the Valence-Arousal-Dominance (VAD) model, together with the position of six Basic Emotions; as determined by Russell and Mehrabian (1977).

 

word2vec (with its variants Sgns and Cbow) features an extremely trimmed down neural network Mikolov et al. (2013). FastText is a derivative of word2vec, also incorporating sub-word character n-grams Bojanowski et al. (2017). Unlike the former two algorithms which fit word embeddings in a streaming fashion, GloVe trains word vectors directly on a word co-occurrence matrix under the assumption to make more efficient use of word statistics Pennington et al. (2014). Somewhat similar, SVDpMI performs singular value decomposition on top of a point-wise mutual information co-occurrence matrix Levy et al. (2015).

In order to increase the reproducibility of our experiments, we rely on the following widely used, publicly available embedding models trained on very large corpora (summarized in Table 3): the Sgns model trained on the Google News corpus2 (Google), the FastText model trained on Common Crawl3 (Common), as well as the FastText models for a wide range of languages trained on the respective Wikipedias4 (Wiki). Note that Wiki denotes multiple embedding models with different training and vocabulary sizes (see Grave et al. (2018) for further details). Additionally, we were given the opportunity to reuse the English embedding model from Sedoc et al. (2017) Giga), a strongly related contribution (see below). Their embeddings were trained on the English Gigaword corpus Parker et al. (2011).

Footnote 2: [https://code.google.com/archive/p/word2vec/](https://code.google.com/archive/p/word2vec/)

Word-Level Prediction.One of the early approaches to word _polarity_ induction which is still popular today Koper and Schulte im Walde (2016) was introduced by Turney and Littman (2003). They compute the polarity of an unseen word based on its point-wise mutual information (PMI) to a set of positive and negative seed words, respectively.

SemEval-2015 Task 10E featured polarity induction on Twitter Rosenthal et al. (2015). The best system relied on support vector regression (SVR) using a radial base function kernel Amir et al. (2015). They employ the embedding vector of the target word as features. The results of their SVR-based system were beaten by the Densifier algorithm Rothe et al. (2016). Densifier learns an orthogonal transformation of an embedding space into a subspace of strongly reduced dimensionality.

Hamilton et al. (2016) developed SentProp, a graph-based, semi-supervised learning algorithm which builds up a word graph, where vertices correspond to words (of known as well as unknown polarity) and edge weights correspond to the similarity between them. The polarity information is then propagated through the graph, thus computing scores for unlabeled nodes. According to their evaluation, Densifier seems to be superior overall, yet SentProp produces competitive results

\begin{table}
\begin{tabular}{l l c r r} \hline \hline Source & ID & Language & Format & \# Entries \\ \hline Bradley and Lang (1999) & EN & English & VAD & 1,034 \\ Warriner et al. (2013) & EN+ & English & VAD & 13,915 \\ Redondo et al. (2007) & ES & Spanish & VAD & 1,034 \\ Stadthagen-Gonzalez et al. (2017) & ES+ & Spanish & VA & 14,031 \\ Schmidtle et al. (2014) & DE & German & VAD & 1,003 \\ Yu et al. (2016a) & ZH & Chinese & VA & 2,802 \\ Imbir (2016) & PL & Polish & VAD & 4,905 \\ Montefinese et al. (2014) & IT & Italian & VAD & 1,121 \\ Soares et al. (2012) & PT & Portuguese & VAD & 1,034 \\ Moors et al. (2013) & NL & Dutch & VAD & 4,299 \\ Sianipar et al. (2016) & ID & Indonesian & VAD & 1,490 \\ \hline \hline  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \end{tabular}
\end{table}
Table 1: Emotion lexicons used in our experiments (with their bibliographic source, identifier, language they refer to, emotion representation format, and number of lexical entries they contain).

\begin{table}
\begin{tabular}{|l||r|r|} \hline Word & Valence & Arousal & Dominance \\ \hline \hline sunshine & 8.1 & 5.3 & 5.4 \\ \hline terrorism & 1.6 & 7.4 & 2.7 \\ \hline organsm & 8.0 & 7.2 & 5.8 \\ \hline \end{tabular}
\end{table}
Table 2: Three sample entries from Warriner et al. (2013). They use 9-point scales ranging from 1 (most negative/calm/submissive) to 9 (most positive/excited/dominant).

 

only when the seed lexicon or the corpus the word embeddings are trained on is very small.5

Footnote 4: [https://github.com/wiki-lang/socialsent](https://github.com/wiki-lang/socialsent)

For word _emotion_ induction, a very similar approach to SentProp has been proposed by Wang et al. (2016a). They also propagate affective information Valence and Arousal, in this case) through a word graph with similarity weighted edges.

Sedoc et al. (2017) recently proposed an approach based on signed spectral clustering where a word graph is constructed not only based on word similarity but also on the considered affective information (again, Valence and Arousal). The emotion value of a target word is then computed based on the seed words in its cluster. They report to outperform the results from Wang et al. (2016a).

Contrary to the trend to graph-based methods, the best system of the IALP 2016 Shared Task on Chinese word emotion induction Yu et al. (2016b) employed a simple feed-forward neural network (FFNN) with one hidden layer in combination with boosting Du and Zhang (2016).

Another very recent contribution which advocates a supervised set-up was published by Li et al. (2017). They propose ridge regression, again using word embeddings as features. Even with this simple approach, they report to outperform many of the above methods in the VAD prediction task.6

Footnote 6: However, they also report extremely weak performance figures for some of their reference methods.

Sentence-Level and Text-Level PredictionDifferent from the word-level prediction task (the one we focus on in this contribution), the determination of emotion values for higher-level linguistic units (especially sentences and texts) is also heavily investigated. For this problem, DL approaches are meanwhile fully established as the method of choice Wang et al. (2016b); Abdul-Mageed and Ungar (2017); Felbo et al. (2017); Mohammad and Bravo-Marquez (2017).

It is important to note, however, that the methods discussed for these higher-level units cannot easily be transferred to solve the word emotion induction problem. Sentence-level and text-level architectures are either adapted to _sequential_ input data (typical for RNN, LSTM, GRNN and related architectures) or _spatially arranged_ input data (as with CNN architectures). However, for word embeddings (the default input for word emotion induction) there does not seem to be any meaningful order of their components. Therefore, these more sophisticated DL methods are, for the time being, not applicable for the study at hand.

## 3 Methods

In this section, we will first introduce various reference methods (two originally polarity-based for which we offer adaptations for VAD prediction) before defining our own neural MTL model and discussing its difference from previous work.

Let \(V:=\{w_{1},w_{2},...,w_{m}\}\) be our word vocabulary and let \(E:=\{e_{1},e_{2},...,e_{m}\}\) be a set of embedding vectors such that \(e_{i}\in\mathbb{R}^{n}\) denotes the \(n\)-dimensional vector representation of word \(w_{i}\). Let \(D:=\{d_{1},d_{2},...,d_{l}\}\) be a set of emotional dimensions. Our task is to predict the empirically determined emotion vector \( emo(w)\in\mathbb{R}^{l}\) given a word \(w\) and the embedding space \(E\).

### Reference Methods

Linear Regression Baseline (LinReg).We propose (multi-variate) linear regression as an obvious baseline for the problem:

\[ emo_{LR}(w_{k}):=We_{k}+b \tag{1}\]

where \(W\) is a matrix, \(W_{i*}\) contains the regression coefficients for the \(i\)-th affective dimension and \(b\) is the vector of bias terms. The model parameters are fitted using ordinary least squares. Technically, we use the scikit-learn.org implementation with default parameters.

\begin{table}
\begin{tabular}{l l l l l c c} \hline \hline ID & Language & Method & Corpus & \# Tokens & \# Types & \# Dimensions \\ \hline Google & English & Sons & Google News & \(1\times 10^{11}\) & \(3\times 10^{6}\) & \(300\) \\ Common & English & FastText & Common Crawl & \(6\times 10^{11}\) & \(2\times 10^{6}\) & \(300\) \\ Giga & English & Cbow & Gigawords & \(4\times 10\)9  & \(2\times 10^{6}\) & \(300\) \\ WikI & all & FastText & Wikipedia & — & — & \(300\) \\ \hline \hline \end{tabular}
\end{table}
Table 3: Embedding models used for our experiments with identifier, language, embedding algorithm, training corpus, its size in the number of tokens, size of the vocabulary (types) of the resulting embedding model and its dimensionality.

 

Ridge Regression (RidgReg).Li et al. (2017) propose ridge regression for word emotion induction. Ridge regression works identically to linear regression during prediction, but introduces \(L_{2}\) regularization during training. Following the authors, for our implementation, we again use the scikit-learn implementation with default parameters.

Turney-Littman Algorithm (TL).As one of the earliest contributions in the field, Turney and Littman (2003) defined a simple PMI-based approach to determine the semantic polarity \(SP_{TL}\) of a word \(w\):

\[SP_{TL}(w)\coloneqq\!\!\sum_{s\in seeds^{+}}\!pmi(w,s)\!-\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\! 

step, since we did not find any evidence that this procedure actually results in improved performance. The hyperparameters \(\alpha\) and \(\beta\) were set to \(.7\) and \(.5\) (respectively) for all experiments based on a pilot study. Since the original implementation is not accessible, we devised our own using tensorflow.org.

Boosted Neural Networks (ensembleNN).Du and Zhang (2016) propose simple FFNNs in combination with a boosting algorithm. An FFNN consists of an _input_ or _embedding layer_ with activation \(a^{(0)}\in\mathbb{R}^{n}\) which is equal to the embedding vector \(e_{k}\) when predicting the emotion of a word \(w_{k}\). The input layer is followed by multiple hidden layers with activation

\[a^{(l+1)}\coloneqq\sigma(W^{(l+1)}a^{(l)}+b^{(l+1)}) \tag{10}\]

where \(W^{(l+1)}\) and \(b^{(l+1)}\) are the weights and biases for layer \(l+1\) and \(\sigma\) is a nonlinear activation function. Since we treat emotion prediction as a regression problem, the activation on the output layer \(a^{out}\) (where \(out\) is the number of non-input layers in the network) is computed as the affine transformation

\[a^{(out)}\coloneqq W^{(out)}a^{(out-1)}+b^{(out)} \tag{11}\]

Boosting is a general machine learning technique where several weak estimators are combined to form a strong estimator. The authors used FFNNs with a single hidden layer of \(100\) units and rectified linear unit (ReLU) activation. The boosting algorithm AdaBoost.R2 Drucker (1997) was used to train the ensemble (one per affective dimension). Our re-implementation copies their technical set-up7 exactly using scikit-learn.

Footnote 7: Original settings available at [https://github.com/StevenLOL/ialp2016_Shared_Task](https://github.com/StevenLOL/ialp2016_Shared_Task)

### Multi-Task Learning Neural Network

The approaches introduced in Section 3.1 and Section 2 vary largely in their methodological foundations, i.e., they comprise semi-supervised and supervised machine learning techniques--both statistical and neural ones. Yet, they all have in common that they treat the prediction of the different emotional dimensions _as separate_ tasks. That is, they fit one individual model per VAD dimension without sharing parameters between them.

In contradistinction, the key feature of our approach is that we fit a single FFNN model to predict _all VAD dimensions jointly_, thus applying multi-task learning to word emotion induction. Hence, we treat the prediction of Valence, Arousal and Dominance as three independent tasks. Our multi-task learning neural network (MTLNN) depicted in Figure 2) has an output layer of three units such that each output unit represents one of the VAD dimensions. However, the activation in our two hidden layers (of 256 and 128 units, respectively) is _shared_ across all VAD dimensions, and so are the associated weights and biases.

Thus, while we train our MTLNN model it is forced to learn intermediate representations of the input which are generally informative for all VAD dimensions. This serves as a form of regularization, since it becomes less likely for our model to fit the noise in the training set as noise patterns may vary across emotional dimensions. Simultaneously, this has an effect similar to an increase of the training size, since each sample now leads to additional error signals during backpropagation. Intuitively, both properties seem extremely useful for relatively small-sized emotion lexicons (see Section 4 for empirical evidence).

The remaining specifications of our model are as follows. We use _leaky_ ReLU activation (LReLU) as nonlinearity Maas et al. (2013).

\[LReLU(z_{i})\coloneqq max(\gamma z_{i},z_{i}) \tag{12}\]

with \(\gamma\coloneqq.01\) for our experiments. For regularization, dropout Srivastava et al. (2014) is applied during training with a probability of \(.2\) on the embedding layer and \(.5\) on the hidden layers. We train for \(15,000\) iterations (well beyond convergence on each data set we use) with the Adam optimizer Kingma and Ba (2015) of \(.001\) base learning rate,

Figure 2: MTL architecture for VAD prediction.

 

batch size of \(128\) and Mean-Squared-Error loss. The weights are randomly initialized (drawn from a normal distribution with a standard deviation \(.001\)) and biases are uniformly initialized as \(.01\). Tensorflow is used for implementation.

## 4 Results

In this section, we first validate our assumption that MTL is superior to single-task learning for word emotion induction. Next, we compare our proposed MTLNN model in a large-scale evaluation experiment.

Performance figures will be measured as Pearson correlation (\(r\)) between our automatically predicted values and human gold ratings. The Pearson correlation between two data series \(X=x_{1},x_{2},...,x_{n}\) and \(Y=y_{1},y_{2},...,y_{n}\) takes values between \(+1\) (perfect positive correlation) and \(-1\) (perfect negative correlation) and is computed as

\[r_{xy}:=\frac{\sum_{i=1}^{n}(x_{i}-\bar{x})(y_{i}-\bar{y})}{\sqrt{\sum_{i=1}^{ n}(x_{i}-\bar{x})^{2}}~{}\sqrt{\sum_{i=1}^{n}(y_{i}-\bar{y})^{2}}} \tag{13}\]

where \(\bar{x}\) and \(\bar{y}\) denote the mean values for \(X\) and \(Y\), respectively.

### Single-Task vs. Multi-Task Learning

The main hypothesis of this contribution is that an MTL set-up is superior to single-task learning for word emotion induction. Before proceeding to the large-scale evaluation of our proposed model, we will first examine this aspect of our work.

For this, we use the following experimental setup: We will compare the MTLNN model against its single-task learning counterpart (SepNN). SepNN simultaneously trains three separate neural networks where only the input layer, yet no parameters of the intermediate layers are shared across the models. Each of the separate networks is identical to MTLNN (same layers, dropout, initialization, etc.), yet has only one output neuron, thus modeling only one of the three affective VAD dimensions. SepNN is equivalent to fitting our proposed model (but with only one output unit) to the different VAD dimensions individually, one after the other. Yet, training these separate networks simultaneously (not jointly!) makes both approaches, MTLNN and SepNN, easier to compare.

We will run MTLNN against SepNN on the EN and the EN+ data set (the former is very small, the latter relatively large; see Table 1) using the following set-up: for each gold lexicon and model, we randomly split the data \(9/1\) and train for \(15,000\) iterations on the larger split (the same number of steps is used for the main experiment). After each one-thousand iterations step, model performance is tested on the held-out data. This process will be repeated \(20\) times and the performance figures at each one-thousand iterations step will be averaged. In a final step, we will average the results for each of the three emotional dimensions and only plot this average value. The results of this experiment are depicted in Figure 3.

First of all, each combination of model and data set displays a satisfactory performance of at least \(r\approx.75\) after 15,000 steps compared to previous work (see below). Overall, performance is higher for the smaller EN lexicon. Although counterintuitive (since smaller lexicons lead to fewer training samples), this finding is consistent with prior work Sedoc et al. (2017); Li et al. (2017) and is probably related to the fact that smaller lexicons usually comprise a larger portion of strongly emotionbearing words. In contrast, larger lexicons add more neutral words which tend to be harder to predict in terms of correlation.

As hypothesized, the MTLNN model does indeed outperform the single task model on both data sets. Our data also suggest that the gain from the MTL approach is larger on smaller data sets (again in concordance with our expectations). Figure 3 reveals that this might be due to the regularizing effect of MTL, since the SepNN model shows signs of overfitting on the EN data set. Yet, even

Figure 3: Performance of our proposed MTLNN model vs. its single-task learning counterpart SepNN against training steps.

 

when the separate model does not overfit (as on the EN+ lexicon), MTLNN reveals better results.

Although SepNN needs fewer _training steps_ before convergence, the MTLNN model trains much faster, thus still converging faster in terms of _runtime_ (about a minute on a middle-class GPU). This is because MTLNN has only about a third as many parameters as the separate model SepNN.

### Comparison against Reference Methods

We combined each of the selected lexicon data sets (Table 1) with each of the applicable publicly available embedding models (Section 2; the embedding model provided by Sedoc et al. (2017) will be used separately) for a total of 15 conditions, i.e, the rows in Table 4.

For each of these conditions, we performed a 10-fold cross-validation (CV) for each of the 6 methods presented in Section 3 such that each method is presented with the identical data splits.8 For each condition, algorithm, and VA(D) dimension, we compute the Pearson correlation \(r\) between gold ratings and predictions. For conciseness, we present only the average correlation over the respective affective dimensions in Table 4 (Valence and Arousal for ES+ and ZH, VAD for the others). Note that the methods we compare ourselves against comprise the current state-of-the art in both polarity and emotion induction (as described in Section 2).

Footnote 8: This procedure constitutes a more direct comparison than using different splits for each method and allows using _paired_\(t\)-tests.

As can be seen, our proposed MTLNN model outperforms all other approaches in each of the 15 conditions. Regarding the average over all affective dimensions and conditions, it outperforms the second best system, ensembleNN, by more than \(5\%\)-points. In line with our results from Section 4.1, those improvements are especially pronounced on smaller data sets containing one up to two thousand entries (EN, ES, IT, PT, ID) with close to \(10\%\)-points improvement over the respective second-best system.

Concerning the relative ordering of the affective dimensions, in line with former studies (Sedoc et al., 2017; Li et al., 2017), the performance figures for the Valence dimension are usually much higher than for Arousal and Dominance. Using MTLNN, for many conditions, we see the pattern that Valence is about \(10\%\)-points above the VAD average, Arousal being \(10\%\)-points below and Dominance being roughly equal to the average over VAD (this applies, e.g., to EN, EN+ and IT). On other data sets (e.g., PL, NL and ID), the ordering between Arousal and Dominance is less clear though Valence still stands out with the best results. We observe the same general pattern for the reference methods, as well.

Concerning the comparison to Sedoc et al. (2017), arguably one of most related contributions, they report a performance of \(r=.768\) for Valence and \(.582\) for Arousal on the EN+ data set in a 10-fold CV using their own embeddings. In contrast, MTLNN using the Common model achieves \(r=.870\) and \(.674\) in the same set-up--about \(10\%\)

\begin{table}
\begin{tabular}{l l r c c c c c c} \hline \hline Language & Data & Embeddings & LinReg & RidgReg & TL & Densifier & ensembleNN & MTLNN \\ \hline English & EN+ & Google & 0.696 & 0.696 & 0.631 & 0.622 & 0.728 & **0.739*** \\ English & EN+ & Common & 0.719 & 0.719 & 0.659 & 0.652 & 0.762 & **0.767*** \\ English & EN+ & Wiki & 0.666 & 0.666 & 0.591 & 0.584 & 0.706 & **0.712*** \\ English & EN & Goodle & 0.717 & 0.732 & 0.723 & 0.712 & 0.688 & **0.810*** \\ English & EN & Common & 0.731 & 0.741 & 0.741 & 0.726 & 0.717 & **0.824*** \\ English & EN & Wiki & 0.656 & 0.667 & 0.674 & 0.665 & 0.681 & **0.777*** \\ Spanish & ES & Wiki & 0.698 & 0.709 & 0.704 & 0.690 & 0.700 & **0.804*** \\ Spanish & ES+ & Wiki & 0.693 & 0.694 & 0.603 & 0.598 & 0.766 & **0.778*** \\ German & DE & Wiki & 0.709 & 0.719 & 0.714 & 0.710 & 0.700 & **0.801*** \\ Chinese & ZH & Wiki & 0.716 & 0.717 & 0.586 & 0.599 & 0.737 & **0.744*** \\ Polish & PL & Wiki & 0.650 & 0.650 & 0.577 & 0.553 & 0.687 & **0.712*** \\ Italian & IT & Wiki & 0.656 & 0.665 & 0.672 & 0.659 & 0.630 & **0.751*** \\ Portuguese & PT & Wiki & 0.673 & 0.684 & 0.685 & 0.678 & 0.672 & **0.768*** \\ Dutch & NL & Wiki & 0.651 & 0.652 & 0.559 & 0.532 & 0.704 & **0.730*** \\ Indonesian & ID & Wiki & 0.581 & 0.586 & 0.581 & 0.576 & 0.575 & **0.660*** \\ \hline Average & & & & 0.638 & 0.659 & 0.611 & 0.605 & 0.676 & **0.728*** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Results of our main experiment in averaged Pearson correlation; best result per condition (in rows) in bold, second best result underlined; significant difference (paired two-tailed \(t\)-test) over the second best system marked with “*”, ‘**’, or “***” for \(p<.05\), \(.01\), or \(.001\), respectively.

 

points better on both dimensions. However, the Common model was trained on much more data than the embeddings Sedoc et al. (2017) use. For the most direct comparison, we also repeated this experiment using _their_ embedding model (Giga). We find that MTLNN still clearly outperforms their results with \(r=.814\) for Valence and \(.607\) for Arousal.9

Footnote 9: We also clearly outperform their results for the NL and ES+ data sets. For these cases, our embedding models were similar in training size.

MTLNN achieves also very strong results in direct comparison to human performance (see Table 5). Warriner et al. (2013) (who created EN+) report an inter-study reliability (ISR; i.e., the correlation of the aggregated ratings from two different studies) between the EN and the EN+ lexicon of \(r=.953\), \(.759\) and \(.795\) for VAD, respectively. Since EN is a subset of EN+, we can compare these performance figures against our own results on the EN data set where we achieved \(r=.918\), \(.730\) and \(.825\), respectively. Thus, our proposed method did actually outperform human reliability for Dominance and is competitive for Valence and Arousal, as well.

This general observation is also backed up by split-half reliability data (SHR; i.e., when randomly splitting all individual ratings in two groups and averaging the ratings within each group, how strong is the correlation between these averaged ratings?). For the EN+ data set, Warriner et al. (2013) report an SHR of \(r=.914\), \(.689\) and \(.770\) for VAD, respectively. Again, our MTLNN model performs very competitive with \(r=.870\), \(.674\) and \(.758\), respectively using the Common embeddings.

## 5 Conclusion

In this paper, we propose multi-task learning (MTL) as a simple, yet surprisingly efficient method to improve the performance and, at the same time, to deal with existing data limitations in word emotion induction--the task to predict a complex emotion score for an individual word. We validated our claim that MTL is superior to single-task learning by achieving better results with our proposed method in performance as well as training time compared to its single-task counterpart. We performed an extensive evaluation of our model on 9 typologically diverse languages, using different kinds of word embedding models for a total 15 conditions. Comparing our approach to state-of-the-art methods from word polarity and word emotion induction, our model turns out to be superior in each condition, thus setting a novel state-of-the-art performance for both polarity _and_ emotion induction. Moreover, our results are even competitive to human annotation reliability in terms of inter-study as well as split-half reliability. Since this contribution was restricted to the VAD format of emotion representation, in future work we will examine whether MTL yields similar gains for other representational schemes, as well.

## Acknowledgments

We would like to thank the Positive Psychology Center, University of Pennsylvania for providing us with the embedding model used in Sedoc et al. (2017), Johannes Hellrich, Julie Lab, for insightful discussions, and the reviewers for their valuable comments.

## References

* Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics_. Vancouver, British Columbia, Canada, July 30
- August 4, 2017, volume 1: Long Papers, pages 718-728.
* Amir et al. (2015) Silvio Amir, Ramon F. Astudillo, Wang Ling, Bruno Martins, Mario J. Silva, and Isabel Trancoso. 2015. Inesc-Id: A regression model for large scale Twitter sentiment lexicon induction. In _SemEval 2015 -- Proceedings of the 9th International Workshop on Semantic Evaluation @ NAACL-HLT 2015_. Denver, Colorado, USA, June 4-5, 2015, pages 613-618.
* Baxter (1997) Jonathan Baxter. 1997. A Bayesian/information theoretic model of learning to learn via multiple task sampling. _Machine Learning_ 28(1):7-39.
* Bojanowski et al. (2017) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. _Transactions of the Association for Computational Linguistics_ 5(1):135-146.
* Berman et al. (2017)

\begin{table}
\begin{tabular}{|l|r|r|r|} \hline  & Valence & Arousal & Dominance \\ \hline \hline MTLNN EN & .918 & .730 & .825 \\ MTLNN EN+ & .870 & .674 & .758 \\ \hline ISR EN \(\sim\) EN+ & .953 & .759 & .795 \\ SHR EN+ & .914 & .689 & .770 \\ \hline \end{tabular}
\end{table}
Table 5: Comparison of the MTLNN model against inter-study reliability (ISR) between the EN and the EN+ data set and split-half reliability (SHR) of the EN+ data set (in Pearson correlation).

 

* M. Bradley and J. Lang (1994)Measuring emotion: The Self-Assessment Manikin and the semantic differential. Journal of Behavior Therapy and Experimental Psychiatry25 (1), pp. 49-59. Cited by: SS3.
* M. Bradley and P. J. Lang (1999)Affective Norms for English Words (Anew): Stimuli, instruction manual and affective ratings. Technical Report C-1, The Center for Research in Psychology, University of Florida, Gainesville, FL. External Links: ISBN 078-1-695-381-1, Link, Document Cited by: SS1.
* Proceedings of the 22nd European Conference on Artificial Intelligence, Including Prestigious Applications of Artificial Intelligence (PAIS 2016), The Hague, Netherlands, August 29
- September 2, 2016, Vol. 285, pp. 1114-1122. External Links: ISSN 078-1-695-381-1, Link, Document Cited by: SS1.
* S. Buechel and U. Hahn (2017)EmoBank: Studying the impact of annotation perspective and representation format on dimensional emotion analysis. In EACL 2017 -- Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, Vol. , Vol. , pp. 3-7. External Links: Document Cited by: SS1.
* S. Buechel and U. Hahn (2018)Representation mapping: A novel approach to generate high-quality multi-lingual emotion lexicons. In LAREC 2018 -- Proceedings of the 11th International Conference on Language Resources and Evaluation, Miyazaki, Japan, May 7-12, Republic, Korea, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic,, Republic, Republic, Republic, Republic, Republic, Republic, Republic, Republic,, Republic, Republic, Republic, Republic, Republic, Republic,, Republic, Republic, Republic, Republic, Republic,, Republic, Republic, Republic, Republic, Republic,, Republic, Republic, Republic, Republic,, Republic, Republic, Republic, Republic,, Republic, Republic, Republic,, Republic, Republic, Republic,, Republic, Republic, Republic,, Republic, Republic,, Republic, Republic, Republic,, Republic, Republic,, Republic, Republic,, Republic, Republic,, Republic, Republic,, Republic, Republic,, Republic, Republic,, Republic, Republic,, Republic, Republic,, Republic, Republic,, Republic, Republic,, Republic, Republic,, Republic, Republic,, Republic, Republic,, Republic,, Republic, Republic,, Republic, Republic,, Republic,, Republic, Republic,, Republic,, Republic, Republic,, Republic,, Republic,, Republic, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,, Republic,,, Republic,, Republic,, Republic,, Republic,,, Republic,, Republic,, Republic,, Republic,,, Republic,, Republic,,, Republic,, Republic,,, Republic,, Republic,,, Republic,, Republic,,, Republic,, Republic,,, Republic,, Republic,,, Republic,,, Republic,,, Republic,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,, Republic,,, Republic,,, Republic,,, Republic,,, Republic,, Republic,,, Republic,,, Republic,,, Republic,, Republic,,, Republic,, Republic,,, Republic,, Republic,,, Republic,, Republic,,, Republic,, Republic,,, Republic,, Republic,,, Republic,, Republic,, Republic,,, Republic,, Republic,, Republic,, Republic,,, Republic,, 

PengFei Liu, Xipeng Qiu, and Xuanjing Huang. 2017. Adversarial multi-task learning for text classification. In _ACL 2017 -- Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics_. Vancouver, British Columbia, Canada, July 30 - August 4, 2017, volume 1: Long Papers, pages 1-10.
* Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Denver, Colorado, USA, May 31
- June 5, 2015, pages 912-921.
* Maas et al. (2013) Andrew L. Maas, Awni Y. Hannun, and Andrew Y. Ng. 2013. Rectifier nonlinearities improve neural network acoustic models. In _Proceedings of the Workshop on Deep Learning for Audio, Speech and Language Processing @ ICML 2013_. Atlanta, Georgia, USA, 16 June 2013.
* Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In _Advances in Neural Information Processing Systems 26 -- NIPS 2013. Proceedings of the 27th Annual Conference on Neural Information Processing Systems_. Lake Tahoe, Nevada, USA, December 5-10, 2013, pages 3111-3119.
* Mohammad and Bravo-Marquez (2017) Saif M. Mohammad and Felipe Bravo-Marquez. 2017. WASSA-2017 Shared Task On Emotion Intensity. In _WASSA 2017 -- Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis @ EMNLP 2017_. Copenhagen, Denmark, September 8, 2017, pages 34-49.
* Montefinese et al. (2014) Maria Montefinese, Ettore Ambrosini, Beth Fairfield, and Nicola Mammarella. 2014. The adaptation of the Affective Norms for English Words (Anew) for Italian. _Behavior Research Methods_ 46(3):887-903.
* Moors et al. (2013) Agnes Moors, Jan De Houwer, Dirk Hermans, Sabine Wanmaker, Kevin Van Schie, Anne-Laura Van Harmelen, Maarten De Schryver, Jeffrey De Winne, and Marc Brysbaert. 2013. Norms of valence, arousal, dominance, and age of acquisition for 4,300 Dutch words. _Behavior Research Methods_ 45(1):169-177.
* Parker et al. (2011) Robert Parker, David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. 2011. English Gigaword Fifth Edition. Technical Report LDC2011T07, Linguistic Data Consortium, Philadelphia, Pennsylvania, USA. [https://catalog.ldc.upenn.edu/LDC2011T07](https://catalog.ldc.upenn.edu/LDC2011T07).
* Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics_. Vancouver, British Columbia, Canada, July 30
- August 4, 2017, volume 1: Long Papers, pages 2037-2048.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global vectors for word representation. In _EMNLP 2014 -- Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing_. Doha, Qatar, October 25-29, 2014, pages 1532-1543.
* Redondo et al. (2007) Jaime Redondo, Isabel Fraga, Isabel Padron, and Montserrat Comesana. 2007. The Spanish Adaptation of Anew (Affective Norms for English Words). _Behavior Research Methods_ 39(3):600-605.
* Rosenthal et al. (2015) Sara Rosenthal, Preslav I. Nakov, Svetlana Kiritchenko, Saif M. Mohammad, Alan Ritter, and Veselin Stoyanov. 2015. SemEval 2015 Task 10: Sentiment analysis in Twitter. In _SemEval 2015 -- Proceedings of the 9th International Workshop on Semantic Evaluation @ NAACL-HLT 2015_. Denver, Colorado, USA, June 4-5, 2015, pages 451-463.
* Rothe et al. (2016) Sascha Rothe, Sebastian Ebert, and Hinrich Schutze. 2016. Ultradense word embeddings by orthogonal transformation. In _NAACL-HLT 2016 -- Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. San Diego, California, USA, June 12-17, 2016, pages 767-777.
* Russell (1980) James A. Russell. 1980. A circumplex model of affect. _Journal of Personality and Social Psychology_ 39(6):1161-1178.
* Russell and Mehrabian (1977) James A. Russell and Albert Mehrabian. 1977. Evidence for a three-factor theory of emotions. _Journal of Research in Personality_ 11(3):273-294.
* Schmidtke et al. (2014) David S. Schmidtke, Tobias Schroder, Arthur M. Jacobs, and Markus Conrad. 2014. Angst: Affective Norms for German Sentiment Terms, derived from the Affective Norms for English Words. _Behavior Research Methods_ 46(4):1108-1118.
* Sedoc et al. (2017) Joao Sedoc, Daniel Preotiuc-Pietro, and Lyle H. Ungar. 2017. Predicting emotional word ratings using distributional representations and signed clustering. In _EACL 2017 -- Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics_. Valencia, Spain, April 3-7, 2017, volume 2: Short Papers, pages 564-571.
* Setiawan et al. (2015) Hendra Setiawan, Zhongqiang Huang, Jacob Devlin, Thomas Lamar, Rabih Zbib, Richard M. Schwartz, and John Makhoul. 2015. Statistical machine translation features with multitask tensor networks. In _ACL-IJCNLP 2015 -- Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics & 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing_. Beijing, China, July 26-31, 2015, volume 1: Long Papers, pages 31-41.

 

Agnes Sianipar, Pieter van Groenestijn, and Ton Dijkstra. 2016. Affective meaning, concreteness, and subjective frequency norms for Indonesian words. _Frontiers in Psychology_ 7:#1907.
* Soares et al. (2012) Ana Paula Soares, Montserrat Comesana, Ana P Pinheiro, Alberto Simoes, and Carla Sofia Frade. 2012. The adaptation of the Affective Norms for English Words (Anew) for European Portuguese. _Behavior Research Methods_ 44(1):256-269.
* Sogaard and Goldberg (2016) Anders Sogaard and Yoav Goldberg. 2016. Deep multi-task learning with low level tasks supervised at lower layers. In _ACL 2016 -- Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Berlin, Germany, August 7-12, 2016, volume 2: Short Papers, pages 231-235.
* Srivastava et al. (2014) Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan R. Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. _Journal of Machine Learning Research_ 15:1929-1958.
* Stadthagen-Gonzalez & Imbault (2017) Hans Stadthagen-Gonzalez, Constance Imbault, Miguel A. Perez-Sanchez, and Marc Brysbaert. 2017. Norms of valence and arousal for 14,031 Spanish words. _Behavior Research Methods_ 49(1):111-123.
* Stevenson et al. (2007) Ryan A. Stevenson, Joseph A. Mikels, and Thomas W. James. 2007. Characterization of the affective norms for English words by discrete emotional categories. _Behavior Research Methods_ 39(4):1020-1024.
* Turney and Littman (2003) Peter D. Turney and Michael L. Littman. 2003. Measuring praise and criticism: Inference of semantic orientation from association. _ACM Transactions on Information Systems_ 21(4):315-346.
* Wang et al. (2016a) Jin Wang, Liang-Chih Yu, K. Robert Lai, and Xuejie Zhang. 2016a. Community-based weighted graph model for valence-arousal prediction of affective words. _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ 24(11):1957-1968.
* Wang et al. (2016b) Jin Wang, Liang-Chih Yu, K. Robert Lai, and Xuejie Zhang. 2016b. Dimensional sentiment analysis using a regional CNN-LSTM model. In _ACL 2016 -- Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics_. Berlin, Germany, August 7-12, 2016, volume 2: Short Papers, pages 225-230.
* Warriner et al. (2013) Amy Beth Warriner, Victor Kuperman, and Marc Brysbbert. 2013. Norms of valence, arousal, and dominance for 13,915 English lemmas. _Behavior Research Methods_ 45(4):1191-1207.
* Yu et al. (2016a) Liang-Chih Yu, Lung-Hao Lee, Shuai Hao, Jin Wang, Yunchao He, Jun Hu, K. Robert Lai, and Xuejie Zhang. 2016a. Building Chinese affective resources in valence-arousal dimensions. In _NAACL-HLT 2016 -- Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. San Diego, California, USA, June 12-17, 2016, pages 540-545.
* Yu et al. (2016b) Liang-Chih Yu, Lung-Hao Lee, and Kam-Fai Wong. 2016b. Overview of the IALP 2016 Shared Task on dimensional sentiment analysis for Chinese words. In _IALP 2016 -- Proceedings of the [20th] 2016 International Conference on Asian Language Processing_. Tainan, Taiwan, November 21-23, 2016, pages 156-160.