Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper presents a method for classifying Tumblr posts with associated images according to associated single emotion word hashtags.  The method relies on sentiment pre-processing from GloVe and image pre-processing from Inception.  
 
My strongest criticism for this paper is against the claim that Tumblr post represent self-reported emotions and that this method sheds new insight on emotion representation and my secondary criticism is a lack of novelty in the method, which seems to be simply a combination of previously published sentiment analysis module and previously published image analysis module, fused in an output layer. 

The authors claim that the hashtags represent self-reported emotions, but this is not true in the way that psychologists query participants regarding emotion words in psychology studies.  Instead these are emotion words that a person chooses to broadcast along with an associated announcement.  As the authors point out, hashtags and words may be used sarcastically or in different ways from what is understood in emotion theory.  It is quite common for everyday people to use emotion words this way e.g. using #love to express strong approval rather than an actual feeling of love.   

In their analysis the authors claim:
“The 15 emotions retained were those with high relative frequencies on Tumblr among the PANAS-X scale (Watson & Clark, 1999)”.
However five of the words the authors retain: bored, annoyed, love, optimistic, and pensive are not in fact found in the PANAS-X scale:

Reference: The PANAS-X Scale: https://wiki.aalto.fi/download/attachments/50102838/PANAS-X-scale_spec.pdf Also the longer version that the authors cited: 
https://www2.psychology.uiowa.edu/faculty/clark/panas-x.pdf

It should also be noted that the PANAS (Positive and Negative Affect Scale) scale and the PANAS-X (the “X” is for eXtended) scale are questionnaires used to elicit from participants feelings of positive and negative affect, they are not collections of "core" emotion words, but rather words that are colloquially attached to either positive or negative sentiment.  For example PANAS-X includes words like:“strong” ,“active”, “healthy”, “sleepy” which are not considered emotion words by psychology.  

If the authors stated goal is "different than the standard sentiment analysis goal of predicting whether a sentence expresses positive or negative sentiment" they should be aware that this is exactly what PANAS is designed to do - not to infer the latent emotional state of a person, except to the extent that their affect is positive or negative.


The work of representing emotions had been an field in psychology for over a hundred years and it is still continuing.  https://en.wikipedia.org/wiki/Contrasting_and_categorization_of_emotions.

One of the most popular theories of emotion is the theory that there exist “basic” emotions: Anger, Disgust, Fear, Happiness (enjoyment), Sadness and Surprise (Paul Ekman, cited by the authors).  These are short duration sates lasting only seconds.  They are also fairly specific, for example “surprise” is sudden reaction to something unexpected, which is it exactly the same as seeing a flower on your car and expressing “what a nice surprise.”  The surprise would be the initial reaction of “what’s that on my car?  Is it dangerous?” but after identifying the object as non-threatening, the emotion of “surprise” would likely pass and be replaced with appreciation.  

The Circumplex Model of Emotions (Posner et al 2005) the authors refer to actually stands in opposition to the theories of Ekman.  From the cited paper by Posner et al : 
"The circumplex model of affect proposes that all affective states arise from cognitive interpretations of core neural sensations that are the product of two independent neurophysiological systems. This model stands in contrast to theories of basic emotions, which posit that a discrete and independent neural system subserves every emotion."
From my reading of this paper, it is clear to me that the authors do not have a clear understanding of the current state of psychology’s view of emotion representation and this work would not likely contribute to a new understanding of the latent structure of peoples’ emotions.

In the PCA result, it is not "clear" that the first axis represents valence, as "sad" has a slight positive on this scale and "sad" is one of the emotions most clearly associated with negative valence.

With respect to the rest of the paper, the level of novelty and impact is "ok, but not good enough."  This analysis does not seem very different from Twitter analysis, because although Tumblr posts are allowed to be longer than Twitter posts, the authors truncate the posts to 50 characters.  Additionally, the images do not seem to add very much to the classification.  The authors algorithm also seems to be essentially a combination of two other, previously published algorithms.

For me the novelty of this paper was in its application to the realm of emotion theory, but I do not feel there is a contribution here.  This paper is more about classifying Tumblr posts according to emotion word hashtags than a paper that generates a new insights into emotion representation or that can infer latent emotional state. 












review_title: Hashtag classification of Tumblr Posts 

final_score: rating: 6: Marginally above acceptance threshold
review_content: 
The authors present a study that aims at inferring the "emotional" tags provided by Thumblr users starting from images and texts in the captions. For text processing the authors use a standard LSTM taking as input GLOVE vectors of words in a sentence. For visual information, authors use a pretrained CNN (with fine tuning). A fully connected layer is used to fuse the multimodal information. Experimental results are reported in a self generated data set. 

The contribution from the RL perspective is limited, in the sense that the authors simply applied standard models to predict a bunch of labels (in this case, emotion labels). It is interesting the "psychological" analysis that the authors present in Section 6. Still, I think the contribution in that part is a: sentiment-psychologically inspired analysis of the Thumbrl data set. 

I think the author's statement on that this study leads to a more plausible psychological model of emotion is not well founded (they also mention to learn to recognize the latent emotional state). Whereas it is true that psychological studies rely on self - filled questionnaires, comparing a questionnaire (produced by expert psychologist) to the tags provided by users in a social network is to ambitious. (in some parts the authors make explicit this is an approximation, this should be stressed in every part of the paper)

review_title: The authors present a study that aims at inferring the "emotional" tags provided by Thumblr users starting from images and texts in the captions. 

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper presents a multi-modal CNN model for sentiment analysis that combines images and text.  The model is trained on a new dataset collected from Tumblr.

Positive aspects:
+ Emphasis in model interpretability and its connection to psychological findings in emotions
+ The idea of using Tumblr data seems interesting, allowing to work with a large set of emotion categories, instead of considering just the binary task positive vs. negative. 

Weaknesses:
- A deeper analysis of previous work on the combination of image and text for sentiment analysis (both datasets and methods) and its relation with the presented work is necessary. 
- The proposed method is not compared with other methods that combine text and image for sentiment analysis.
-  The study is limited to just one dataset.

The paper presents interesting ideas and findings in an important challenging area. The main novelties of the paper are: (1) the use of Tumblr data, (2) the proposed CNN architecture, combining images and text (using word embedding. 

I missed a "related work section", where authors clearly mention previous works on similar datasets. Some related works are mentioned in the paper, but those are spread in different sections. It's hard to get a clear overview of the previous research: datasets, methods and contextualization of the proposed approach in relation with previous work. I think authors should cite Sentibanks. Also, at some point authors should compare their proposal with previous work. 

More comments:

- Some figures could be more complete: to see more examples in Fig 1, 2, 3 would help to understand better the dataset and the challenges. 
- In table 4, for example, it would be nice to see the performance on the different emotion categories.
- It would be interesting to see qualitative visual results on recognitions.

I like this work, but I think authors should improve the aspects I mention for its publication.

review_title: Interesting paper on sentiment analysis combining image and text 

Meta-review: This work combines words and images from Tumblr to provide more fine-grained sentiment analysis than just positive-negative. The contribution is too slight, as a straightforward combination of existing architectures applied on an emotion classification task with conclusions that aren't well motivated and are not providing any comparison to existing related work on finer emotion classification.
Decision: Reject