IMAGE SENTIMENT ANALYSIS USING LATENTCORRELATIONS AMONG VISUAL, TEXTUAL, ANDSENTIMENT VIEWS
M. Katsurai and S. SatohCopyright 2016 IEEE. Published in the IEEE 2016 International Conferenceon Acoustics, Speech, and Signal Processing (ICASSP 2016), scheduled for
20-15 March 2016 in Shanghai, China. Personal use of this material ispermitted. However, permission to reprint/republish this material foradvertising or promotional purposes or for creating new collective works forresale or redistribution to servers or lists, or to reuse any copyrighted
component of this work in other works, must be obtained from the IEEE.
Contact: Manager, Copyrights and Permissions / IEEE Service Center / 445
Hoes Lane / P.O. Box 1331 / Piscataway, NJ 08855-1331, USA. Telephone: +Intl. 908-562-3966.

IMAGE SENTIMENT ANALYSIS USING LATENT CORRELATIONS AMONG VISUAL,TEXTUAL, AND SENTIMENT VIEWSMarie Katsurai

Shin’ichi Satoh

Department of Information Systems Design
Doshisha University
Kyoto, Japankatsurai@mm.doshisha.ac.jp

Digital Content and Media Sciences Research
Division, National Institute of Informatics
Tokyo, Japansatoh@nii.ac.jp

ABSTRACTAs Internet users increasingly post images to express their daily sentiment and emotions, the analysis of sentiments in user-generated
images is of increasing importance for developing several applications. Most conventional methods of image sentiment analysis focus
on the design of visual features, and the use of text associated tothe images has not been sufficiently investigated. This paper proposes a novel approach that exploits latent correlations among multiple views: visual and textual views, and a sentiment view constructed using SentiWordNet. In the proposed method, we find alatent embedding space in which correlations among the three views
are maximized. The projected features in the latent space are used
to train a sentiment classifier, which considers the complementary
information from di↵erent views. Results of experiments conducted
on Flickr and Instagram images show that our approach achieves
better sentiment classification accuracy than methods that use a single modality only and the state-of-the art method that jointly uses
multiple modalities.
Index Terms— image sentiment analysis, multi-view embedding, canonical correlation analysis, SentiWordNet
1. INTRODUCTION
With the popularity of image capturing devices and social media
platforms, we have seen a dramatic increase in our ability to collect digital images in various situations and share them on the Web.
Two pertinent examples that are currently popular are Flickr, which
hosted over 10 billion photos in 2015 [1], and Instagram, which has
grown to have more than 400 million monthly active users [2]. These
images uploaded by Internet users can be considered to reflect visual aspects of their daily lives. Such ever-growing user-generated
images have potential as a new information source to analyze users’
opinions and sentiment, which enables several applications including opinion mining about social events, product marketing, and affective human-machine interaction [3]. Thus, automatic inference ofthe sentiment implied in the images has received increasing researchattention in recent years [4–7].
Conventional methods of image sentiment analysis have aimed
to design e↵ective visual features for training sentiment polarity
classifiers [4–6]. However, due to the a↵ective gap between lowlevel visual features and high-level concepts of human sentiments,
This research has been partly funded by Harris Science Research Institute of Doshisha University.

978-1-4799-9988-0/16/$31.00 ©2016 IEEE

it is difficult to directly associate the visual features with sentiment labels. On the other hand, studies about image annotation,
not particularly focusing on sentiment analysis, have reported thatthe collaborative use of textual features around training images(e.g., tags and descriptions) can improve the image content recognition [8, 9]. Inspired from these studies, to bridge images andsentiment, we should investigate how to introduce additional views
obtained from textual information to the feature space for training asentiment classifier.
In this paper, we present a novel image sentiment analysis
method that uses latent correlations among visual, textual, and sentiment views of training images. In the proposed method, we firstextract features from pairs of images and text to construct visualand textual views. To highlight the sentiment information in thetext, we introduce an external sentiment knowledge base, SentiWordNet [10], which forms the sentiment view. Then, using aframework of multi-view canonical correlation analysis (CCA) [11],we calculate a latent embedding space in which correlations among
the three views are maximized. Specifically, to capture the nonlinear relationship between features, we introduce explicit featuremaps [12, 13] to CCA. Finally, using the features that are projectedto the latent embedding space, we train a sentiment classifier. Because the latent space learns the alignments of multiple views, ourmethod corresponds to e↵ectively exploiting the textual informationof the training images even if a testing image only has a visual view.
Our experiments were conducted on a collection of images from
Flickr and Instagram, to which sentiment labels were assigned via
crowdsourcing. Results of the experiments show that our three-view
approach outperforms the conventional methods.
In summary, the main contributions of this paper are twofold:(i) most conventional methods use only visual features of training
images, while we propose a novel image sentiment classification
method that can exploit visual, textual, and sentiment views of the
training images; and (ii) with experiments designed via crowdsourcing, we show that the complementary use of multiple views of theimages can classify image sentiment better than the conventional
methods do.
2. RELATED WORKThe idea of associating low-level visual features with sentiments has
been investigated based on psychology and art theory using relatively small and controlled datasets [14,15], while recent works havestarted to analyze the sentiments of unconstrained real-world images
on social media [4–7]. Typically, the goal is to determine the sentiment polarity of images, i.e., positive or negative. To train a sen-

2837

ICASSP 2016

timent polarity classifier, color histogram and SIFT-based featuresof images are used in [4]. In [5], emotion-related adjective-noun
pairs were selected for image sentiment analysis, and their classifiers, called SentiBank, were trained based on low-level visual features. The detector response of SentiBank was used to form a midlevel representation of an image. Similarly, attribute features including facial expression were used as mid-level features in [6]. These
conventional methods focus on how to design visual representation
for sentiment analysis, and other available views of the data (e.g., tag
concurrence) are discarded in training classifiers. Recently, Wanget al. [7] exploited both visual content and textual information for
sentiment-based image clustering in a nonnegative matrix factorization framework. However, the method in [7] has severe sensitivity to
the initialization, and the experiments in this paper demonstrate that
our method outperforms the conventional method.
The use of correlations among visual and textual features associated to images has improved several image annotation and crossmodal retrieval tasks [8, 9, 16–20], but its e↵ectiveness has not been
fully demonstrated in image sentiment analysis. Thus, this paper
aims to use the latent correlations among multiple views for better
sentiment analysis. Canonical correlation analysis (CCA) [21] is oneof the techniques typically used to learn the alignments of multiple
views, but it only models the linear relationship between randomvariables. Several nonlinear extensions such as kernel CCA [11]and Deep CCA [22] have been proposed to reveal nonlinear relationship between the variables. However, these methods are intractable
for large-scale datasets due to their high computational complexity
and memory use. In contrast, recent advances of explicit featuremaps [12, 13] can convert nonlinear problems to linear problems,
which can be solved by linear frameworks with a low computation
cost [9, 23]. Following these studies, we introduce the explicit feature maps to CCA in the proposed method.
3. IMAGE SENTIMENT ANALYSIS USING LATENTCORRELATIONS AMONG MULTIPLE VIEWSThis section presents a novel image sentiment analysis method thatuses latent correlations among multiple views. An overview of the
proposed method is shown in Fig. 1. As shown, we first extract
features from each view (See 3.1). Then, after learning the multiview embedding space (See 3.2), the latent embedding space is used
to train an image sentiment polarity classifier (See 3.3).
3.1. Design of views for learning a latent embedding space
Our image sentiment analysis approach exploits three types of features: visual, textual, and sentiment views. This subsection describes
the details of feature extraction from each view.
Visual features: Following the feature design used in recent visualclassification methods [9, 18, 19], we represent image appearance using a combination of di↵erent visual descriptors: a 3⇥
256 dimensional histogram extracted from RGB color channels, a 512 dimensional GIST descriptor, a Bag-of-Words
quantized descriptor using a 1, 000 word dictionary with a2-layer spatial pyramid and max pooling. We also extract
the following mid-level features: 2,000-dimensional attributefeatures [24] and 1,200-dimensional SentiBank outputs [5].
For GIST features, attribute features, and SentiBank features,
we use the random Fourier feature mapping [12] to approximate the Gaussian kernel. All other histogram-based features were mapped using the exact Bhattacharyya kernel map-...

Training dataset

SentiWordNet
christmas
flowerspresent
christmas
newyorkpresentlightnewyork....
light
....

Visual View

Textual View

love
happy
nice

sad
death
disgustbored

Sentiment View

Multi-view Embedding

Projection
Testing image

Embedding Space
Positive or Negative?

Sentiment Polarity Classification

Fig. 1. An overview of the proposed method.
ping [13]. Finally, similar to [9], we reduce each kernelmapped feature to 500 dimensions using PCA and the final
concatenated feature results in a 2,500-dimensional vector.
Textual features: The second view consists of textual features,
which are extracted from text associated to images. We first
construct a vocabulary from a training dataset and represent
the textual features of an image using a traditional bagof-words approach, which counts how many times a word
appears in text around the image. Following [8, 9], we use
the linear kernel for the textual features, which counts thenumber of words shared between two images. Since thisrepresentation is highly sparse, we exploit SVD for large
and sparse matrices [25] to reduce the dimensions of the
textual feature matrix. In this paper, we experimentally set
the dimension of final textual representation to 1, 500.
Sentiment features: The third view aims to characterize the sentiment aspect of the associate text. For this, we use an external knowledge base, called SentiWordNet [10]. It is basedon the well-known English lexical dictionary WordNet [26],and has been utilized in text-based opinion mining tasks [27].
In SentiWordNet, three types of sentiment scores, “positivity,” “negativity,” or “objectivity,” are assigned to each WordNet synset. We use these scores to construct a vocabulary ofsentiment-related words. Specifically, we select words whose
sentiment scores of either positive or negative are larger thana pre-defined threshold. Then, based on the constructed vocabulary, we calculate the sentiment features of an image inthe bag-of-words approach. Finally, we apply the SVD to thefeature matrix to reduce its dimensionality. The resulting feature is represented as a 20-dimensional vector.
We will use v, t, s to denote the indexes of the visual, textual, andsentiment views, respectively.
3.2. Finding Latent Correlations Among Multiple Views
This subsection describes how to find latent correlations among multiple views using a framework of the generalization of canonical2838

correlation analysis [11]. Let Xi (i 2 {v, t, s}) denote the featurematrix of the i-th view, and the similarity between two feature vec0
tors x, x in the i-th view is defined by a kernel function Ki such that00
Ki (x, x ) = 'i (x)'i (x ). We want to find projection matrices Wi
which maps the i-th view into the latent embedding space. The
canonical correlation problem can be transformed into a distanceproblem such that the distances in the resulting space between eachpair of views for the same image are minimized [11]. The objectivefunction to learn the latent space is as follows:min

Wv ,Wt ,W sXi, j2{v,t,s}k'i (Xi )Wi

where ⌃i j is a covariance matrix between 'i (Xi ) and ' j (X j ), and wik
represents the k-th column of the matrix Wi . In the conventional
kernel CCA [11], kernel trick is used in Eq. (1). To reduce thecomputation complexity, one can use explicit feature maps [12, 13].
0
Let ˆ (x) denote an explicit feature mapping such that Ki (x, x ) =ˆ (x) ˆ (x). Instead of using the kernel trick, the mapping ˆ (x) canbe substituted to the objective function [9]. Solving the following
generalized eigenvalue problem provides the solution of Eq. (1):
S12
S22
S32

10 1
S13 CC BBw1 CC
CB C
S23 CCCC BBBBw2 CCCC =A@ A
S33 w30
BBBS11
BBB 0
B@
0

0
S22
0

10 1
0 CC BBw1 CC
CB C
0 CCCC BBBBw2 CCCC ,A@ A
S33 w3(2)where Si j = 'ˆ i (Xi )'ˆ j (X j ) is the covariance matrix between the i-thand j-th views, and wi is a column of Wi . This multi-view formulation has recently proven to be e↵ective for cross-modal retrieval andimage annotation [9, 19]. In the following subsection, we describe
how to use the latent space learned from multiple views for image
sentiment analysis.
3.3. Sentiment polarity classification using latent correlations
among multiple views
Using the projection matrices Wi , the features of the i-th view of the
training images can be represented in the latent space as follows:
Pi = 'ˆ i (Xi )Wi D p ,Flickr dataset
Instagram dataset

Positive
48,13933,076

Negative
12,6069,780

4. EXPERIMENTS
4.1. Dataset construction' j (X j )W j k2F

subject to WTi ⌃ii Wi = I, wTik ⌃i j w jl = 0, i, j 2 {v, t, s}, i , j
k, l = 1, · · · , d, k , l.
(1)

0
BBBS11
BBBS
B@ 21
S31

Table 1. The number of positive and negative images in each dataset.

(3)

where D is a diagonal matrix whose elements are the eigenvalues ofeach dimension in the embedding space. p is a weighting parameter, which is set to 4 as in [9, 19]. Using Eq. (3) for each view,
we represent the final feature matrix of training images as the concatenation of Pv , Pt , and P s . If we consider the case in which text
of testing images is unavailable, we concatenate the projection Pv tothe original feature, following the conventional cross-modal retrieval
method [18]. Based on the new feature representation of the training
dataset with sentiment labels, we learn a sentiment polarity classifier.
In this paper, we exploit a linear SVM, which is also used in the conventional methods [4, 5]. Note that although this paper focuses on
binary classification as well as the conventional methods [4, 5], ourmethod can be easily extended to multi-class sentiment classification(e.g., positive, negative, and neural). Given a testing image, we also
extract features from available views (either or both of visual andtextual views) and classify the features projected to the embedding
space.

To conduct experiments, we collected a set of images from Flickr
and Instagram as follows.
• Flickr dataset. From Flickr, we first downloaded a set ofimage IDs provided by [28]. Some images were unavailable,
and limiting the number of images for each Flickr user to 70,
we obtained 105, 587 images. The most frequent words are“view,” “black,” “photo,” “canon,” “nikon,” and “film.”• Instagram dataset. This dataset was constructed by ourselves from Instagram. Using each of the emotional words
listed in SentiWordNet as a query keyword, we crawl a setof images. The total number of images was 120, 000. Thisdataset contains more images that reflect users’ daily lives
than Flickr dataset. The most frequent words are “love,”“like,” “life,” “day,” and “new.”In this experiment, we extracted textual and sentiment features fromtags and descriptions associated to images.
To evaluate the performance of image sentiment classification,
we prepared sentiment labels of images via crowdsourcing. Conventional methods exploited pseudo sentiment labels using the automatic annotation algorithm based on image tags [4, 7], but it isunreliable due to the noisy tags or lack of tags. To the best of ourknowledge, this paper is the first to provide sentiment polarity labels to large-scale image datasets by crowdsourcing-based humanannotations. Specifically, we chose CrowdFlower1 as a platform,
and presented each image for subjective evaluation. For each image, three workers were asked to provide a sentiment score. They
could choose on a discrete five-point scale labeled with “highly positive,” “positive,” “neutral,” “negative,” and “highly negative.” Thefinal construction of the ground truth exploited the majority votesof polarity for each image. Table 1 shows the details of the number of positive and negative images in each dataset. Since this experiment targets on the binary classification problem following theprevious works [4, 5], we discarded the images labeled by “neutral”
and the images resulting in disagreement among workers. Note that
our method can be extended to the multi-class classification problem, which will be performed in our future work. The datasets withsentiment labels is available on the Web2 .
4.2. Baselines
We compare the performance of our multi-view embedding-based
approach with the following conventional methods, which exploit
either visual or textual view: a low-level visual feature-based
method [4] (denoted as Low), a mid-level visual feature-based
method [5] (denoted as SentiBank), a method that concatenates
low-level visual features with the mid-level features (denoted as
Low&SentiBank), and a textual feature-based method [10] (denoted as SentiStrength3 ). Note that for Low [4], we use the same2839

1 http://www.crowdflower.com/

2 http://mm.doshisha.ac.jp/senti/CrossSentiment.html
3 http://sentistrength.wlv.ac.uk/

Table 2. Average and standard deviation of the classification accuracy of image sentiment polarity for 10 runs in each dataset. Notethat for Low [4], we use the same visual feature set as those described in Sec. 3.1, except for SentiBank outputs.
MethodFlickr dataset
Instagram dataset
Random
49.78 ± 1.05%
50.06 ± 1.09%
Low [4]69.44 ± 0.85%
67.16 ± 1.28%
SentiBank [5]70.01 ± 0.63%
67.26 ± 1.12%
Low&SentiBank
70.54 ± 1.00%
68.03 ± 1.36%
SentiStrength [29]59.30 ± 0.87%
62.78 ± 0.91%
USEA [7]51.87 ± 1.76%
52.61 ± 2.00%
LC(V+T)+P(V)70.94 ± 0.67%
68.29 ± 1.42%
LC(V+S)+P(V)70.67 ± 0.78%
65.44 ± 1.16%
LC(V+T+S)+P(V)72.36 ± 0.41%
68.54 ± 1.14%
LC(V+T)+P(V+T)
74.42 ± 0.67%
72.43 ± 1.54%
LC(V+S)+P(V+S)
68.98 ± 1.01%
69.35 ± 1.08%
LC(T+S)+P(T+S)
64.63 ± 0.91%
66.50 ± 0.49%
LC(V+T+S)+P(V+T+S) 74.77 ± 0.82%
73.60 ± 0.88%
visual feature set as those described in Sec. 3.1, except for SentiBank outputs. By comparing these methods in terms of using asingle view of the testing data, we investigate the e↵ectiveness of
our multi-view embedding approach. For each method, we used
Liblinear4 to train a linear SVM, and the soft margin parameter C of
the linear SVM was determined by cross validation. We also compare our method with the state-of-the-art method that exploits visualand textual features of the testing data [7] (denoted as USEA). For
reference, the random classification results are shown as Random.
To validate the contribution of the latent correlations among
multiple views, we split up di↵erent views with and without embedding. The views used for calculating latent correlations are denotedby LC, and the features projected from images for classification
are shown by P. LC(V+T) will refer to the two-view embedding
based on visual and tag features; LC(V+T+S) to the three-viewembedding based on visual, textual, and sentiment features; P(V) to
the projection of only visual features of the images; and P(V+T) to
the projection of visual and textual features of the images.
4.3. Performance evaluation and discussion
Each dataset was randomly separated into a training set and a test
set for 10 runs. In the Flickr dataset, for each sentiment polarity ateach run, we randomly sampled 11,346 images and 1,260 images
for training and testing, respectively. In the Instagram dataset, foreach sentiment polarity at each run, we randomly sampled 8,802
images and 978 images for training and testing, respectively. As aperformance evaluation metric, we calculated the average and standard deviation of classification accuracy over all runs. The results
are shown in Table 2. As shown, our method using the three viewsof the training and testing images obtained the best average classification accuracy. This result validates the e↵ectiveness of the complementary use of the multiple views for image sentiment analysis.
Even in the case in which the textual and sentiment views of a testing image are unavailable due to the lack of associated text (i.e.,
LC(V+T+S)+P(V)), our approach presents a better representationof visual features because the latent space learns the alignments ofmultiple views.
Examples of classification results by our three-view embeddingbased method are shown in Fig. 2, in which the red border indicates63022030@N00

27597792@N03

76994867@N00

78745957@N00

35034347485@N0153454935@N00

11334344@N00

10508943@N00

33758480@N00

64232630@N00

15790252@N06

48600091327@N01

55064893@N00

65802182@N00

92132559@N00

58812071@N00

68009656@N00

29146273@N03(a) Images classified as positive.

24128704@N08

17715663@N00

27403767@N00

79753359@N00

83555001@N00

92422987@N00

95288238@N00

43360570@N02

61495861@N00

98816109@N00

81813159@N00

70346960@N00

20473443@N07

33049952@N08

35034354960@N01

29555476@N06

82967074@N00

40526588@N00(b) Images classified as negative.
Fig. 2. Examples of images classified as positive or negative in Flickr
dataset. Image captions correspond to Flickr user IDs. The red border indicates a misclassified image.
a misclassified image. We found some difficult cases that cannot
be accurately classified by the proposed method. For example, current visual features do not characterize facial expression, letters anddrawings in the images. Thus, the design of better features of eachview will be performed in our future work.
5. CONCLUSION AND FUTURE WORKIn this paper, we present a novel image sentiment analysis methodthat uses the latent correlations among multiple views of training
images. In the proposed method, we first extract features from visual, textual, and sentiment views. Then, to project the features fromthese views, we follow the framework of multi-view CCA using explicit feature mappings. Finally, in the embedding space, a sentiment polarity classifier is trained based on the projected features. To
validate the e↵ectiveness of the proposed method, we constructed
image datasets via crowdsourcing. Experiments conducted on thedatasets show that our multi-view embedding space is more e↵ective for classifying image sentiment polarity than methods that use asingle modality only and the state-of-the art method that jointly uses
multiple modalities.
The features used in our framework should be investigated forfurther performance improvement. We will introduce additionalviews or features such as facial expressions [6]. In addition, we will
introduce the deep learning-based features [30, 31], which have significantly improved many computer vision tasks, into the proposed
framework. Furthermore, we will tackle the multi-class sentimentclassification such as positive, negative, and neutral.

4 http://www.csie.ntu.edu.tw/˜cjlin/liblinear/2840

6. REFERENCES[1] flickrBLOG, “Find every photo with flickrs new unified
search experience,” http://blog.flickr.net/en/2015/05/07/flickr-unified-search/, May 2015, Last accessed: 09/24/2015.
[2] Instagram Blog, “Celebrating a community of 400 million,”
http://blog.instagram.com/post/129662501137/
150922-400million, Sep 2015, Last accessed: 09/24/2015.
[3] E. Cambria, B. Schuller, Y. Xia, and C. Havasi, “New avenues
in opinion mining and sentiment analysis,” IEEE Intelligent
Systems, vol. 28, no. 2, pp. 15–21, 2013.
[4] S. Siersdorfer, E. Minack, F. Deng, and J. Hare, “Analyzing
and predicting sentiment of images on the social web,” in Proc.
Int. Conf. Multimedia (MM), 2010, pp. 715–718.
[5] D. Borth, R. Ji, T. Chen, T. Breuel, and S.-F. Chang, “Largescale visual sentiment ontology and detectors using adjective
noun pairs,” in Proc. Int. Conf. Multimedia (MM), 2013, pp.
223–232.
[6] J. Yuan, S. Mcdonough, Q. You, and J. Luo, “Sentribute: Image sentiment analysis from a mid-level perspective,” in Proc.
Int. Workshop on Issues of Sentiment Discovery and Opinion
Mining (WISDOM), 2013, pp. 10:1–10:8.
[7] Y. Wang, S. Wang, J. Tang, H. Liu, and B. Li, “Unsupervised
sentiment analysis for social media images,” in Proc. Int. Joint
Conf. Artificial Intelligence (IJCAI), 2015.
[8] M. Guillaumin, J. Verbeek, and C. Schmid, “Multimodal semisupervised learning for image classification,” in Proc. Int.
Conf. Computer Vision and Pattern Recognition (CVPR), June
2010, pp. 902–909.
[9] Y. Gong, Q. Ke, M. Isard, and S. Lazebnik, “A multi-view embedding space for modeling internet images, tags, and their semantics,” International Journal of Computer Vision, vol. 106,
no. 2, pp. 210–233, 2014.
[10] A. Esuli and F. Sebastiani, “SentiWordNet: A publicly available lexical resource for opinion mining,” in Proc. Int. Conf.
Language Resources and Evaluation (LREC), 2006, pp. 417–
422.
[11] D Hardoon, S Szedmak, and J Shawe-Taylor, “Canonical correlation analysis: An overview with application to learning
methods,” Neural Computation, vol. 16, no. 12, pp. 2639–
2664, Dec 2004.
[12] A. Rahimi and B. Recht, “Random features for large-scale
kernel machines,” in Proc. Neural Information Processing Systems (NIPS), 2007.
[13] F. Perronnin, J. Sánchez, and Y. Liu, “Large-scale image categorization with explicit data embedding,” in Proc. Int. Conf.
Computer Vision and Pattern Recognition (CVPR), June 2010,
pp. 2297–2304.
[14] V. Yanulevskaya, J. C. van Gemert, K. Roth, A. K. Herbold,
N. Sebe, and J. M. Geusebroek, “Emotional valence categorization using holistic image features,” in Proc. Int. Conf. Image Processing (ICIP), Oct 2008, pp. 101–104.
[15] J. Machajdik and A. Hanbury, “A↵ective image classification
using features inspired by psychology and art theory,” in Proc.
Int. Conf. Multimedia (MM), 2010, pp. 83–92.

[16] N. Rasiwasia, J. C. Pereira, E. Coviello, G. Doyle, G. R. G.
Lanckriet, R. Levy, and N. Vasconcelos, “A new approach to
cross-modal multimedia retrieval,” in Proc. Int. Conf. Multimedia (MM), 2010, pp. 251–260.
[17] Z. Li, J. Liu, J. Tang, and H. Lu, “Robust structured subspacelearning for data representation,” IEEE Trans. Pattern Analysis
and Machine Intelligence, vol. 37, no. 10, pp. 2085–2098, Oct
2015.
[18] Y. Gong, L. Wang, M. Hodosh, J. Hockenmaier, and S. Lazebnik, “Improving image-sentence embeddings using large
weakly annotated photo collections,” in Computer Vision
ECCV 2014, vol. 8692 of Lecture Notes in Computer Science,
pp. 529–545. Springer International Publishing, 2014.
[19] Y. Fu, T. M. Hospedales, T. Xiang, Z. Fu, and S. Gong, “Transductive multi-view embedding for zero-shot recognition andannotation,” in Computer Vision ECCV 2014, vol. 8690 of
Lecture Notes in Computer Science, pp. 584–599. Springer International Publishing, 2014.
[20] M. Katsurai, T. Ogawa, and M. Haseyama, “A cross-modal approach for extracting semantic relationships between concepts
using tagged images,” IEEE Trans. Multimedia, vol. 16, no. 4,
pp. 1059–1074, June 2014.
[21] H. Hotelling, “Relations between two sets of variates,”
Biometrika, vol. 28, no. 3/4, pp. 321–377, Dec. 1936.
[22] G. Andrew, R. Arora, J. Bilmes, and K. Livescu, “Deep canonical correlation analysis,” in Proc. Int. Conf. Machine Learning(ICML), 2013, pp. 1247–1255.
[23] D. Lopez-Paz, S. Sra, A. Smola, Z. Ghahramani, and
B. Schölkopf, “Randomized nonlinear component analysis,”in Proc. Int. Conf. Machine Learning (ICML), 2014.
[24] F. X. Yu, L. Cao, R. S. Feris, J. R. Smith, and S.-F. Chang,“Designing category-level attributes for discriminative visual
recognition,” in Proc. Int. Conf. Computer Vision and Pattern
Recognition (CVPR), June 2013, pp. 771–778.
[25] R. M. Larsen, “Lanczos bidiagonalization with partial reorthogonalization,” Tech. Rep. 537, Department of Computer
Science, Aarhus University, 1998.
[26] G. A. Miller, “WordNet: A lexical database for English,” Commun. ACM, vol. 38, no. 11, pp. 39–41, Nov. 1995.
[27] C. Hung and H.-K. Lin, “Using objective words in SentiWordNet to improve word-of-mouth sentiment classification,” IEEE
Intelligent Systems, vol. 28, no. 2, pp. 47–54, March 2013.
[28] Y. Yang, J. Jia, S. Zhang, B. Wu, Q. Chen, J. Li, and J. Tang,“How do your friends on social media disclose your emotions?,” in Proc. AAAI Conf. Artificial Intelligence (AAAI),
2014, pp. 306–312.
[29] M. Thelwall, K. Buckley, G. Paltoglou, D. Cai, and A. Kappas,“Sentiment in short strength detection informal text,” Journalof the American Society for Information Science and Technology, vol. 61, no. 12, pp. 2544–2558, Dec. 2010.
[30] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell, “Ca↵e: Convolutionalarchitecture for fast feature embedding,” in Proc. Int. Conf.
Multimedia (MM), 2014, pp. 675–678.
[31] A. Frome, G. S. Corrado, J. Shlens, S. Bengio, J. Dean, M. A.
Ranzato, and T. Mikolov, “DeViSE: A deep visual-semantic
embedding model,” in Proc. Advances in Neural Information
Processing Systems (NIPS), 2013, pp. 2121–2129.

2841