DEIM Forum 2016 A4-2

Estimating Desired Actions Stimulated by Annotated Images
Bei LIU† , Makoto P. KATO† , and Katsumi TANAKA†† Department of Social Informatics, Kyoto University
Yoshida-honmach, Sakyo-ku, Kyoto
606–8501 Japan
E-mail: †{liubei,kato,tanaka}@dl.kuis.kyoto-u.ac.jp
Abstract This paper addresses the problem of finding desired actions motivated from images with annotation.
Many images in the Web are created to promote, trigger, or motivate for people to take some actions after viewing
those images (called Call-to-Action, CTA), such as advertisement images. We decompose the problem of learning
the relationship between images and desired actions into two sub-problems for which plenty of text data are possibly available for learning. The first sub-problem is to estimate objects and impressions from images, while thesecond sub-problem is to estimate desired actions from the combination of objects and impressions. We conducted
experiments to demonstrate the performance of our desired action prediction, and showed that the problem decomposition could improve the desired action prediction especially when a limited number of manually labeled images
were available.
Key words Image, desired action, object, impression

1. Introduction
We are surrounded by a huge number of images online and!""#$%#"&'"($&)($*&%%+$,-$./"'*$
0)1/"(0"),'!

oﬄine nowadays. Images have become a way of expression
and a resource of inspiration. What we can perceive from

2"'0/"($345-)!"4--60)1!

images has been widely researched, including object recognition and action detection. Recently, how images could aﬀect

Figure 1 User’s reaction to an image with fresh ingredients.

us human becomes to attract attentions while retain undocumented, especially after the popularity of image sharing

What happens when we look at something? This ques-website, such as Pinterest. According to [16], 85.6% of re-tion has been researched a lot in the field of psychology. A

spondents agreed or strongly agreed that they use Pinterest

commonly know theory is “ABC of psychology” [15] which

to find “something I might be doing later” and 86.7% agreed

involves aﬀect, behavior and cognition. However, there is no

to use Pinterest to look for “inspiration for something I will

accurate saying about their relationship. In this research,

be doing soon”. We have to admit that we are aﬀected by

we are not going further to discuss them. Instead, we willthose images we look at every day, or those images are en-use some concepts to help with the explanation. Our firstcouraging us to take further actions. Some images are cre-interaction with stimuli is perception, which is out of ourated to promote, trigger, or motivate people to take actions

awareness. In Goldstein’s book about sensation and percep-

on purpose, such as advertisement images. Some aﬀectedtion [4], he claimed that perception is the result of complex

us unconsciously, like images shared by friends in social net-“behind the scenes” processes. In the case of image view-work. Most of us have experience that after looking at aning, perceiving information in the images, which is called

image about delicious food, we will feel hungry, even if weperception, is conscious sensory experience. Then our past

just had dinner two hours ago. Another typical example isknowledge leads us to place objects in a category, such as

when we see very beautiful images shared by friends about

recognizing car in the image as “car”. Having the concept

their trip, we would feel like to go the same place. We canbeen categorized in our mind, we might feel like to move oureven assume that images taken by friends have more attrac-

head to focus more on the image, which is defined as actions.

tion than advertisement images about the same place, due

Another key part when we look at an image is aﬀect, whichto people’s trust in friends and expectation to share similar

is the experience of feeling or emotion. For example, an im-experience with friends.

age of scary snake will make us feel upset while images ofaesthetic, and objects are widely researched [13] [21]. Recent

beautiful scenery will please us.

years, more properties have attracted researchers’ attention,

In this research, we are talking about a further concept insuch as sentiment [23], interestingness [6], popularity [8], andaﬀect than emotion and arousal, which we call it desired ac-

memorability [9]. These properties involves users as an im-tion. For example, image in Figure 1 is about somebody deal-

portant factor and computers are trained to simulate as hu-

ing with fresh ingredient, cutting mushroom. Based on our

man’s thought.

general knowledge, we can imagine that he/she is preparing

When mentioning images’ eﬀect to user behavior, we could

for cooking. The freshness of ingredients in the image makenaturally come up with advertisement images. However, re-viewers feel pleased and happy. Moreover, viewers may feelsearches in advertisement related field have few eﬀorts inlike to eat vegetables or cook by themselves. In this case,predicting motivated behaviors from images. Instead, theyeating vegetables and cooking is the desired action that arefocus on images’ impact on consumers’ behaviors [12] andstimulated by the image.

how to improve advertisement quality by matching technol-

In this work, we are trying to find what desired action

ogy [22].

could be stimulated by an image and estimate to what ex-

There are researches in the field of psychology which aretent the desired action is stimulated. However, since thetrying to find image’s influence to user behavior. For exam-

appearance of machine, semantic gap has been a significant

ple, Patterson [16] have done surveys among Pinterest users

problem for computers, not to mention understanding peo-and exams how Pinterest images aﬀect users’ motivation andple’s reaction to a certain type of information by computersbehaviors. Their results show that images can do influence

automatically. In this paper, we would like to introduce ouruser’s future behaviors. In this research, we are trying toapproach to solve this problem by decomposing it into two

use technology of visual analysis to find the bridge between

sub-problems, for which external resources and information

images and possible future behaviors that are motivated (de-can be utilized. The first sub-problem is to extract objects

sired action as we call).

and impressions from annotated images, and the second sub-2. 2 Deep Learning

problem is to estimate desired actions from the combination

Conventional neural network has a long history in fields.

of objects and impressions. We design experiments to con-

In early researches, it has been used for digit recognition withduct our approach and to see the performance of utilizing

supervised back-propagation networks and successful results

object and impression to estimate desired action for images.

were achieved [11]. Recent years, it got a lot of attractions,

The contribution is this paper can be summarized into two

especially when it is applied on large benchmark datasets

points:consisting of more than one million images. Krizhevsky et（ 1 ） We propose the problem of estimating desired ac-al. [10] has shown that deep convolutional neural networks

tions that are stimulated by annotated images.

(CNNs) is able to achieve great performance improvement（ 2 ） To overcome lack of understanding gap between

and eﬃciency for classification on similar datasets such as

computer and images, we propose to decompose this problem

ImageNet [18].

into two sub-problems, by which plenty of external resources
and information are available for learning.

3. Preliminaries

By accurately estimating desired actions from images, we

In this section, we will first introduce definition of desired

could use find images that could stimulate certain desired

action that we study, and then explain the problem we ad-

actions and then use these images for commercial advertise-dress in this research.

ment, presentation, PSAs (Public Service Announcement),
image sharing in SNS (social network systems), and so on.
In the rest of the paper, related work is discussed in Sec-3. 1 Desired Action
From viewers’ perspective, there is a gradual process when
we look at an image, from perception to aﬀect.

In per-tion 2.. Section 3. defines the problem investigated in thisception, there is also a hierarchical model based on Greis-

research. Our approach is presented in Section 4., and fol-

dorf’s model [5], which includes primitive features (color,

lowed by experimental explanation in Section 5.. Finally, we

shape, texture), objects (person/thing, place/location, ac-draw a conclusion and discuss future work in Section 6..

tivity, event), and inductive interpretation (symbolic value,

2. Related Work

prototypical displacement). After perception, aﬀect will be
stimulated. Aﬀect includes and is not limited to emotion

2. 1 Visual Analysis

and feeling. Action (or behavior in some researches) is also

Visual analysis has a long history in the field of visuala product of image viewing. Before action, the feeling of de-computing. Regular properties of images includes quality,

sire to perform the action is belonged to aﬀect and we call

it desired action. Desired action (also referred as DA in thislike “delicious” and “amazing”. Chen et al. [3] addressed thepaper) is defined as:

big “aﬀective gap” between the low-level visual features and［Definition 1］（Desired Actions） Desired actions are ac-

the high-level sentiment when they tried to modeling adjec-

tions that a user wish to carry out after viewing an image.

tives correlated with visual sentiments. Similar to sentiment

The reason we want to research on desired action rather

analysis, desired action is more challenging because desired

than real action is that we want to decrease diﬀerence of ac-actions correspond to high level abstractions from a given

tions brought by feasibility. For example, action of “eating”

image, which may require viewer’s knowledge beyond theis easier to conduct than action “traveling”, and it results inimage content itself. Meanwhile, a more challenging task in

the fact that the possibility of performing “eating” is much

this problem is the lack of data which indicates image and its

higher than “traveling” as a result of viewing images. How-corresponding desired actions. And the fact that each classever, the desire of “traveling” could be in similar level with(desired action) contains much more diverse images add thedesire of “eating”.

diﬃculty to discover features which can distinguish much

3. 2 Problem Definition

more diverse classes from each other.

The problem of this research can be defined as: given an

Imagine we have only 10 images that are corresponding

image p ∈ P , an image set S with annotation data (includ-to desired actions “eating”, and these ten images contains

ing tags, comments, viewer number, etc.), after a series ofdiﬀerent types of food. It is quite challenging to find visualcomputing, we could find the desired actions {a} that mightfeatures to directly match from images to desired action “eat-be stimulated by the image and the degrees of desires:ing”. What if we could find the common knowledge of thosef (p) = {(a, d)|a ∈ A, d ∈ R+ }.

images? From we human’s eyes, it is easier to get that thoseimages are all about food, and they all looks very delicious.

The image set S is a large set of images that were taken by

However can we make computer recognize the same things?

diﬀerent users and viewed by others with comments. Usu-

The answer is we can utilize knowledge obtained in resources

ally, these comments include viewers’ impression about theseother than these ten images and this knowledge is more con-images in terms of adjectives.

fident because of much more data. This is the key idea of ourapproach, which is to decompose desired action estimation

4. Approach

problem into two sub-problems: to extract objects and impressions from images, and to estimate desired actions from!"#$%!

2%.!-%51
3+6(/!

the combination of objects and impressions. The reason we&%'&!

they both have implications for actions. Figure 2 also ex-choose objects and impressions as a bridge is the fact that
hibits the main idea.

()*%+&!

!",-%..!(/!

4. 1 Extracting Objects and Impressions from Images&%'&!

!"#$%01!"#$%1&$%#$%0123!

!"#$%01!"#$%1!"#$%!"()*%+&41&$%#$%01()*%+&41 " !",-%..!(/1
1&$%#$%0123!
!",-%..!(/!

Object detection from images has a long history in visualcomputing literature, and deep learning is considered thestate of the art with quite high performance in terms accuracy [21] [19]. In this research, we are going to use existing

Figure 2 Problem decomposition: match from image to objects
and impression, and match from objects and impression to desired action. Objects, impression and desired
action are in form of text.

state of art research to detect objects from images. 1000classes model in ImageNet [10] is used with regular conventional neural network (CNN) for object detection.
We use deep conventional network to train impression from

The problem in this research can be mainly summarized

images. We use social clues attached to annotated images to

as a problem of matching from image to desired action. It

acquire labeled images. As addressed in research of image

can also be treated as an image classification problem if we

sentiment analysis [23], to obtain highly reliable labeled in-

replace object with desired action in the object recognition

stances is nontrivial, let alone a large number of them. In

problem. Recognition and detection of objects such as “cat”

this research, we utilize users’ comments to those images.

and “sky” has been studied extensively with relatively highWe first extract adjectives from comments and use those ad-

performance in the field of computer vision, especially byjectives as viewers’ impressions. In total, 888 classes of im-conventional neural network these years. However, it re-pressions are extracted with more than 10 images for eachmains diﬃcult to model impressions indicated by adjectives

class in the training. Then these images are transformed into

format following Caﬀe’s requirement.

Table 2 indicates each layer’s shape. The first convolu-

We describe briefly the overall architecture of the deep con-tional layer filters the 227 × 227 × 3 input image with 96 ker-

ventional neural networks for training the impression classi-

nels of size 11 × 11 × 3 with a stride of 4 pixels. The output

fication, called ImpressionNet. The architecture mostly fol-of the first convolutional layer is processed after pooling andlows [10]. The while net consists of eight main layers (con-response normalization and then taken as input of the second

volutional and fully-connected) with weights with first fiveconvolutional layer with 256 kernels of size 5 × 5 × 48. The

convolutional and other three fully-connected. Image values

following three convolutional layers are connected to eachpropagate through the five convolutional layers with pooling,other without output processing like pooling or normaliza-

normalization and ReLU. And the three fully-connected lay-tion. Each of the three fully-connected layers has 4096 neu-

ers are used to determine the final neuron activities. Neurons

rons.

in the fully-connected layers are connected to all neurons in

4. 2 Estimating Desired Actions from Objects andtheir previous neuron. For each layer, including convolu-

Impressions

tional layer and fully-connected layer, its output is applied

There are many resources that provide a wealth of informa-with Rectified Linear Units (ReLUs) [14]: f (x) = max(0, x).

tion to computers about how people use and understand the

Local response normalization layers locate following the firstworld in terms of text, such as Wikipedia and Wiktionary.

and second pooling layers.

And the prerequisite is that the knowledge can be compiled

Table 1 The size of input and output data

into a useful representation. In this work, we will utilize ConceptNet [20], which is a project that creates representation

Name

Size

input

3 × 256 × 256

of crowd-sourced knowledge. It provides a large semanticdata

3 × 227 × 227

graph that describes general human knowledge and how it isconv1

96 × 55 × 55

expressed in natural language. In ConceptNet, relations arepool1

96 × 27 × 27

referred as how words are related through common knowl-

norm1

96 × 27 × 27
256 × 27 × 27

edge. Several relations are used especially for this work, as

conv2
pool2

256 × 13 × 13

norm2

256 × 13 × 13

conv3

384 × 13 × 13

to potential actions. However, in our research, impression

conv4

384 × 13 × 13

is also an important factor that decides desired action, es-

conv5

256 × 13 × 13

pecially degree of the desired action. And ConceptNet lacks

pool5

256 × 6 × 6of this kind of information. To match from impression to

fc6

4096

desired action, we will need to add object, since object indi-

fc7

4096

fc8

888

label

1

output 1

shown in Table 3.
ConceptNet provides a connection to bridge from objects

cated “what desired action can be stimulated” and impression decides “how much a desired action can be stimulated”.
There are two types of impressions, general impression which
can be used for all objects, like good or bad, and specific im-Size of input and output data is shown in Table 1. All images are resized to 256 × 256 first and then cropped random
227 × 227 patches from the 256 × 256 images. The network isthen trained on the cropped patches.The output of last fullyconnected layer is fed to a 888-way softmax which produces
a distribution over the 888 class labels.

pression which is specific for certain objects, like delicious orclean. We are not going to distinguish them in this stage,
and our focus will lay on general impression.
Considering impressions, we have to pay attention that
diﬀerent saying of impressions might indicate various degreeof desire. For example, we can tell from “Yummy!” and“Yummy!!!!!!!!!” that the later one expresses much stronger

Table 2 The shape of each layer
Name Shapeconv1 96 × 3 × 11 × 11
conv2 256 × 48 × 11 × 11

of desired action “eating”.

Another example is “I want

to go!” and “I would like to book a flight and fly there
now!”. Many types of indicators imply the diﬀerence of im-

conv3 384 × 256 × 3 × 3

pressions. To make it simple, we will take some explicit

conv4 384 × 192 × 3 × 3

forms into account, such as repeated characters in a word

conv5 256 × 192 × 3 × 3(“cooooooool”) [2] and usage of special symbols and emo-

fc64096 × 9216

fc7

4096 × 4096

fc8

888 × 4096

jis [1].

Table 3 Relations of objects and actions used in this work

Relation

Description

Example: (A, B)

UsedForA is used for B; the purpose of A is B.

(ingredient, cook)

CapableOf

Something that A can typically do is B.

(cut, knife)

MotivatedByGoal Someone does A because they want result B; A is a step toward accom- (arrange, comfortable house)
ObstructedBy

plishing the goal B.
A is a goal that can be prevented by B; B is an obstacle in the way of (bad lung, stop smoking)CreatedByA.
B is a process that creates A.

(food, cook)that the concepts that cause the desire of a certain action

5. Experimentis not that proper for the situation of image viewing. The

Corresponding to the approach, the experiment is divided

reason is this result comes from people’s real feeling in real

into two parts: extracting objects and impressions from im-

life, which including a lot of scenarios where image viewing

ages, and estimating desired actions from combination of ob-

or other visualization is not necessary. This indicates thatjects and impressions.

we still need other approach to improve the performance of

5. 1 Object and Impression Extraction

matching from objects and impressions to desired actions, in

We first extract impressions from annotated images bywhich image viewing is significant.

keeping adjectives in viewers’ comments.

5. 3 Result and Discussion

We utilize dataset in [17] which contains 1000 categories

Figure 3 shows some results of impression extraction.

and 1.2 million images and then apply Caﬀe [7] for object

Those impressions indicates how users who looked at thedetection. For impression detection, we use crawled more

image describe about the image.

than 161513 images from the Flickr, each with more than
five comments. The number of impressions extracted from

Table 5 Result: accuracy of impression prediction
Accuracy

ImpressionNet Random

those comments is 4301. Then we calculated number of im-

Precision@1 0.0148

0.0056ages for each impression class and filtered impressions with

Precision@5 0.1632

0.0281less than 10 images. As a result, 888 impressions were kept
with more than 153 thousand images in total. Those images

Table 5 shows the accuracy of impression prediction. We

were divided into two parts, 80% for training and 20% for

use two scores, “Precision@1” and “Precision@5”. “Preci-

testing and validation.

sion@1” means only if the predicted top one result is exactly

We set the batch size of image as 256 × 256 and cropping it

the same as testing result, the score is 1, otherwise the result

into 227 × 227 for training. The regression objective is mini-

is 0. And “Precision@5” means as long as the testing result

mized by stochastic gradient descent with a batch size of 256

is among the top five results, the score is 1. We compare

examples, momentum of 0.9, and weight decay of 0.0005.

our result with randomly give impression. From the result,

The learning rate is initialized to 0.01. We run a total of

we can see that the accuracy of impression detection cannot

250,000 iterations.

match object detection, but our approach still outperform

5. 2 Match between Objects and Desired Actions

randomly impression detection. In the experiment, we con-Table 4 Example of relation “CauseDesire”

ducted single-class classification, which will also decrease theConcept

Actionbad weather

go somewhere

be boresurf net

good weather travel
hatredpunch someone

be hungry

cook meal

lonelinessmake phone callaccuracy of prediction.

6. Conclusion
In this paper, we introduced our proposal of problem: estimating desired actions from annotated images. We demonstrated the diﬃculty of directly matching from images todesired actions, and thus proposed our idea of decomposing

In the current stage, we got result from ConceptNet withrelation between concepts and actions. Table 4 shows someresults about relation CauseDesire. Here, the concepts include both objects and impressions. From this result, we see

this problem into two sub-problems. Object and impression
are selected as a bridge between images and desired actions,and the problem becomes to detect object and impression
from images, and estimate desired action from object and(a) Impressions: delicious, nice, (b) Impressions:spicy, delightful, thai

outstanding, pro- (c) Impressions: great, sweet, cute,fessional, great, vibrant, compelling

lovely, adorable

Figure 3 Examples of impressions extracted from viewers’ comments annotated with images

impression. Object and impression are in the form of natural language. The advantage of this approach is that we canhave much more resources to learn and that common-sense
knowledge are introduced to help computer understand images in the perspective of people.

7. Acknowledgement
This work was supported in part by Grants-in-Aid for
Scientific Research (Nos. 15H01718, and 26700009) from
MEXT of Japan.

References[1] Becker, L., Erhart, G., Skiba, D., Matula, V.: Avaya: Sentiment analysis on twitter with self-training and polarity
lexicon expansion. In: Second Joint Conference on Lexicaland Computational Semantics (* SEM). vol. 2, pp. 333–340(2013)[2] Brody, S., Diakopoulos, N.: Cooooooooooooooollllllllllllll!!!!!!!!!!!!!!: using word lengthening to detect sentiment inmicroblogs. In: Proceedings of the conference on empiricalmethods in natural language processing. pp. 562–570. Association for Computational Linguistics (2011)[3] Chen, T., Borth, D., Darrell, T., Chang, S.F.:
Deepsentibank: Visual sentiment concept classification
with deep convolutional neural networks. arXiv preprint
arXiv:1410.8586 (2014)[4] Goldstein, E.: Sensation and perception. Cengage Learning(2013)[5] Greisdorf, H., O’Connor, B.: Modelling what users see when
they look at images: a cognitive viewpoint. Journal of Documentation 58(1), 6–29 (2002)[6] Gygli, M., Grabner, H., Riemenschneider, H., Nater, F.,
Van Gool, L.: The interestingness of images. In: Computer
Vision (ICCV), 2013 IEEE International Conference on. pp.
1633–1640. IEEE (2013)[7] Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J.,
Girshick, R., Guadarrama, S., Darrell, T.: Caﬀe: Convolutional architecture for fast feature embedding. In: Proceedings of the ACM International Conference on Multimedia.
pp. 675–678. ACM (2014)[8] Khosla, A., Das Sarma, A., Hamid, R.: What makes animage popular? In: Proceedings of the 23rd international
conference on World wide web. pp. 867–876. ACM (2014)[9] Khosla, A., Raju, A.S., Torralba, A., Oliva, A.: Understanding and predicting image memorability at a large scale(June 2015)[10] Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems. pp. 1097–
1105 (2012)[11] LeCun, Y., Boser, B., Denker, J.S., Henderson, D., Howard,
R.E., Hubbard, W., Jackel, L.D.: Backpropagation applied
to handwritten zip code recognition. Neural computation
1(4), 541–551 (1989)[12] Malik, M.E., Ghafoor, M.M., Iqbal, H.K., Ali, Q., Hunbal,
H., Noman, M., Ahmed, B.: Impact of brand image and advertisement on consumer buying behavior. World Applied
Sciences Journal 23(1), 117–122 (2013)[13] Murray, N., Marchesotti, L., Perronnin, F.: Ava: A largescale database for aesthetic visual analysis. In: Computer
Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on. pp. 2408–2415. IEEE (2012)[14] Nair, V., Hinton, G.E.: Rectified linear units improve restricted boltzmann machines. In: Proceedings of the 27th
International Conference on Machine Learning (ICML-10).
pp. 807–814 (2010)[15] Ogden, C.K.: The ABC of psychology. Routledge (2013)[16] Patterson, M.: What happens after the pin? examining how
pinterest influences users motivation and behavior (2015)[17] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh,S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein,M., et al.: Imagenet large scale visual recognition challenge.
International Journal of Computer Vision pp. 1–42 (2014)[18] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh,S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein,M., et al.: Imagenet large scale visual recognition challenge.
International Journal of Computer Vision 115(3), 211–252(2015)[19] Schmidhuber, J.: Deep learning in neural networks: An
overview. Neural Networks 61, 85–117 (2015)[20] Speer, R., Havasi, C.: Conceptnet 5: A large semantic network for relational knowledge. In: The People s Web Meets
NLP, pp. 161–176. Springer (2013)[21] Szegedy, C., Toshev, A., Erhan, D.: Deep neural networks
for object detection. In: Advances in Neural Information
Processing Systems. pp. 2553–2561 (2013)[22] Yeo, B.L., Gangjiang, L.: Using image match technology
to improve image advertisement quality (Jan 25 2013), uS
Patent App. 13/750,552[23] You, Q., Luo, J., Jin, H., Yang, J.: Robust image sentiment
analysis using progressively trained and domain transferred
deep networks. In: The Twenty-Ninth AAAI Conference on
Artificial Intelligence (AAAI) (2015)