2012 IEEE 12th International Conference on Data Mining

Towards Automatic Image Understanding and Mining via Social Curation
Katsuhiko Ishiguro, Akisato Kimura, and Koh Takeuchi
NTT Communication Science Laboratories
NTT Corporation, Kyoto, Japanishiguro.katsuhiko@lab.ntt.co.jp, akisato@ieee.org

Abstract—The amount and variety of multimedia data suchas images, movies and music available on over social networks
are increasing rapidly. However, the ability to analyze andexploit these unorganized multimedia data remains inadequate,
even with state-of-the-art media processing techniques. Ourﬁnding in this paper is that the emerging social curation service
is a promising information source for the automatic understanding and mining of images distributed and exchanged via
social media. One remarkable virtue of social curation service
datasets is that they are weakly supervised: the content in theservice is manually collected, selected and maintained by users.
This is very diﬀerent from other social information sources,
and we can utilize this characteristics for media content mining
without expensive media processing techniques. In this paper
we present a machine learning system for predicting viewcounts of images in social curation data as the ﬁrst step toautomatic image content evaluation. Our experiments conﬁrm
that the simple features extracted from a social curation corpus
are much superior in terms of count prediction than the goldstandard image features of computer vision research.

rates of the social curation service and the eﬀectivenessof advertisements. However, the automatic understandingof images, one of the ultimate goals of computer visionresearch, remains quite diﬃcult for the unorganized image
content in the Web and SNSs, even for state-of-the-art
techniques (e.g. [5], [6], [7]).
The main claim of this paper is that we are able tounderstand and mine images in SNSs by utilizing social
curation data. We assume that the contents of a curated
list, including image content, are manually organized to
fully convey the curators intentions. It follows that we
can infer the context or evaluation of an image from thesocial information and contextual features of the other (nonimage) content in the curation list, and the curation list
itself; the goal is to dispense with expensive computer
vision techniques. The only technical study related to social
curation service is related to text corpus analysis [8]. To
the best of our knowledge, this is the ﬁrst work to examinesocial curation data for automatic image understanding andmining.
The rest of the paper is organized as follows. In the second
section, we review related works. In the third section, we
explain the social curation service, our target data source,and detail the dataset speciﬁcations. The fourth section
is devoted to the problem formulation of predicting viewcounts of an image included in a curation list, as a ﬁrst step
towards image understanding and mining via social curation.
The ﬁfth section describes experiments conducted to conﬁrm
the eﬀectiveness of the social curation information comparedto the current gold-standard features used in computer vision research. The last section concludes this paper with a
discussion about future works.

Keywords-Social curation, automatic image understanding
and evaluation, feature extraction, regression

I. Introduction
The amount and value of information distributed andconsumed in social networking services (SNS) are increasing
rapidly in both business and academic settings. At the sametime, there are urgent demands to develop an eﬃcient andintuitive method to identify the interesting and valuable information from the growing volume of social media content
such as texts , images, movies and music [1], [2], [3], [4].
In this setting, social curation service is emerging as a
new way to interact with social media. At the most basic
level, curation service oﬀers the ability to (i) bundle a collection of content from diverse sources, (ii) re-organize themto give one own perspective, and (iii) publish the resulting
story to consumers. In Fig. 1, we present a schematic view of
curation service. A curator organizes a curation list, which
is a compilation of social media content, from a pool ofcontent created by others. It is worth noting that a curation
list is a collection of social media content that is manually
collected, selected and maintained by its curator.
As described later, social curation service and curation
lists contain a lot of images and movies. The former is particularly attractive to many SNS users. Therefore, an automatic
scheme for understanding and evaluating image content
on SNSs would be beneﬁcial for improving click-through1550-4786/12 $26.00 © 2012 IEEE
DOI 10.1109/ICDM.2012.37

II. Related Works
One notable trend in SNS-related research is agglomerating multiple information sources or services to obtain adeeper understanding of social media content. For example,
Mejova and Srinivasan [9] employ a domain adaptation technique for sentiment analysis in three diﬀerent social media
streams: weblogs, review articles, and tweets on Twitter.
The authors of [10] extend a topic model [11] to associate
tweets and real events to discover topical segmentation
in a event. Kulshrestha et al. [12] studied the impact ofoﬄine geolocations on online social network activities and675
906

Content Creators

Content Consumers

Curatorcurated!

tweet
curated!

tweet

photo
photo

A curation list

blog
photo

curated!

video

tweet

tweet
tweet
not curated!

Figure 1. Schematic view of social curation service. There are three kinds of users. Content creators generate and post content in a variety of data formats
and domains. Curators select posted social media content from various sources, evaluate and re-organize it as a curation list to represent their perspective,
opinions, or interests. Content consumers enjoy and share these curation lists as a new type of social media content.

participants. However, the ﬁrst two studies focus on thesame modality: namely, text-based datasets. In this paper,
we employ the social curation service as a complimentary
information source for the automatic understanding andmining of image content in social media. This is closer
to [12] in the sense that the information source is crossmodal: a social network structure with oﬄine geographicalinformation, as in our case social curation lists are associatedwith images.
To the best of our knowledge, there are no studies dealing
with social curation service, excepting a work by Duh etal. [8]. This paper analyzed curation lists consisting of
Twitter messages (tweets). They also studied the objectives
and topics of curation lists, and reported that there aremany styles and usages among social curation services. The
diﬀerence from our work is again in the modality. The focusof the authors was unimodal: the authors of [8] mainly
focus on text messages (i.e. tweets). In our work, we extract
various kinds of information (features) from a curation list
to understand and evaluate the image data.

content creators, as well content expressed by the curation
lists. Note that a user can be a content creator, curator, andcontent consumer at the same time.
We cannot emphasize too much that each curation list
is a kind of weakly supervised, organized social dataset.
This means that social media items in the same curation list
are expected to share the same context to a certain degree:
a curation list is manually generated to fully convey one
idea to the consumer. This is a very distinct characteristiccompared to other social media datasets that are unorganizedin many cases. Our main idea is avoiding expensive andcomplicated computer vision techniques for mining image
content on SNSs. Instead, we employ the easily available
features extracted from social curation lists for easy andprecise image-content understanding. Because curation lists
are weakly organized and supervised, such easy solutions
may perform well against sophisticated technique appliedon unorganized datasets.

III. Social Curation

In this paper, we focus on the social curation service
called Togetter1 . Togetter is a social curation service mainly
based on tweets (microblogs) generated on Twitter. A
screen-shot of a curation list on Togetter is presented inFig. 2. One reason for choosing Togetter is its great numberof social curation lists. Togetter is rapidly growing in Japan.
The number of monthly page views exceeded 10 Million by
May 2012, which is three times larger than year before2 .
We collected curation lists that were created from September 2009 to April 2011 and that contain image or movie
content. It is impossible to extract a complete set of such
lists, thus we resort to a simple alternative. Most image andmovie content is posted as a hyperlink to the ﬁle on various

B. Dataset of Social Curation: Togetter

A. Social Curation
Users involved in social curation service are classiﬁed into
three types in [8] (Fig. 1). First, content creators generatesocial media content (or simply, content) that is posted toSNSs. Formats and domains of the content are diverse:
text messages like tweets, photos taken by mobile phones,
weblogs, movies, and so on. Second, curators collect andevaluate this posted content, and re-organize it to form
compound content (called a curation, a summary or a curation list) based on the opinions, perspectives and interestsof the curators. Usually, a curation list is created by one
user. However, some curation lists are generated through theinteraction of multiple curators. Third, content consumers
enjoy, share and consume social media content created by

1 http://togetter.com
2 Based on reports of donnnamedia (donnamedia.shoeisya.jp)

907
676

Table I
Specifications of crawled Togetter dataset

Title
Description

Total curation lists
Total curation lists including image or movie content
Total tweets
Total tweets including image or a movie content
Unique image or movie content
Unique users
Unique curators
Unique words in tweets

Views & Fav.s recieved

Listed and re-ordered contents (tweets)

Number
96,50632,823
10,238,802
1,585,448
316,384
106,066
31,661
768,041

IV. Image View Count Prediction via Social Curation
A. Objective
Automatic image and movie understanding is one ultimate
goal of computer vision research. Many researches related tovisual features [13], [14], statistical models [15], [16], [17],and dataset studies [18] were studied for that purpose. These
techniques enable us to recognize objects in images, detecthuman faces, and track moving targets in movies. However,it remains diﬃcult to quantify subjective assessments ofimage content such as “Is this image funny?” with stateof-the-art computer vision methodology.
This paper introduces an alternative approach; we employ view counts of image content as a quantitative andmeasurable proxy of such evaluations. As explained, mostimages are stored on image uploading services. Some ofthose are equipped with view counters, or access countersto the contents. We think that view count is a naive but goodproxy for human subjective evaluations.
Our task here is to predict view counts of image content
included in curation lists based on simple features found in
curation lists; no computer vision method is used. Predicting
view counts of images will help catch signs of emerging
trends in SNSs, and mine popular content from social media.
Ideally, we would like to predict view counts of image
content before the images are included in some curation lists.
A popular image will increase views and access to the list
including the image. Thus, we can recommend to curators
image content that will draw much attention from content
consumers. We are aware that there is some technical gap
between this goal and the task validated in this paper. Thus,
we focus on presenting the usefulness of social curation
services and the dataset in image content mining.

Image contents (hyperlink)

Figure 2. Screen-shot example of a curation list on Togetter, which isa Japanese social curation service for Twitter. Tweets selected by curators
are listed and displayed in an arbitrary order. Hyperlinks to other sites, andmultimedia contents such as images could be included.

uploading services. We speciﬁed the major image and movie
uploading services and their domains in advance. We checkall tweets with hyperlinks and abbreviated hyperlinks as towhether the hyperlink matched one of these domains. A
match indicated that the tweet had image or movie content.
Table I summarizes the speciﬁcations of the dataset. The
number of total curation lists in Togetter was 96,506. 32,823
lists (34%) out of 96,506 lists contained an image or movie
content. The number of total tweets was approximately 10
million. Of these 10 million tweets, 1.58 million contained
hyperlinks to image or movie content. This means 16% of
tweets in social curations refer to images or movies. Out of
the 1.58 million hyperlinks, the number of unique addresses,
namely unique images and movies, was 316,384.

B. Problem Formulation
Formally, we predict view count yi of image or movie
content i from information of the content xi . This is atypical regression problem: i.e. we try to minimize the error
between the predicted view count yi and the true view countŷi by modifying an unknown parameter w that governs theregression function ŷi = f (xi ; w).
Given image content and social curation lists, we extract
several features xi and predicted a view count for each

It is surprising that the fully curated Togetter dataset
contains such a large number of image and movie items.
These ﬁgures indicate the popularity of image content insocial media. Based on this observation, we assume that
our curation dataset of Togetter is useful for conﬁrming theautomatic image understanding and mining in social media.

908
677

Curation List
Title
Descriptionii) The number of tweets that contain hyperlinks to thespeciﬁed image or video content.

Other Lists# Views# Favorites

Feature Extractioni) social curation features
Features related to curation lists

Content Creator

tweet

These two features measure popularity of the image andmovie content in the social curation service.
2) Text features: Text messages can directly represent theintentions, opinions, or emotions of content creators andcurators. Thus, carefully designed text features would be
useful in predicting responses to image and movie content.
Since the objective of our problem is to compare the social
curation features with computer vision techniques, however,
we resort to simple and easy features. Our assumption isthat if the topics or contexts of the list and the comments
attached to images match well, then the images will attract
much attention and gain view counts.
Our text features are computed as follows. First, we
extract three parts of texts that are found in Togetter curation
lists, and compute a Bag-of-Words (BoW) histogram fromeach part. The ﬁrst part is the title and description of thecuration list. This part is edited directly by curators; so we
expect this part concisely describes the entire context of thelist. The second part is all texts of the curated contents
in the list: tweets, comments to images, and so on. The
BoW histogram of this part is a direct summarization of thecuration list. The third part is all tweets among curation lists
that have a hyperlink to the image content. The histogramof this part encompasses the responses of SNS users withregard to the content. From these BoW histograms, we
compute three cosine distances based on our assumption.

photoii) text features
Features extracted from text
photo
photo

tweet

iii) creator features
Features related tothe creator of the image

tweet

View Count Prediction= ?? views

photo

Figure 3.

Experimental setup.

image content. Social curation lists contain many kinds ofinformation that are useful for predicting view counts. For
example, if the curation list that the image is included in isaccessed by many content consumers, then the view countof the image content is expected to increase; or, if the image
well matches the context of the curation list, the image will
attract much more attention.
We compare these curation-based features to goldstandard image features used in the computer vision researchcommunity to show that social curation is a promising
new data source for automatic image understanding andmining in SNSs. Note that our method is applicable to both
image and movie content because it does not use any image
processing.
We prepare three curation-related features in this paper as
explained below.
1) Social curation features: Social curation features arecomputed based on social inﬂuence and characteristics ofthe curation list and the image content.
The ﬁrst ﬁve features are dependent on the curation list
the image is included in. Therefore, all images in the same
list will have the same feature values.
i) The total number of tweets in the curation list.
ii) The total number of unique content creators (who post
tweets) in the curation list.
iii) The total number of unique image and movie content
included in the curation list.
iv) The total number of favorites the curation list receives.
v) The total number of view counts the curation list
receives.
The last two statistics are oﬃcially provided by the Togetter
service. These ﬁve features measure popularity and diversityof the curation list.
The following two features are dependent on the image
and movie content.
i) The number of lists that contain this content.

i) Distances between the ﬁrst and the second BoWs.
ii) Distances between the ﬁrst and the third BoWs.
iii) Distances between the second and the third BoWs.
Feature i) computes text context similarities between the{title, description} and the tweets in the list. In other words,
this feature is a measure of the similarity between curators’
intention and the actual context of the list. Feature ii) computes text context similarities between the {title, description}and the responses to the focused image content. Namely,
this feature is a measure of the similarity between curators’
intention and observed responses to the image content inSNSs. Feature iii) computes text context similarities between
the tweets in the list and the responses to the image content.
In other words, this feature is a measure of the similarity
between the actual context of the list and observed responses
to the image content in SNSs.
Also, we compute binarized versions of three BoWs. We
binaraize the BoW histograms by thresholding in order toabsorb the diﬀerence in lengths and numbers of tweets
among the curation lists. We compute three cosine distances
for these binarized BoWs in the same manner. Thus, we
ﬁnally obtain six text features.
3) Content creator features: We observe that there might
be a correlation between the view count of an image content
and the information provided by the creator of that content.

909
678

dataset of Togetter. Also, the images used in the experiments
must be linked to true view counts. We found 22,024 images
satisfying this requirement, and used all of them in theexperiments. The vocabulary size of text features was set toV = 50, 000. The vocabulary size of image features was set
to K = 1, 000. The evaluation criterion was mean squared
error (MSE). We computed the MSEs of each feature by
10-fold cross validation.
The distribution of view counts is skewed: the minimum
view count is 0, while the maximum count is 1,288,507.
Therefore, we used the logarithm of view counts in theexperiment. This yielded the average and the variance ofthe log view counts of 4.3698 and 3.0125, respectively.

We compute the following features related to creators of thecontent to utilize this correlation.
i) The number of users who follow the creator of thecontent.
ii) The number of users the creator of the content follows.
iii) The number of favorites set by the creator.
iv) The number of “Lists” (an oﬃcial Twitter function,
not curation lists) the creator is included in.
v) The main language of the creator of the content.
vi) When the creator of the content started using Twitter.
4) Combining features: We concatenate three types offeatures into a single vector and use the vector as the social
curation feature vector of content i xSi .
There are several approaches to combining the three typesof features. In the experiments, we tested the following
combinations: i) social curation features, ii) social curation
features + text features, iii) social curation features + creatorfeatures, and iv) social curation features + text features +creator features. If an image is selected by multiple curation
lists, we simply add the feature values taken from all lists.
5) Regression model: We employ Support Vector Regression (SVR) (cf. [19]) as the regression function. SVR
is known for its powerful regression performances, and isused as one of the standard regression models. We use an
implementation provided by libSVM [20]. As the kernel
function, we choose the standard RBF kernel. We experimentally optimized the soft margin parameter and the kernelparameter. Other parameters were set to default values.

B. Results and Discussions
Table II lists the MSEs for each feature choice. As evident
from the table, social features xSi always beat image features
xiI . The MSEs of social features are almost half those ofimage features, which are high-dimensional and known for
great performance in computer vision tasks. In fact, theMSEs of image features are close to the variance of log viewcounts. This means that the predictions by image features areat the “chance level”.
As the table shows, a combination of social curation
features and text features improves the prediction accuracy
slightly. However, incorporating text features to a combination of social curation features and creator features degrades
the MSE. For the best performance, obviously we needfurther investigation of feature designs and combinations.
libSVM [20] provides a feature scaling function in order
to absorb the scale diﬀerences among feature elements. We
re-scaled all the feature elements between [0, 1], and reran
the experiments. The results are also shown in Table II. We
conﬁrmed that the social features are much better than image
features. Also, we observed that feature scaling improves
prediction accuracy in some cases, but not in others. To further improve prediction accuracy, we need to more carefully
consider the diﬀerences in scales between features.
Finally, we discuss the diﬀerences between our results
and the works by van Zwol et al. [23]. The authors of [23]proposed a classiﬁer that predicts whether a user will favora photo on Flickr. They tested combinations of text features,visual features and social features, and reported that thesocial features performed well, and combinations with otherfeatures improve, in most cases, prediction accuracy. Thisis diﬀerent from our results. One possible explanation isthat popularity among SNSs diﬀers fundamentally from thepreference of a speciﬁc user: in SNS, popularity is the resultof many users with diﬀerent preferences.

V. Experiments
A. Image features and data preparations
For comparisons, we need to choose visual features forimage content. In this paper, we focused on only image
content for ease of experimentation, and chose SIFT [13] as
the main visual feature: a gold-standard feature in current
computer vision research. A SIFT feature is deﬁned in a
local “interest” point in the image; it measures the orientations of edges at multiple scales. We collect a set of SIFT
features from many interest points of among images, anddiscretize them by K-means clustering. After that, the setof SIFT features of an image is encoded to a histogram of
Bag-of-visual words (K words) as in BoW of texts.
We employ an implementation provided by the authorof [21] for image feature extraction. We extract SIFT, CSIFT [22], OpponentSIFT, and Transformed Color Histogram features from each image. All features are discretized
into K visual words, and encoded as histograms of K
visual words. We have tested two types of image feature
usage. The ﬁrst one only employs SIFT features, resultingin K-dimensional image feature vector xiI . The second one
employs all four features, resulting in 4K-dimensional xiI .
Next, we explain how we prepared the dataset. Since wefocus on only image content, we cannot utilize the whole

VI. Conclusion
In this paper, we focused on the emerging social curation
service as a new source for image and movie mining in social

910
679

Table II
Mean squared errors (MSE) of view count regression by SVR. The
target is a natural logarithm of view counts. Smaller values are better.

[7] D. Parikh and K. Grauman, “Interactively building a discriminative vocabulary of nameable attributes,” in Proc. CVPR,
2011.

Types of features
Social curation
Social curation + text
Social curation + creator
Social curation + text +creatorimage features (SIFT)
image features (All)var. of log view counts[8] K. Duh, T. Hirao, A. Kimura, K. Ishiguro, T. Iwata, and C.M. Au Yeung, “Creating stories: Social curation on twitter
messages,” in Proc. ICWSM, 2012.

Feature Dim.
7
13
13
19

MSE
1.5441
1.52341.26501.2668MSE(scaled)
1.42471.9646
1.16291.3013

1000
4000
-

3.02893.02873.013

3.31903.3124
3.013[9] Y. Mejova and P. Srinivasan, “Crossing media streams withsentiment: Domain adaptation in blogs, reviews and Twitter,”in Proc. ICWSM, 2012.
[10] Y. Hu, A. John, D. D. Seligmann, and F. Wang, “What were
the tweets about? topical associations between public eventsand twitter feeds,” in Proc. ICWSM, 2012.

media. A key insight is that a curation list, which is a usergenerated agglomeration of social media content, is weakly
supervised: represent the manual collection, selection, andmaintenance by curators. This is a unique characteristiccompared to other social data, and we are able to understandand mine images in SNSs by fully utilizing social curation
data. We conﬁrmed that our Togetter dataset contained many
image and video items. Experiments were conducted on theprediction of view counts of image content as a ﬁrst step forautomatic image understanding and mining in social media.
The results show that the social curation information is far
superior in predicting the view counts of images to the goldstandard image features used in computer vision research.
In this paper, we investigated only a speciﬁc curation
dataset for a speciﬁc task. We are aware that there are many
open problems. First, there is no guideline for designingand choosing social curation features, and combining visualfeatures. Second, we have to investigate social features ina larger dataset, and other tasks such as image retrieval.
Finally, applying social curation information to other domains such as natural language processing, music and audio
processing would be fruitful for further development ofsocial data mining technologies.

[11] D. M. Blei, A. Y. Ng, and M. I. Jordan, “Latent Dirichlet
allocation,” Journal of Machine Learning Research, vol. 3,
pp. 993–1022, 2003.
[12] J. Kulshrestha, F. Kooti, A. Nikravesh, and K. P. Gummadi,“Geographic dissection of the twitter network,” in Proc.
ICWSM, 2012.
[13] D. G. Lowe, “Distinctive image features from scale-invariant
keypoints,” International Journal of Computer Vision, vol. 60,
no. 2, pp. 91–110, 2004.
[14] A. Torralba, K. P. Murphy, W. T. Freeman, and M. A.
Rubin, “Context-based vision system for place and object
recognition,” in Proc. ICCV, 2003.
[15] V. N. Vapnik, The Nature of Statistical Learning Theory.
Springer, 1995.
[16] P. Viola and M. Jones, “Rapid object detection using a boosted
cascade of simple features,” in Proc. CVPR, 2001.
[17] J. Sivic, B. C. Russell, A. Zisserman, W. T. Freeman, and
A. A. Efros, “Unsupervised discovery of visual object classhierarchies,” in Proc. CVPR, 2008.
[18] A. Torralba, R. Fergus, and W. T. Freeman, “80 million tiny
images: A large dataset for nonparametric object and scene
recognition,” IEEE Trans. PAMI, vol. 30, no. 11, pp. 1958–
1970, 2008.

References[1] L. Hong, O. Dan, and B. D. Davison, “Predicting popularmessages in twitter,” in Proc. WWW, 2011.

[19] A. J. Smola and B. Schölkopf, “A tutorial on support vector
regression,” Statistics and Computing, vol. 14, no. 3, pp. 199–
222, 2004.

[2] H. Lakkaraju, A. Rai, and S. Merugu, “Smart news feeds forsocial networks using scalable joint latent factor models,” inProc. WWW, 2011.

[20] C. C. Chang and C. J. Lin, “LIBSVM: A library for support
vector machines,” ACM Trans. Intelligent Systems and Technology, vol. 2, pp. 27:1–27:27, 2011, software available at
http://www.csie.ntu.edu.tw/ cjlin/libsvm.

[3] J. Lin, R. Snow, and W. Morgan, “Smoothing techniques
for adaptive online language models: Topic tracking in tweet
streams,” in Proc. KDD, 2011.

[21] K. E. A. van de Sande, T. Gevers, and C. G. M. Snoek, “Evaluating color descriptors for object and scene recognition,”
IEEE Trans. PAMI, vol. 32, no. 9, pp. 1582–1596, 2010.

[4] D. Rao, M. Paul, C. Fink, D. Yarowsky, T. Oates, and
G. Coppersmith, “Hierarchical Bayesian models for latentattribute detection in social media,” in Proc. ICWSM, 2011.

[22] G. J. Burghouts and J.-M. Geusebroek, “Performance evaluation of local colour invariants,” Computer Vision and Image
Understanding, vol. 113, no. 1, pp. 48–62, 2009.

[5] Y. LeCun, F. J. Huang, and L. Bottou, “Learning methods
for generic object recognition with invariance to pose andllighting,” in Proc. CVPR, 2004.

[23] R. Van Zwol, A. Rae, and L. G. Pueyo, “Prediction offavourite photos using social, visual, and textual signals,” inProc. ACMMM, 2010.

[6] A. Opelt, A. Pinz, M. Fussenegger, and P. Auer, “Genericobject recognition with boosting,” IEEE Trans. PAMI, vol. 28,
no. 3, pp. 416–431, 2006.

911
680