DEIM Forum 2014 A1-2

Search for Images of Historical Objects Using Wikipedia
Yating ZHANG1

Adam JATOWT1

Katsumi TANAKA1

1

Graduate School of Informatics, Kyoto University Yoshida Honmachi, Sakyo, Kyoto 606-8501, Japan
E-mail:{zhang, adam, tanaka}@ dl.kuis.kyoto-u.ac.jp

Abstract We investigate the temporal image retrieval problem by utilizing knowledge from Wikipedia to help retrieve
images. Since current search engines do not explicitly consider the time related to the objects displayed in images, we believethat our proposed temporal image retrieval algorithm can more completely satisfy search intentions. A search is done by
inputting an entity name (with/without location and time information), and chronologically sorted search results are output tocapture the evolution of the queried entity and consider its co-occurrence with other historically important entities. We propose
a novel way to map images to Wikipedia articles to utilize Wikipedia’s structure to estimate the historical importance of
displayed objects. A combination of criteria is proposed to measure the historical importance of images under entity evolution
scenarios and co-occurrence with other historically important objects.
Keyword Temporal Image Retrieval, Historical Importance, Wikipedia
Object evolution is a new facet discussed in temporal

1. INTRODUCTION
Digital images continue to gain popularity as a form ofimage retrieval problems. Intuitively, the evolution ofdelivering information and conveying ideas . Image search

objects displayed in images can be visualized by theand retrieval play important roles. Within the imagechanges in the appearance of entities over time. Among

retrieval research community, two paradigms have been

the few works [9, 10] that have addressed this issue, thecommonly followed: text-based and content-based. Whennotion of time associated with objects has n ever been

applied to annotated image collections, the first, which is

exploredoften called tag-based image retrieval (TBIR), is both

Kinkaku-ji, users may not only want images of it each year,

efficient and effective. To alleviate the problem of missing

but also those taken during crucial moments of its longand noisy tags, Wu et al. [7] proposed a tag completion

history. In fact, Kinkaku-ji burned down in 1950 and wasstrategy that automatically adds missing tags and corrects

rebuilt in 1955. We believe that its visual timeline

noisy ones. Other proposals [4, 5] solve this problem byrepresentation could be automatically constructed with

applying neighbor voting algorithms to confer more

representative images taken from the three time periods

credibility on visually representative tags. On the otherseparated by the above two dates. In particular, thehand,visualexpected results would contain: an image of the originaldescriptions extracted from images and presents users

Kinkaku-ji before 1950 (Fig. 1a), one taken when it waswith the images that best match the example input image.

burned down (Fig. 1b), and one taken after it was rebuilt

However, in both approaches, the information employed to(Fig. 1c). In our proposed method, such crucial historical

retrieve images is confined to the image itself, such as

moments of an entity are extracted from Wikipedia totags or the visual content of images and queries,

represent the significant evolution of queried objects. The

preventing the usage of such external knowledge bases asadvantage of utilizing Wikipedia reflects its authority andWikipedia.

In

inadequate

handling

thecontent-based

addition,

approach

weofrelieswereoncontext

information

of

but

also

to reflect the evolution of objects over time

http://download.wikipedia.org/enwiki/

oninformation related to many real-world entities.

and their interaction for retrieving historically important

1

query

in

image simages.

atemporality

Therefore, we address these problems by leveraging not
Wikipedia 1

forcredibility as well as its capability to provide historicaleffective/multi-view temporal image retrieval methods.
theexample,thestate-of-the-art image retrieval as well as by the lack of

only

For

by

inspiredobjectexplicitly.

it to retrieve images. We create a set of entities defined
as 𝐸, where each entity 𝑒 is associated with its Wikipedia
article (we limit the entities to those with corresponding
Wikipedia pages). For entity 𝑒, two facets of information
are taken into consideration : object evolution and entity
co-occurrence. We assume that the images depicting any
significant snippets of the given object’s evolution or
essential interaction with other entities can be viewed ashistorical important images for a given object.
We formulate the problem of object evolution as
detecting crucial moments throughout the entire history of⃗ as a vector of historicallya given entity. We define 𝑇
important dates related to entity 𝑒 (we use “year” as a⃗ ); for each year 𝑡, we detect andtime unit for each 𝑡 in 𝑇
extract ⃗⃗⃗⃗⃗⃗𝑊 𝑡 , which represents a vector of representative
words with weights to describe what happened in year 𝑡.

Fig. 1 Expected images of Kinkaku -ji

In addition to the images of an object showing

The other facet, entity co-occurrence, is approached bysignificant changes over time, an image’s historicalcalculating the global importance of co-occurring entities

importance is probably higher when it depicts the relations

based on Wikipedia’s entire link structure and the relative

between a given object and other historically important

importance of co-occurring objects that exist in local

entities. Visual co-occurrence can be considered one

graphsrelation. For instance, Fig. 1d depicts that President Bushdefining local graphs are in Section 5). We utilize thisvisited Kinkaku-ji with Prime Minister Koizumi. Thisco-occurrence information to measure the importance ofimage becomes important due to the appearance ofinteractions

President

given

historically important images in which multiple entities

Kinkaku-ji. A hidden co-occurrence relationship can exist

visually co-occur or are non-visually related to each other.

between objects. For example, The Temple of the Golden

We call co-occurring entity 𝑐𝑜𝑒, so its global importance

Pavilion, (novel by Yukio Mishima) hows a novel calledis 𝐼𝑔 (𝑐𝑜𝑒|𝐸), given all the entities under Wikipedia link

The Temple of the Golden Pavilion (Kinkaku -ji) bystructure and computing the importance of entity 𝑐𝑜𝑒. For

Japanese writer Yukio Mishima. Although the author does

the relative importance denoted by 𝐼𝑟 (𝑐𝑜𝑒|𝑒), the problem

not visually co-occur with Kinkaku-ji in any image, thisis formulated as follows. Given entity 𝑒, we compute thepicture is deemed important since the book significantly
addresses the temple’s history. Accordingly, Enjo (1958“importance” of entity 𝑐𝑜𝑒 with respect to 𝑒 and obtain avector of co-occurring objects ⃗⃗⃗⃗⃗⃗⃗⃗
𝐶𝑂𝐸 with corresponding

Japanese movie) is also chosen. Enjo is a movie based on

combined importance (weighted global importance andMishima’s novel. Under this scenario, we developed arelative importance) of each entity. In this paper, thequantified model to study the historical importance of

co-occurrence importance is independent from time.

imagesofIntuitively, the interaction relationship among entities

co-occurring objects based on Wikipedia’s link structurechanges over time. However, to simplify the problem, we(such as using PageRank) and analyzing the relative

fix this relationship once the link structure of theimportance

Wikipedia pages is established.

Bush,

by

who

is

calculating

of

other

an

importanttheentities

global(both

entityimportancevisuallyandnon-visually) that co-occur with the queried object in agiven image.

1.1 Problem Formulation“centered”

among

on

given

entities

entity 𝑒 (details

forfurther

about

exploring

At query time, when a user issues query 𝑞 (for now welimit queries to entities in Wikipedia), we extend 𝑞 with⃗⃗⃗⃗⃗⃗𝑡 for a given
the top 𝑘 most representative words in 𝑊

Our work is mainly composed of two stages: knowledge

entity 𝑒 and year 𝑡 to build a new query. Then wecollection during the offline stage and image retrieval atextended queries to image search engine such as Flickr to

the query time.

get a collection of images that can be viewed as a

In the offline stage, our task gathers as much seminal

candidate set of images. In terms of entity co-occurrence,

information as possible about the entities and later utilizes

query 𝑞 is extended with the names of other entities withhigh importance given the queried object . The images

detecting not only the historically important time periods

returned from Flickr are then considered candidate imagesof entities but also utilizing the surrounding knowledge torepresenting the interaction of other entities wit h thesupport the significance. Another work by Kim and Xing

given object.

[10] investigates a time-sensitive image retrieval problem

The final step is to select historically important images

by customizing search results depending on the time when

for the queried object from the candidate image set

a user issues a query. For example, if a query is issued for

obtained from Flickr. The knowledge gathered in the

Kinkaku-ji in mid-January, the system will return images

offline stage is leveraged to filter the candidates byshowing it in winter. Although such an approach iscomparing the contextual information of the image withintuitive, we focus on the temporal information related tothe extracted snippets annotating the object evolution anddisplayed objects rather than on the timestamps of images

entity co-occurrence.

to portray the objects at different seasons. Our objective

We clarify our query model as follows:

provides users with important images of a queried entity



Input: entity name 𝑒 (optionally with location 𝑙

considering its entire timeline and its interaction withand time 𝑡).

other entities over history.



Output: list of top k historically important images

Measuring the historical importance of entities isof entity 𝑒 such that each image illustrates theanother important component of our system, and we use it

evolution of entity 𝑒 or annotates the interaction

as a facet to compute the historical importance of images .

with other entities, and overall the results visually

A similar approach was proposed by Takahashi [3] torepresent the history of e.

evaluate the significance of historical entities using

The remainder of the paper is structured as follows. In

Wikipedia’s link structure. However, this approach only

Section 2 we briefly discuss related work followed byfocuses on the global measures of entity imp ortance inclarifying our definitions in Section 3. Sections 4 and 5

which each entity is ranked relative to all other entities

elucidate how to utilize Wikipedia to collect important

mapped in a graph. We propose a composite importancetemporal information and co -occurring entities. In Section

measurement

4 we propose a novel algorithm for detecting the evolution

importance of entities in a local graph with respect to aof objects. Section 5 introduces composite historicalgiven object. Evaluation of relative importance is often

importance considering both the global and relative

essential. For example, regarding Kinkaku-ji, the bookimportance of co-occurring objects with respect to a given

The Temple of the Golden Pavilion is of low importanceentity. Section 6 discusses a two-step image retrieval stage

compared to other novels. However, since it makes a

by gathering candidate images from Flickr and selecting aseminal contribution to Kinkaku-ji’s history, it is highly

final list of images with high historical imp ortance.

important to the temple itself, even if it is not as crucial inConclusions are presented in Section 7.

general as many other books. White and Smyth focus onthat

additionallyexplorestherelativedefining and computing the importance of nodes in a graphrelative to one or more root nodes [11] . Their approach,

2. RELATED WORK
Detecting the object evolution displayed in images canwhich defines weighted paths and explicitly computes theandrelative importance, is also employed in our work. It is

re-ranking problem, which is one of our main objectives.

assumed that the longer a path is from the root node to the

In contrast to temporal text retrieval, temporal image

other node, the less importance is conferred along that

retrieval is a fairly new research topic. [9] first proposed

path. For the computation of global importance, we

the idea of automatically capturing the time dimension

applied Topic-sensitive PageRank [1]. We modified their

included in web images from explicit or implicit temporal

work by introducing Wikipedia as a category and atext queries based on query logs. However, even thoughknowledge basis and extend ed the topics to capture both

extracting dates from query logs may lead to high recall,time and location dimensions to evaluate the global

the precision is generally low. In addition, although theimportance of entities with respect to a specific topic, time ,authors mentioned the notion of the evolution of entities,

and location. Considering our final objective, applying thetheyhistoricalbegeneralized

failedtoasatemporal

implement

any

image

methodretrievalforfinding

importanceofentitiestoquantify

thehistorically important images. In contrast, we solve thisimportance of images is a challenge that has not been

problem by utilizing knowledge from Wikipedia forstudied so far to the best of our knowledge.

the⃗⃗⃗⃗⃗⃗𝑡 at
For entity 𝑒 , we generate a vector of words 𝑊
recorded time point 𝑡 (a year) to describe what happened

performance of search engines to retrieve the mostwith the entity in a particular year. Among those words,

relevant facets of given object query logs and associatedonly those are needed which distinguish the event

tags. Our work is different from theirs in two aspects.

experienced by the entity at time 𝑡 from other events. The

First and foremost, its scope is different: we solve a“representativeness” of these words can be computed bytemporal image retrieval problem, while they focused on

the following equations:

The idea of utilizing Wikipedia resources to retrieve
imageswas

alsointroduced

[8]

to

enhance

retrieving diverse facets with high relevance to an object(without considering temporal notion). Second, our work
enriches the contextual information of images by mapping
each image to several Wikipedia pages that are highly
related to the displayed objects in images. This projection
allows us to extend our work in the future to automaticallycreatehistoricalsummariesofentitiesforbetter𝑃(𝑤𝑗 |𝑒, 𝑡 ) =𝑃(𝑤𝑗 |𝑒) =𝑃(𝑤𝑗 , 𝑒, 𝑡) 𝑁(𝑤𝑗 , 𝑒, 𝑡)=𝑃(𝑒, 𝑡)𝑁(𝑒, 𝑡)(1)

𝑃(𝑤𝑗 , 𝑒) 𝑁(𝑤𝑗 , 𝑒)
=
𝑃(𝑒)
𝑁(𝑒)

𝜎(𝑤𝑗 |𝑒, 𝑡 ) =𝑃(𝑤𝑗 |𝑒, 𝑡 )𝑃(𝑤𝑗 |𝑒)

(2)

.

(3)

Equation (1) denotes the probability of the occurrence

understanding the story behind images.

of 𝑤𝑗 given the occurrence of entity 𝑒 and time 𝑡 .

3. PROBLEM DEFINITION

𝑁(𝑤𝑗 , 𝑒, 𝑡) is the number of sliding windows in which
𝑤𝑗 , 𝑒, 𝑡 are recorded together. Here, a sliding window is

3.1 Object Evolution
Object evolution can be viewed as the conceptual or
physical changes in an object’s attributes, such as namechanges, object splittings, object mergers, or temporalvariations in shape and color. Intuitively, the object
evolution displayed in images can be visualized by thechanges in the appearance of entities by time or identified
as the variations in the contextual information of images .

defined as a fixed number of sentences. Eq. (2) denotes theprobability of the occurrence of 𝑤𝑗 given only theoccurrenceofentity

𝑒

.

In

this

sense,“representativeness” 𝜎 is defined in Eq. (3) as the divisionof the above two probabilities. Intuitively, the word ismore representative if it is more likely to appear in aparticular time and less likely to appear over the entire
timeline of the entity.
We select top 𝑐 words with high “representativeness” to

3.2 Historical Importance of Entities
We define the historical importance of entities as a
combination of global importance and relative importance.
The web can be defined as a very large graph where
nodes represent web pages and directed edges represent
hyperlinks among pages. We construct graphs from
Wikipedia and consider each Wikipedia article an entity(node in the graph). In this context, the global importancerefer to the events that occurred in time 𝑡. Representative
words ⃗⃗⃗⃗⃗⃗
𝑊 𝑡 and corresponding “representativeness ” score⃗⃗⃗⃗
𝑅𝑡 are denoted as follows:⃗⃗⃗⃗⃗⃗
𝑊 𝑡 = [𝑤1 , 𝑤2 , 𝑤3 … … 𝑤𝑐 ]⃗⃗⃗⃗𝑡 = [𝑟1 , 𝑟2 , 𝑟3 … … 𝑟𝑐 ].
𝑅
In this sense, pairs of words and representative scores⃗⃗⃗⃗⃗⃗{𝑊 𝑡 , ⃗⃗⃗⃗
𝑅𝑡 } imply the evolution of an object over time.

of an entity is the “importance” of the corresponding
article given the entire link structure of Wikipedia. Unlike
global measures, the relative importance represents theinteraction relations among entities in the graph.

5. COMPUTING HISTORICAL IMPORTANCE
OF CO-OCCURRING ENTITIESThe historical importance of co-occurring entities isapproached by computing the global importance of

3.3 Historical Importance of Images
The historically important images of a queried entity

entities based on the entire link structure of Wikipedia

are identified as (1) the images depicting the significant

pages and the relative importance of the co-occurred

evolution of an entity in time; (2) the images in which
queried objects co-occur with other entities with highentities with respect to a given entity. Thereafter, we
obtained a vector of co-occurred objects ⃗⃗⃗⃗⃗⃗⃗⃗
𝐶𝑂𝐸 withhistorical importance with respect to the queried object.

corresponding combined importance (weighted global andrelative importance) for a given entity.

4. DETECTING OBJECT EVOLUTION
This

sectiondescribes

ourmethodthat

5.1 Computing Global Importance of Entity
extracts

temporal information related to a given entity and detects
the changes in an entity’s attributes over time.

Based on the link structure of Wikipedia , we computethe historical importance of all the entities in 𝐸 within aspecific topic, time, and location by developing a model ofguarantee convergence to a unique rank vector. ⃗⃗𝑠𝑗 is

topic-time-location sensitive PageRank.

considered a teleport vector in which the user is designed

5.1.1 Outline of Historical Importance Measurement

to jump to a random page chosen non-uniformly where1
𝑠𝑗𝑘 = {
0In our approach to measure the historical importance ofentities, we precompute the importance scores offline as intopic-sensitive PageRank [1]. However, instead of taking|𝑠𝑗 | is the number of 1s in ⃗⃗𝑠𝑗 .

only the topic into consideration, we introduce two otherdimensions for each entity: temporal and spatial. For each

5.1.4 Computation of PageRank Vector of Time Periods

dimension, we compute offline a vector of the scores oftheimportanceofanentitywithrespectto𝑘 ∈ 𝑆𝑗 ,𝑘 ∉ 𝑆𝑗 .

As for the dimension of time, we consider decade thethiscategory to divide time periods [0-2013] into 202 clusters,

dimension’s categories. To be more precise, for each entity,

𝑡0 … 𝑡201 . We also apply the methodology discussed in

we compute a vector of scores with respect to various

Section 5.3 to compute the PageRank vector over different

topics, a vector of scores during different time period s,time periods.

and a vector of scores over distinctive regions. At the

5.1.5 Computation of PageRank Vector of Regions

query time, the historical importance scores are generated

We chose “country” as a unit of regions to denotebased on the topics, time period s, and regions identified

spatial information. The historical importance of an entity

from the query to compose a mixture of scores for thosein most cases varies across countries. For example,

entities that are closely related to the query. Here, we

Yoshida Shōin, a distinguished intellectual in Japan,

assume that the link structure of the Wikipedia pages

wielded great influence during the Meiji Restoration.

implicitly confers the importance of the entities. Under

However, he had no influence on Western countries. In

this hypothesis, we regard each Wikipedia article as an

this scenario, we employ the list of administrative

entity, and the page with the highest topic-time-space

divisions by country [6] based on the ISO 3166 standard 2

sensitive PageRank score has the highest historical

published

importance within a specific topic, during a unique timeStandardization in which there are 198 countries. We use

period, and in a particular region.

𝑟0 … 𝑟197 to represent 198 non-overlapped regions. When

5.1.2 Identification of Topic Category of Entity

considering the definition of countries, we must consider

by

the

International

OrganizationforSince we utilize Wikipedia pages as webpage resources

the changes in borders over time as well as a country’s

to compute the historical importance scores of entities, we

existence, changes in its names, and so on. However, foremployed a list of Wikipedia’s major topic classifications

simplicity, we assume that countries existed more or less[2] to map each Wikipedia page: each entity to one ofwithin their current borders and use fixed regions 𝑟0 … 𝑟197

these categories to further compute the PageRank vectors

throughout time. Obviously, since this assumption is quiteof the topics. This list contains 26 categories and is used

strong, in the future, we plan to extend this approach using

throughout Wikipedia to organize the presen tation of links

distinctive lists for different periods to capture certain

toaspects of country evolution over time.

articles[2].

Therefore,

these

26

categories

areFinally, we compute PageRank vectors within different

guaranteed to cover all Wikipedia articles.
For each topic category 𝑐𝑗 , we also compute a bag ofregions based on the methodology discussed in Section

words representation 𝑊𝑗 , consisting of all the nouns

5.3.

displayed in Wikipedia articles belonging to this category.

5.1.6 Computation of Historical Importance Score

Let 𝑊𝑗𝑘 denote the number of occurrences of word k in

Based on Query

Wikipedia articles under category 𝑐𝑗 .

The final historical importance score of an entity is

5.1.3 Computation of PageRank Vector of Topics
We adopt the topic-sensitive PageRank and computerank vectors over the topics:⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗
𝑅𝑎𝑛𝑘𝑗 = (1 − 𝛽)𝑀 × ⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗
𝑅𝑎𝑛𝑘𝑗 + ⃗⃗𝑠𝑗

computed at the query time. Our system provides users
with a query box with three input areas: entity, time , andregion. The time and region query parts are optional.

𝛽.

Given query q, we compute the class probabilities for each|𝑠𝑗 |Let 𝑆𝑗 be the set of pages in Wikipedia-topic category 𝑐𝑗 .
⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗𝑗 for topic 𝑐𝑗 . 1 − βWe compute PageRank vector 𝑅𝑎𝑛𝑘
is a damping factor added during the rank propagation to2

ISO 3166 is the International Standard for country codesand codes for their subdivisions.
http://www.iso.org/iso/country_codescategory under three dimensions: topic, time, and region.

|𝒫(𝑒,𝑐𝑜𝑒)|a. For topics, 𝑃(𝑐𝑗 |𝑞) can be easily computed from the𝐼𝑟 (𝑐𝑜𝑒|𝑒) =∑𝜆−|𝒫𝑖 | ,

(4)

𝑖=1

topic bag of words 𝑊𝑗 , and part of 𝑃(𝑐𝑗 ) is treated aswhere λ is a scalar coefficient, 1 ≤ λ ≤ ∞, that determines

uniform:

how much importance is conferred from 𝑒 to 𝑐𝑜𝑒. In this𝑃(𝑐𝑗 ) ∙ 𝑝(𝑞|𝑐𝑗 )𝑃(𝑐𝑗 |𝑞) =∝ 𝑃(𝑐𝑗 ) ∙ ∏ 𝑝(𝑞𝑖 |𝑐𝑗 ).
𝑝(𝑞)formulation,theamountofimportancedecaysexponentially with the increase in the path length.

𝑖

Before exploring the set of paths from 𝑒 to 𝑐𝑜𝑒, web. To calculate 𝑃(𝑡𝑗 |𝑞), for the time period, we map thedefine our paths as the shortest path, which are the sets oftime in the query box to one of the time period categories.

K-short Node-Disjoint Paths that have neither edges nor

The input of time is currently limited to a year format,

nodes in common. We use a set of paths instead of one

such as 1889, a period format, such as 1889 -1950, or ashortest path between two vertices because multiple paths

century format, such as 20 t h . Then we map the time

add more importance to 𝑐𝑜𝑒 relative to 𝑒 than only

information to one of the time classes.

considering the shortest path. A breadth first searchc. For regions, we limit the queries to city and country

algorithm

isemployed

to

find

a

good

set

of

names. Just as the case of time period s, we let queries fall

paths 𝒫(𝑒, 𝑐𝑜𝑒).

into one spatial category, 𝑃(𝑟𝑞 |𝑞) = 1, and the

The final historical importance is given by linearly
combining global importance 𝐼𝑔 (𝑐𝑜𝑒|𝐸) and relative

probabilities for other classes are 0.
Next,

we

computetheimportance 𝐼𝑟 (𝑐𝑜𝑒|𝑒):query-sensitive

historicalimportance score of each Wikipedia page. Let rank jd be therank of Wikipedia page d given by the rank vector.
Weights 𝛽1 , 𝛽2 and 𝛽3 denote the weight of each dimension.
In most cases, they are supposed to be equal, and when thequerycontains

no

time

or

space

information,thecorresponding weight is set to 0:
𝑆𝑐𝑜𝑟𝑒𝑡𝑜𝑝𝑖𝑐⋅𝑡𝑖𝑚𝑒∙𝑠𝑝𝑎𝑐𝑒𝐼(𝑐𝑜𝑒|𝑒) = (1 − 𝜃) ∙ 𝐼𝑔 (𝑐𝑜𝑒|𝐸) + 𝜃 ∙ 𝐼𝑟 (𝑐𝑜𝑒|𝑒).

(5)

Here, 𝜃 represents the “closeness” between 𝑒 and 𝑐𝑜𝑒. In
Eq. (5), when 𝜃 is larger than 0.5, then such 𝑐𝑜𝑒 closer
to 𝑒 will be preferred. For example, in the Kinkaku-ji
case, according to Fig. 2, The Temple of The Golden
Pavilion has higher relative importance than George W.
Bush, relative to Kinkaku-ji, since there is no link
between them in the Wikipedia pages. In this sense, if 𝜃 is

= 𝛽1 ∙ ∑ 𝑝(𝑐𝑗 |𝑞) ∙ 𝑟𝑎𝑛𝑘𝑐𝑗𝑑 + 𝛽2

set to 1, Fig. 1d will not be returned.

𝑗∙ ∑ 𝑝(𝑡𝑗 |𝑞) ∙ 𝑟𝑎𝑛𝑘𝑡𝑗𝑑 + 𝛽3𝑗∙ ∑ 𝑝(𝑟𝑗 |𝑞) ∙ 𝑟𝑎𝑛𝑘𝑟𝑗𝑑 .
𝑗

5.2 Computing Relative Importance of Entity
Using Weighted Path
Wikipedia is mapped to a directed graph 𝐺 = (𝑉, 𝐸)
in which nodes 𝑉 are defined as Wikipedia articles(also entities) and edges 𝐸 are the hyperlinks among

Fig. 2 Interaction among Kinkaku-ji and other entities

articles. We use 𝒫(𝑒, 𝑐𝑜𝑒) to denote a particular set ofpaths between a given entity and its co -occurring
entities.
According to [11], the following are the two main
properties that should be satisfied using the approach ofweighted paths: 1) two nodes are related by the paths thatconnect them and 2) the longer a path is, the lessimportance is conferred along that path. To achieve these
we define 𝐼𝑟 (𝑐𝑜𝑒|𝑒) as follows:

6. IMAGE RETRIEVALIn this paper, we proposed a two-step image retrieval
approach by first utilizing Flickr API to obtain as many
images as possible related to the queried entity andretrieved a final list of images with high historicalimportance.

6.1 Creating Image Collection

At the query time, when a user issues query 𝑞 (i.e., anentity with a corresponding Wikipedia page), we extend 𝑞⃗⃗⃗⃗⃗⃗𝑡 and with years in 𝑇
⃗ . Then we callwith words in 𝑊
Flickr API with such a query to get a set of candidate
images 𝑆. Each image is represented by a vector of tags.

6.2 Computing Historical Importance of Images
The final list of images is classified in two groups: onethat implies the evolution of the queried object, andanother that depicts the interaction with other entities. For
object evolution, we compute the TF-IDF score for eachtag of the image, based on the frequency of tag d in image
𝐼𝑖 compared to the frequency of tag d in the entire image
set 𝑆. Then for image 𝐼𝑖 , we obtain a vector of tags with
their TF-IDF scores that consist of pairs of tags and scores⃗ ,𝑉⃗ }. Remembering the pair of values wedenoted as {𝐷⃗⃗⃗⃗⃗⃗𝑡 , 𝑅⃗⃗⃗⃗𝑡 } for queried entities, we
computed in Section 4, {𝑊compare these two values by calculating their cosine
similarity. If the similarity exceeds threshold ε, then image
𝐼𝑖 is identified as a historically important image that isuseful to depict the evolution of a queried object.
With respect to co-occurrence with other entities, we
detect the appearance of co-occurring entity names inimage tags and rank the image by 𝑰(𝒄𝒐𝒆|𝒆) discussed in
Section 5. Candidates with high 𝑰(𝒄𝒐𝒆|𝒆) scores are thenreturned.

7. CONCLUSION
We investigated a variant of temporal image retrieval, in
which object evolution and interaction among entities are
explicitly considered and the historical knowledge derived
from Wikipedia is utilized to help retrieve images. This
paper ’s contribution is providing a novel way to enable
users to visualize the evolution of objects as well as therelationships among entities. In this context, we developed
a two-stage temporal image retrieval model by gathering
the knowledge of entities by time and chronologically
retrieving and re-ranking historically important images.

ACKNOWLEDGEMENTS
This work was supported in part by Grants-in-Aid for
Scientific Research (No. 24240013) from MEXT of Japan.

REFERENCE[1] T. H. Haveliwala. Topic-sensitive PageRank: A
context-sensitive ranking algorithm for web search.
IEEE Transactions on Knowledge and Data
[2] Wikipedia: Category: Main topic classifications

http://en.wikipedia.org/wiki/Category:Main_topic_cl
assifications[3] Y. Takahashi, H. Ohshima, M. Yamamoto, H. Iwasaki,
S. Oyama, K, Tanaka. Evaluating Significance ofHistorical Entities Based on Tempo -Spatial Impacts
Analysis Using Wikipedia Link Structure . In HT2011:
Proceedings of the 22nd ACM Conference onHypertext and Hypermedia, pages 83-92, June 6-9,
2011.
[4] X. Li, C. G. M. Snoek, and M. Worring. Neighbor
Voting: Learning Tag Relevance by Neighbor Voting
for Social Image Retrieval. In Proc. of MIR, pages
180-187, 2008.
[5] A. Sun and S. S. Bhowmick. Quantifying Tag
Representativeness of Visual Content of Social
Images. In Proceedings of the international
conference on Multimedia, Pages 471-480, 2010
[6] Wikipedia. Table of administrative divisions bycountry
http://en.wikipedia.org/wiki/Table_of_adminstrative_divisions_by_country[7] L. Wu, R. Jin and J, A.K. Tag Completion for Image
Retrieval. IEEE Trans. Pattern Analysis and MachineIntelligence, 35(3), 2011.
[8] R. Zwol and B. Sigurbjornsson. Faceted Explorationof Image Search Results. In Proceedings of the 19th
international conference on World wide web , Pages
961-970, 2010.
[9] G. Dias, J. G. Moreno, A. Jatowt and R. Campos.
Temporal Web Image Retrieval. 19th International
Symposium on String Processing and Information
Retrieval 7608(2012), pages 199-204, 2012.
[10] G. Kim and E. P Xing. Time-Sensitive Web Image
Ranking and Retrieval via Dynamic Multi-Task
Regression. In Proceedings of the 6th ACM
International Conference on Web Search and Data
Mining, pages163-172, February 4–8, 2013.
[11] S. White and P. Smyth. Algorithms for Estimating
Relative Importance in Networks. In Proceedings of
the 9th ACM SIGKDD international conference on
Knowledge discovery and data mining, Pages
266-275, August 24-27, 2003.