User-Powered “Content-Free” Approach to Image Retrieval
Takeo Kanade and Shingo Uchihashi
Robotics Institute and ECE Department
Carnegie Mellon University
5000 Forbes Avenue, Pittsburgh, PA USA{tk,shingo}@cs.cmu.edu
Abstract
Consider a stereotypical image-retrieval
problem; a user submits a set of query images to asystem and through repeated interactions during
which the system presents its current choices andthe user gives his/her preferences to them, thechoices are narrowed to the image(s) that satisfies
the user. The problem obviously must deal withimage content, i.e., interpretation and preference.
For this purpose, conventional so-called contentbased image retrieval (CBIR) approach uses
image-processing and computer-vision techniques,
and tries to understand the image content. Suchattempts have produced good but limited success,mainly because image interpretation is a highly
complicated perceptive process.
We propose a new approach to this problem
from a totally different angle. It attempts to
exploit the human’s perceptual capabilities andcertain common, if not identical, tendencies thatmust exist among people’s interpretation andpreference of images. Instead of processing
images, the system simply accumulates records ofuser feedback and recycles them in the form ofcollaborative filtering, just like a purchaserecommendation system such as Amazo.com. To
emphasize the point that it does not deal withimage pixel information, we dub the approach bya term “content-free” image retrieval (CFIR).
We discuss various issues of image retrieval,argue for the idea of CFIR, and present results ofpreliminary experiment.
The results indicatethat the performance of CFIR improves with thenumber of accumulated feedbacks, outperforminga basic but typical conventional CBIR system.
Keywords
Image retrieval, content-free, collaborative
filtering, user preference, relevance feedback,image similarity, content-based.

1. Introduction
A picture is said to be worth a thousand words.
If this statement is true, it is no wonder thatcomputerized image retrieval is a challenging task.
A key to a capable image retrieval system is how
to extract and describe the image contents.
Contents may be described either verbally or nonverbally. Verbal descriptions, such as keywords,
are suitable for human perceptions. And if
obtained, keyword-based retrieval is relatively
straightforward [3]. Attaching keywords toimages is, however, hard; manual labeling is too
expensive and automatic methods, for the moment,
are not reliable.
Non-verbal descriptions used so far arecomputer-centric. Various image features areproposed, including color, shape, and texture. The
current content-based image retrieval (CBIR)
approach uses these image features to define themodel of similarity or visual resemblance
between images [14]. This relatively simple
scheme has shown good success in various image
database applications [7][11]. Some limited object
recognition, such as faces, has also been used
[8][12]. Further, the relevance feedback technique
has produced better results by incorporating user
feedback for tuning the underlying model with
context-dependent variations [4][13].
Capabilities of image retrieval systems basedon the above current “content-based” approaches
are still severely limited. Although image features
capture image characteristics, they are not always
directly related to the meaning of images or image
interpretation. There is a difference between what
image features can distinguish and what people
perceive from the image. This difference, or the“semantic gap,” is the core of the limitation.
A technique called relevance feedback, where
the user provides his preference and the system
adjust the model, brought users into the decision
making, but the advantage of having humans in
the loop has not been fully exploited because thefeedback was restricted to the way that predefined
image features are used. The semantic gap has
persisted.
We propose a new approach to image retrievalthat uses user feedbacks in the form ofinterpretation rather than through image features,thus directly exploiting human perceptive power.
We adopt collaborative filtering techniques toaccumulate feedbacks of all users and use them tohelp future users. By bypassing image features,the performance improvement will not be
restricted by the predefined capabilities of feature
selection or object recognition. Our proposedsystem may be similar to a purchaserecommendation system, like the Amazon.com,
that recommends books when a user purchases a
few books based on purchase history of others
who have bought the same books. Observing
images that a user selects in the early part of thesession, it retrieves “related” images based on theaccumulated usage history of him/herself andothers,
We will name our approach “content-free”
image retrieval (CFIR) in order to illustrate thepoint that it does not analyze image pixels.
Naturally,thetraditional“content-based”
approach must be combined in the final system,
but we will explore and emphasize the “contentfree” aspect throughout this paper.

2. Image retrieval : current approaches
Image retrieval has been an active researcharea for the last decade [14]. In this section, we
review the strength and weakness of current
approaches to identify where we should focus.
2.1 Keyword-based approach
One obvious approach of image retrieval is todescribe contents of images in a database verbally,
typically by keywords, and apply text searchtechniques. The difficulty with this approach lies
in how to get such text data. As manual labeling is
too costly, alternative sources are necessary.
Images on web pages tend to have associated text
surrounding them. Also, image file names orpath names may provide good descriptions ofimage contents [15]. Recently Ahn et al [1] hasproposed an interesting approach to combine
manual image labeling and network games, inwhich game participants are led to willingly do
the labeling task.
Automatic classification of images orrecognition of objects in them is ideal. Some
success is reported using relatively simple image
features, such as color and texture distributions,and certain image classes, such as indoor, outdoor,
sunset, and landscape, are identified [16][17].
Lipson et al enhanced this scheme by introducing
scene configuration [9]. Only a few objects, suchas faces and cars, can be recognized reliably fromgeneral images [8][12]. Several methods havebeen proposed to automatically learn therelationship between image regions of specificcolor or pattern, and keywords [2][19]. We expect
constant progress in these areas, but considering
the complexity of the problem and the number ofobjects that we have to deal with, it will be sometime before the performance of automatic image
understanding becomes comparable to that ofhuman beings.
2.2 Content-based approach
To avoid the difficulty of obtaining realcontents, computer-centric image representations
have been used based on image features, such ascolor, shape, and texture [7][11][18] (see thehistorical summary in [14] and [20]). Images arecharacterized mostly by statistical properties,
such as a histogram, of those features. Similarity
measure between images is defined and used toretrieve target images. This approach is
historically called “content-based”, even though
the name is a somewhat inflated one.
Finding a good set of features is very criticalsince the rest is built upon it. Various features andassociated similarity measures have been
proposed to imitate human visual perception.
These attempts achieved only limited success so
far because human perception of images iscomplex and seems to be dependent on context,
purpose, and individual cases.
2.3 Relevance feedback
Content-based image retrieval can be enhanced
by incorporating user feedbacks into the system.
Typically, as the system shows the retrieved
images to the user, he/she tells the system which
images in the output are more relevant or lessrelevant to his/her query. Given feedbacks from auser, the system determines which image features
are to be used to duplicate the user’s decision andmake changes to the parameters or weights in theunderlying model of image similarity. The
feedback procedures are repeated as necessary.
Many researchers have reported that improved
results are obtained [4][13].

3. User-Powered content-free image retrieval
Relevance feedback methods have proven thathumans can play an important role in the successof image retrieval; even simple user feedbacks
help improve the performance of content-based
image retrieval methods. The fundamental reason
for this is that people know what they are doing.
Humans can immediately judge whether
presented images are relevant to what he/she islooking for, although people usually may not beable to provide its complete descriptions inadvance.
3.1 From “content-based” to “content-free”
We observe two different types of limitation in
the way that the current content-based methods
use user feedback. Firstly, because ourunderstanding of human vision is limited, we
probably do not have a correct set of image
features to begin with. Therefore, perception
models based on those features will not satisfy all
the requirements demanded by the user feedbacks.
Secondly, selecting several images several times
at each session will not provide enough data to
train a complex vision model. To properly adjust
the underlying model with sufficient complexity
requires that a large number of image samples beprovided by the user.
It should be noted that it is the user who
analyzes image contents, and that the feedback is
the result of that analysis. Image features andsimilarity measures are nothing but the
representation tools that aggregate his analyses
into a decision making process that the “contentbased” image retrieval happens to use.
A natural solution to overcome these
difficulties is to bypass the image features and use
the human’s perceptual decisions themselves (i.e.,
which images are similar to which) as therepresentations which need to be aggregated andfrom which the system learns the contents of theimages.
We believe that an effective image retrieval
system can be realized using only the usage
history of users. Imagine a user who is engaged
with a system for image retrieval sessions. In eachsession, a user is telling the system which images
are relevant or similar and which images are not.
Note, however, the user is not telling why or inwhat sense; he/she is simply telling the decision
Now we record all of these feedbacks from allof the users. The accumulated feedbacks should
work as asynchronous voting on relationships
among images in the database. Once enoughfeedbacks are accumulated, the system can learn
and summarize those relationships in a certain
form. Subsequently the system retrieves relevant
images for a new query from a new user using thelearned relationships, and the result is expected toagree with the majority’s perception. Unlike thecontent-based approach, this scheme lets all
image processing and perception tasks be done bya population of users, and uses the learned

relationships from them to do the retrieval task.
Hence the name: “content-free” approach.
3.2 Collaborative filtering
The tool to accumulate user feedbacks andretrieve images for a new query is collaborative
filtering. Collaborative filtering is a technique topredict preferences of one person frompreferences of others [10]. Amazon.com BookStore [22] is one of the best known examples. It
basically works as follows. When a customer
purchases a set of books, the system looks uppurchase histories of other customers who havepurchased the same set of books, identifies themost popular books among those customers, andrecommends the identified books. Naturally, it islikely there are only few or even no previous
users who purchased the exact same set.
Collaborative filtering techniques allow forreasoning related books from many samples.
An image retrieval system with collaborative
filtering would work similarly. When a user forms
a query by selecting a set of images from thedatabase, the system uses usage histories ofprevious users (including his/her own interaction
histories up to then), identifies related images thatwould be most frequently selected with the query
images, and displays the identified images.
3.3 Premises
A few assumptions have to be satisfied for theabove idea to work well with image retrieval
problems. Firstly, one user’s relevance judgment
under the same context will remain relatively
stable over time. Secondly, given the same pair ofimages and the same context, the relevancejudgments of different people are similar. Thirdly,
for the finite set of images, the number ofinterpretations derived from the set does not growtoo fast.
The first assumption says that a user’s
judgment is time-shift invariant. Note that thisinvariance does not have to hold strictly, but itshould hold to the extent that recorded feedbacks
of a user will be helpful for him/her in the future.
With the second assumption, feedbacks fromdifferent users are transferable, and can be treatedas an ensemble. This assumption is the basis forour collaborative solution. This does not mean,
however, that the interpretation or preference of aparticular image must be the same over people.
Rather it requires that their distribution is similar.
Although there are differences in personal
preferences, we assume that people sharecommon perceptive attributes.
While the first two are related to reusability of
feedbacks, the third assumption concerns thesufficiency of feedbacks collected from users tolearn the relationships among images. If there are
too many combinations of interpretations orgroupings for a finite number of images, it will
not be possible to collect a sufficient number ofuser feedbacks, since each feedback would
correspond to only one of such groupings.
We hypothesize that all the above assumptions
are valid. Since it is difficult to derive their proofs
theoretically, we conduct a series of experimentsthat suggest their validity.

logical functions of { ｘ 1 ,…,ｘ n }.
P ( x i = 1 | XE ) ≈ ∑ λij f j ( XE )(1)jλ ij are Lagrange coefficients and they satisfy
the following conditions.
€λi⋅ = p iT P −1

(2)

 P( xi = 1 | f 0 ( X E )
pi = 
M P( xi = 1 | f c ( X E ) 

(3)

T

4. Proof of concept
In order to test our idea, we build a simple
content-free image-retrieval (CFIR) system basedon collaborative filtering. We collected a data ofuser judgments on an image set. We also defined aperformance measure of image retrieval. Using
the data and measure, we compared the system’s
performance with respect to the varying numberof user feedback, as well as with that €oftraditional content-based image retrieval (CBIR)system.

P ( f 0 | f 0 ) P ( f 0 | f1 ) L P ( f 0 | f c )


P =
M
M
O
M

P ( f c | f 0 ) P ( f c | f1 ) L P ( f c | f c )
(4)
P(fi |fj ) denotes P(fi (XE)=1|fj (XE)=1) for all XE.
We set f0 (XE)=0 and fi (XE)=(xi =1|XE) (i=1,...,n).
The pair-wise conditional occurrence probability
matrix P is estimated from the data.
A solution for Λ=[ λ ij ] is,

4.1 Rényi’s entropy-based collaborative
filtering algorithm
In this experiment we used a collaborative
filtering algorithm developed by Zitnick [21],which was derived by maximizing Rényi’s
entropy. Other representative algorithms are
Bayes Net [6].
Suppose there are n images in the database,
X={I 1 ,…,I n }. The variable xi is a logical variableassociated with I i . We denote xi =1 when i-th
image I i is selected and xi =0 when I i is notselected. The image retrieval problem is to predictthe probability of xi =1 given an observed
condition, such as XE ={x1 =1, x2 =0}, which means
I 1 is selected and I 2 is not selected by a user so far.
We call such a condition set XE an evidence set.
Thus an image retrieval problem is computing
P(xi =1|XE) for all xi that are not included in XE. In
subsequent discussion, a notation for XE isomitted, when it is obvious, to avoid clutter.
Since the possible combinations for XE arehuge, there will not be enough data to estimate for
all P(xi =1|XE). Yet, Zitnick showed thatmaximizing Rényi’s entropy results in a good
estimation of P(xi =1|XE) as a weighted sum offunctions F={f0 ,...,fc}. Each of fi is a certain

Λ = P•,E PE+(5)P•,E denotes a matrix whose i-th column isequal to i-th column of P if xi ∈XE or all zerocolumn vector if xi ∉XE. PE is a matrix whose€ at i-th row and j-th column is equal to theelement
element of P at i-th row and j-th column if
xi ,xj ∈XE, or zero otherwise. PE+ denotes a
generalized inverse matrix of PE.
4.2 User feedback data collection
To evaluate an image retrieval system, we needground-truth user data, i.e., a collection of
judgments by people on whether certain images
are relevant to each other within a set of images.
Ideally, the data should be obtained from actual
usage history of a relevance-feedback system.
Here, however, we prepared a special datacollection program. Figure 1 shows the interface
used for data collection of user feedback.
A set of 10,000 images were prepared drawn
from the Corel image library as the underlying
image database. The set consisted of 50 images
from each of 200 vendor-defined categories, sothat the contents are broad and their distribution isbalanced. Fifteen (15) human subjects (mostlysessioncountertarget image

magnifying
view

Images randomly
selected fromdatabase

Figure 1: The interface for data collection of user feedback.
students) were recruited to perform the datacollection sessions. In each task session, a small
subset (roughly 50) of images were randomly
chosen from the underlying database, and were
presented to a subject sitting at a computer
monitor screen, with one image highlighted as atarget image (see Figure 1). The location at which
the target image is shown is randomized. The
subject was asked to group images that are“similar” to the target image and to each other.
The similarity criterion or the number of similar
images to be selected was not specified.
For each performed task, a record R is created,consisting of the displayed image set D, displayedorder O, the target image I, and the user-selected
image set S.
Five subjects performed total of 4010 task
sessions. Each of the other 10 subjects performed
100 tasks, 1000 tasks in total. Note that since theselection of ~50 images to be used as D from10,000 image set, the order O they are displayed,
and the selection of target image I are all
randomized, there is no exact same task among
these 5010 task sessions. Also note that 5010 is an
infinitesimally small fraction of 10000 C50 , all thepossible selections of D.
4.3 Evaluation procedure and performancemeasureWe defined a procedure and criterion toevaluate an image retrieval method using the user
data collected in the previous section. For eachentry of task data R={D, O, I, S}, k images fromthe selected image set S are given to the system asa query set Q. If there are not enough images in S,|S| ≤ k, then the session data is not used.

The image retrieval system ranks the images inD excluding the query images (i.e., images in DQ). The accuracy of the ranking for the task R is
defined as [21].
D −k∑δ (i,S)h(i)accuracy ( R) = i=1 S −k(6)

∑ h(i)i=1−

i−1

h (i) = 2 b−1(7)€ where δ(i, S)=1 if i-th ranked image is in S,
otherwise 0. |D| and |S| denote the number ofimages in D and S respectively. The value b is€called “half-life” for h(i), that is, h(b)=0.5. Here,
we used b=2. The accuracy(R) will be 1 when all
images in S-Q are ranked on top.
The assumptions behind this measure are thefollowing. When using an image retrieval system,
if a user submits one of images in S as a queryand receives a subset of D including some images
from S, the user will most likely select the images
from S as relevant. Also, if the user receives only
images from S in response to the query, the user
will be most satisfied.
Finally, all accuracy(R) are averaged over theentire test data to compute accuracy for the data
set.

accuracy = ∑ accuracy ( Ri )(8)i

4.4 Result
The described method was applied to evaluate€

Figure 2: Image retrieval performance with respect to the number of training data and thenumber of sample images.
our collaborative-filtering based content-free
retrieval system as well as a typical content-based
system that uses color coherent vector which is acombination of two color histograms [4].
For the collaborative-filtering based system,
the 4010 task records from the first five users
were used as training data, and the latter 1000
records were used as test data. The content-based
system was tested with the same 1000 records.
For k, we set the values at 1,2,5, and 10. Note that
not all of the above 1000 test data contained
enough number of selected images for theevaluation. Table 1 lists the number of usable test
data records for each value of k.
k

1

2

5

10

# of valid data

1000

784

462

265

Table 1: # of usable test data records foreach k.
The results were summarized in Figure 2. It
plots the performance measure values withrespect to the number of training data (100, 500,
1000, 1500, 2000, 3000, and 4010) for different
numbers of sample images given as a query (k=1,2, 5, and 10). The thin horizontal solid line at
0.241 corresponds to the accuracy if the system
returns random ranking. The four broken lines
between 0.3 and 0.4 are the performance of thecontent-based image retrieval given different
number of examples k=1, 2, 5, and 10,

respectively.
The results clearly show that the performanceof the content-free retrieval system improves as
the number of feedback data increases. This
indicates that image relatedness judgments madeby one group of users helps another group ofusers and suggests that their decisions more orless agree with each other. In other words, there isgood inter-subject transferability of interpretation.
It is note worthy that our collaborative filtering
method trained with more than 1,000 feedback
data already outperforms the content-based
retrieval method for the task of image retrieval
from 10,000 images. The result indicates thecollaborative filter trained with 2000 feedback
data and a single sample image performs as wellas the tested content-based retrieval method with
10 example images1. Recall that the training data
and the test data are from different users, and thatthe test data do not include any same task as in
the training data. So, the content-free retrieval
system worked well, not because it was given thesame problems as those in training.

We provided all the sample images to the systems at
the same time. The result may have been different if
the images were given incrementally to the contentbased method.
1

techniques in combination. Similarities between
images computed by content-based methods canbe used to initialize the collaborative filtering.
Also, a somewhat sneaky method to deal withnew images is to insert them randomly in retrieval
results as part of operation whenever the system
retrieves images, and see how a user reacts to it.

Figure 3: Retrieval performance using selfproduced feedbacks.
.
It is expected that intra-subject
transferability
is higher than inter-subject transferability; that is,if the training set and test set are drawn from thefeedback data of the same subject, the training
will be faster and better. Figure 3 shows the resultof the intra-subject cases. Like in Figure 2,
different numbers of session records of a single
subject were used as training data, and 100
records of the same subject (but not included in
the training set) were used as test data. Figure 3
shows that at about a few to several hundred
training data the accuracy values are alreadycomparable to those for inter-subject cases at a
few thousand training data. Consistency ofinterpretation is much higher within the same
person as expected.
Some concerns remain with the result.
Accuracy curves appear to flat out as the numberof training data increases. We do not know thereason for this phenomenon yet.

5. Discussions
The experimental results appear promising, but
still are very preliminary. In this section we
discuss a few critical issues that need to beinvestigated further.
5.1 Cold start problem
Collaborative filtering is a “cold start” solution.
The system needs be used for a certain amount oftime before it accumulates enough data forlearning and becomes capable. The more it is used,
the more capable it becomes, but users may notwant to use it unless it is capable. Another related
problem is how to handle new images that havebeen added to the database.
One way to alleviate these difficulties is to use
current text or content-based image retrieval

Indeed, though we have emphasized the“content-free” aspect, we envision the final
system to be a hybrid system. The collaborative
filtering network is supported by other techniquesthat utilize any information associated withimages, including content-based module and textbased module. A content-based module with
relevance feedbacks is used initially and switches
to a content-free method when enough user
feedbacks are collected. For a web-wide image
retrieval problem, statistical and thesaurus
analysis of the textual information, such as
caption or file name, which is often associatedwith images should be combined withcollaborative filtering.
5.2 Number of feedbacks
It is interesting to know how many feedbacks
are required to make our method work as
intended.
In our algorithm, we need to estimate a pairwise conditional occurrence probability matrix P
in Equation (4). That is, probabilities for N(N1)/2≈N2 /2 pairs of images have to be computed
where N is the total number of images in the
database. It should be noted that most of theprobabilities are zeros since each image is likely
to be related to very limited portion of thedatabase. Suppose each image is related to αN
images, where 0≤ α≤1, the number ofprobabilities that has to be estimated is α2 N2 /2.
Let ND denote the number of images a user
sees on the screen during one session whenproviding a feedback. Let also NS denote thenumber of images that the user selects as relevantfrom the ND images shown. We consider the restof NNS =ND-NS images are implicitly labeled as
non-relevant. From this one feedback session,
NS (NS -1)/2 + NS NNS ≈ NS ND - NS 2 /2 pair-wisesamples of image relations are obtained. Sinceinterpretation and preference may differ between
people, let us assume that among NS image
labeled as relevant, only β NS images (0≤ β ≤1) will
have agreement with other people’s interpretation.
The final number of valid or usable relations are
β (NS ND- β NS 2 /2). We assume β to be fairly large
on the basis of our experiments that people’s
perception is more or less comparable. Usingthese notations, we can now discuss the numberof feedbacks required to make our system work.
Suppose we need at least s feedbacks forreliable estimation of each pair-wise conditional
probability. The total number of feedbacks F thatare required is:

Fm < F < FM αN  2sFm =(βN S N D − N S2 /2)  β (9) N 2sFM = (βN S N D − N S2 /2)  β 
FM corresponds to the case when images shownfor the feedback session are completely randomly
chosen, and Fm to the case when they are chosen€
most wisely (i.e., only those that need to berelated are chosen).
If we set N=10000, NS ≈5, ND≈50, α=0.01,
β =0.8 and s=100 (roughly corresponding to ourexperiment), we have FM = 8.3×10 7 and
Fm=8.3×10 3.
A popular Internet search engine Google
claims that it has indexed more than 425,000,000
images [23]. For this case, we have Fm =3.8×10 10 ,assuming each image is related with 2×10 5 images(α=0.05%). Considering that the Google also
answers 10 8 search queries per day, 3.8×10 10
feedbacks can be collected roughly within a year.
The question of how to find the right pairs still
remains. However, the problem is not unique toour method; it is the task for all image retrieval
methods to find a small number of related images
from a vast number of images. One of theadvantages of our scheme is that once a set ofimages is identified as related by any mean, theknowledge is stored and reused. Because of thisproperty, we expect the performance of our
method to increase fairly monotonically as
feedbacks accumulate.
5.3 Standard Data
While designing procedures and measures forevaluating image retrieval systems, we came torealize the strong need for their standardization,
especially, the need for a standard corpus ofimages as we have seen in other research areas
such as speech recognition and face recognition.
Once we have a standard image set and associateduser data set, researchers can refer to and use thesame data to compare the performance directly.

We are planning to continue collecting user
data and make them publicly available for the
research community. Our current image data set isa commercial library that may not be distributed
freely along with the user data. Although nontrivial, we are contemplating to build a free (orminimum-cost) open-source large-scale nonbiased image dataset, following activities in otherareas such as Open Video Library [5].

6. Summary and Conclusions
This paper has argued that having users in theloop is the key to capable image retrieval sincecurrent
automated
image
understanding
techniques have very limited capabilities.
We proposed a new user-powered “contentfree” approach to image retrieval that directly
utilizes and recycles feedbacks from users bymeans of collaborative filtering without doing
image analysis. Recycling feedbacks not only
reduces the burden of the users in providing thesame information repeatedly, but also allows foraccumulating the results of human’s perceptual
decisions on images.
The results of our preliminary experimentshows that the performance of “content-free”
image retrieval system improves with the numberof accumulated feedbacks, even outperforming abasic but typical traditional “content-based”
system. Although many issues remain to be
explored, our ultimate goal is to collect allcomputational powers from resources spread over
networks both in time and space to accomplish alarge-scale image retrieval task. The resources are
human users.

Acknowledgments
We would like to thank Larry Zitnick for
inspiring discussions. Many thanks go to TatSeng Chua and Zhao Yunlong of the National
University of Singapore for providing their image
retrieval program.

References[1] Ahn, L. and Dabbish, L., “Labeling Images
with a Computer Game,” to appear inProceedings of ACM CHI 2004.
[2] Barnard, K. and Forsyth, D., “Learning the
Semantics of Words and Pictures,” inProceedings of International Conference on
Computer Vision, Vol 2, pp. 408-415, 2001.
[3] Chang, N.-S. and Fu, K.-S., “Query by
pictorial example,” in IEEE Transactions on

Software Engineering, Vol. 6, No. 6, pp. 519524, 1980.
[4] Chua, T.-S. and Chu, C.-X., “Color-based
pseudo object model for image retrieval with
relevance feedback,” in Proc. of FirstInternational Conf. on Advanced Multimedia
Content Processing, pp. 145-160, 1998.
[5] Geisler, G. and Marchionini, G., “The OpenVideo Project: A Research-Oriented DigitalVideo Repository,” in Proc. of ACM Digital
Libraries, pp. 258-259, 2000.
[6] Heckerman, H., Geiger, D., Chickering, D.
M., “Learning Bayesian Networks: The
Combination of Knowledge and Statistical
Data,” Machine Learning, Vol. 20, No. 3, pp.
197-243, Kluwer Academic Publishers, 1995.
[7] Horowitz, B., Humphrey, R., Jain, R., Shu.
CF., “Virage image search engine: An open
framework for image management,” in Proc.
of SPIE Conf. on Storage and Retrieval for
Image and Video Databases, pp. 76-87, 1996.
[8] Kumar, S., Hebert, M., “Man-made structure
detection in natural images using a causal
multiscale random field,” in Proc. of IEEE
Conference on Computer Vision and Pattern
Recognition, Vol. 1, pp. 119-126, 2003.
[9] Lipson, P., Grimson, E., Sinha, P.,“Configuration Based Scene Classification
and Image Indexing,” in Proc. of IEEE Conf.
on Computer Vision and Pattern Recognition,
pp. 1007-1013, 1997.
[10]McNee, S. M., Lam, S. K., Konstan, J. A.,
Riedl., J., “Interfaces for Eliciting New User
Preferences in Recommender Systems,” inProc. of The 9th International Conf. on User
Modeling (UM'2003), pp. 178-188, 2003.
[11]Niblack, W., Barber, R., Equitz, W., Flickner,
M., Glasman, E., Pektovic, D., Yanker, P.,
Faloutsos, C., Taubin, G., “The QBIC project:querying images by content using color,
texture, and shape,” in Proceedings of SPIE
Conference on Storage and Retrieval for
Image and Video Databases V, Vol. 1908, pp.
173-187, 1993.
[12]Rowley, H. A., Baluja, S., Kanade, T.,“Neural Network-Based Face Detection,” inIEEE Transactions on Pattern Analysis andMachine Intelligence, 1996.
[13]Rui, Y., Huang, T.S., Ortega, M., Mehrotra, S.,“Relevance feedback: a power tool for interactive contentbased image retrieval,” in IEEE Trans. on Circuits
and Systems for Video Technology, Vol. 8, No.
5, pp. 644-655, 1998.
[14]Smeulders, A. W. M., Woming, S., Santini, S.,
Gupta, A., Jain, R., “Content-Based Image
Retrieval at the End of the Early Years,” inIEEE Trans. on Pattern Analysis andMachine Intelligence, Vol. 22, No. 12, pp.
1349-1380, 2000.
[15]Smith, J. R. and S. F. Chang, S.-F., “AnImage and Video Search Engine for the World
Wide Web,” in Proc. of SPIE Conference on
Storage and Retrieval for Image and Video
Databases, Vol. 3022, pp. 84-95, 1997.
[16]Szummer, M. and Picard, R. W., “IndoorOutdoor
Image
Classification,”inProceedings of IEEE Workshop on Contentbased Access of Image and Video Databases,[17]Vailaya, A., Jain, A., Zhang, H. J., “On image
classification: city images vs. landscapes,” inPattern Recognition, Vol. 31, No. 12, pp.
1921-1935, 1998.
[18]Wang, J. Z., Li, J., Wiederhold, G.,“SIMPLIcity: Semantics-Sensitive Integrated
Matching for Picture Libraries,” in IEEE
Transactions on Pattern Analysis andMachine Intelligence, Vol. 23, No. 9, pp. 947963, 2001.
[19]Wang, J. Z. and Li, J., “Learning-Based
Linguistic Indexing of Pictures with 2-D
MHMMs,” in Proc. of ACM Multimedia, pp.
436-445, 2002.
[20]Zhou, X. S., Rui, Y., Huang, T. S.,
Exploration of Visual Data, Kluwer
Academic Publishers, 2003.
[21]Zitnick,
C.,“Computing
Conditional
Probabilities
in
Large
Domainsby
Maximizing Renyi's Quadratic Entropy,”
doctoral dissertation, tech. report CMU-RITR-03-20, Robotics Institute, Carnegie
Mellon University, May, 2003.
[22]http://www.amazon.com[23]http://www.google.com