Vis Comput
DOI 10.1007/s00371-016-1232-1

ORIGINAL ARTICLE

Retrieval of clothing images based on relevance feedback
with focus on collar designs
Honglin Li1 · Masahiro Toyoura1 · Kazumi Shimizu1 · Wei Yang1 ·Xiaoyang Mao1

© Springer-Verlag Berlin Heidelberg 2016

Abstract The content-based image retrieval methods aredeveloped to help people find what they desire based on preferred images instead of linguistic information. This paper
focuses on capturing the image features representing detailsof the collar designs, which is important for people to choose
clothing. The quality of the feature extraction methods isimportant for the queries. This paper presents several new
methods for the collar-design feature extraction. A prototype of clothing image retrieval system based on relevance
feedback approach and optimum-path forest algorithm is also
developed to improve the query results and allows users tofind clothing image of more preferred design. A series ofexperiments are conducted to test the qualities of the feature extraction methods and validate the effectiveness andefficiency of the RF-OPF prototype from multiple aspects.
The evaluation scores of initial query results are used to test
the qualities of the feature extraction methods. The averagescores of all RF steps, the average numbers of RF iterations
taken before achieving desired results and the score transition of RF iterations are used to validate the effectiveness andefficiency of the proposed RF-OPF prototype.
Keywords Content-based clothing image retrieval · Collar
design · Feature extraction · Saliency map · SIFT · Relevance
feedback · Optimum-path forest

1 Introduction
As e-commerce continues to gain momentum in the market, more and more consumers are searching online shops

B Xiaoyang Mao

mao@yamanashi.ac.jp

1

University of Yamanashi, Yamanashi, Japanfor clothing items. However, sifting through the massive
amounts of available products to find an item that suits one’s
tastes and preferences can be an arduous, time-consumingtask. Many sites support keyword-based searches, but items
in online shops often lack specific design-related tags andinclude technical names that few shoppers are familiar with.
One image search approach that researchers have proposed as
visual query-based alternatives to keyword-driven searches isthe content-based image retrieval (CBIR) method [1], which
involves expressing image content in feature vectors and thencomparing the similarities between various images. Although
CBIR methods eliminate the need to build queries out of
keywords or other linguistic information and allows users tosearch for visual information with visual information input,
their efficiencies largely depend on the quality of the feature vectors, and it remains to be challenged to extract theimage features capturing the design of clothing well. Some
researchers [2] have attempted to use color and texture, but
few researchers have delved into the possibilities of developing feature vectors to capture the designs of clothing indetail.
The main contribution of this paper is twofold. One is a
novel method for extracting the detailed design features ofcollars from 2D clothing images. The other is a prototype ofclothing image retrieval system based on Relevance Feedback approach and Optimum Forest algorithm for allowing
users to find clothing image of preferred design. We focus
on collars because it is a well-known fact among garment
designers that collar is a crucial part of a garment as it serves
as the frame for one’s face [3–5]. Since face is the most important visual attribute for characterizing a person, the design ofa collar can largely affect the look of a garment and the overall
impression of a person. In addition, the artistic formation of acollar is usually the most eye-catching part when people lookfor clothing due to its horizontal perspective position clos-123

H. Li et al.

est to the observers’ eyes [6]. Although computer-assisted 3D
garment design technologies [7,8] and virtual fitting systems[9,10] have become available recently, E-commerce websites
for clothing shopping mainly show 2D clothing images forusers to choose. Our proposed method allows users to retrieve
the clothing of preferred design from such online shoppingsites first, which will narrow down the candidates and speed
up the procedure of virtual fitting [10].
In our study, we consider collar spread, turnover and thefront design as three important design factors of a collar
and develop new methods for automatically extracting thosefeatures from clothing images. For the turnover and front
design, three different feature vectors based on SIFT featureand Saliency Map are designed. We further apply optimumpath forest algorithm to perform image search procedures. By
incorporating the relevance feedback approach into the OPF
process [11], our method enables users to search for images
based on his or her subjective preferences on collar design.
To validate the above proposed ideas and methods, a series ofexperiments are conducted. The experiment results presented
in [12] could not provide sufficient validation to the proposed
methods. First, only four subjects were involved in the experiments and thus the results might not be reliable enough.
Second, neither the effects of the different collar types on theevaluation scores nor the bias caused by the difference in thenumber of images of different collar types was considered.
Third, the improvements of the RF-OPF prototype on thethree feature extraction methods were not discussed. In thiswork, we redesign the experiments and succeed in achieving reliable and detailed results to validate the effectivenessand efficiency of the proposed methods for all referred collar
types.

al. [2] used images with uniform backgrounds as queries for
retrieving a limited scope of clothing items that one mightfind in an online shop. With a piece of clothing serving as
the input for the method, their approach involved comparing
items based on the features—color, texture, SIFT featuresand outline—that the pixels in the clothing regions of thegiven images form. features were also used in our study,
which aimed to extract the designs and other characteristics
of collars. None of these existing methods are capable ofsearching for detailed information that collar designs represent. One recent study proposed the idea of using sketches
to search for clothing items with the desired design [15].
However, the method presents problems for people who lacksufficient sketching skills.
In the CBIR field, meanwhile, relevance feedback (RF)method has been drawing substantial attention for its useof dialogic feedback between the users and the system forlearner-driven learning and searching purposes. Searchingbased on relevance feedback method makes it possible toupdate classifiers by showing results to users. Researchers
have already tested this approach in searching for images
with ambiguous thematic content, such as ocean scenes, cats
and sunsets. One study has proposed a method that produces
high-quality results via minimal amounts of feedback by
incorporating different types of classifiers and reusing pastclassification results [16]. By employing a prototype with the
RF approach for learning and the OPF algorithm for searching procedures, our method enables users to search for collars
that align with their personal preferences, which may havepositive effects on choosing the whole clothing.

3 Collar design
2 Related research
The ongoing spread of e-commerce, among other factors, has
prompted numerous researchers to explore the possibilities
of applying search methods to clothing. Liu et al. [13], forexample, proposed a method that makes it possible to use
snapshots to search for clothing available in online marketplaces. The method developed by Liu et al. involves taking a
picture of a person’s body, separating the body into its constituent parts (feet and legs, for example) and determining thefeatures of each part to enable users to search for images ona fashion website. Bossard et al. [14] proposed a method forclassifying apparel in photographs. Using SVM and random
forest allows their method to establish clothing categories
like long skirts and coats and classify clothing according tosleeve length, material and other attributes, but these classification schemes were the ultimate purpose of the research;
Bossard et al. did not include the idea of searching for clothing based on specific design elements. A project by Hsu et123

Generally, image searches operate on the similarity of visual
attributes of multiple images. Finding a collar that suits
one’s tastes, however, would require the matching of design
elements in greater detail. For our study, we begin by interviewing instructors at fashion colleges about the design
elements of collars. Then we take their suggestions into consideration to design feature vectors that can enable fine-tuned
search functionality. Clothing experts have suggested thatcollars generally come in the following ten types (Fig. 1) [4].
Three important elements should be taken into consideration in describing the collar designs. The first is thecollar spreads—the ways they open. The second is Turnover.
Figure 2 shows two examples of the collar design withturnover. The existence and shape of the turnover are someof the most distinctive features affecting the preferences ofmany consumers looking for buying clothing. The third isfront designs, which refers to the types of ribbons and frills
as shown in Fig. 3.

Retrieval of clothing images based on relevance feedback with focus on collar designs

Fig. 3 Collar with front designs
Fig. 1 Collar types

Fig. 2 Turnover collars

Collar design preferences vary considerably according tobuyers’ tastes and needs. A person looking for work clothing, for example, would probably prefer a simple, clean look
to a busy, loud design. On the other hand, a buyer trying tofind something flashier to wear for a fancy occasion might
opt for an ensemble that features frills or a gather. Personal
tastes affect people’s clothing choices in a wide variety ofother ways, for instance, the likelihood of a person with areserved, quiet personality buying a frilly design is rather
slim. In hopes of enabling a search approach that reflects user
preferences, we design feature vectors to describe detailed
collar designs and build a search system using the relevance feedback method based on OPF algorithm. For the
preliminary feature extraction process, our method involves
obtaining feature vectors describing three design elements:
collar spread, turnover and front design-for each clothing
item in the clothing database. Three kinds of feature extraction methods based on SIFT Saliency and Saliency & SIFT,
respectively, are implemented. In the runtime phase, OPF
algorithm is used to classify the images into relevant or irrelevant based on the initial training images. The OPF classifier
is refined through iterative relevance feedback from the users.
The framework of the proposed approaches is shown in Fig. 4.

Fig. 4 The framework of the proposed approaches

123

H. Li et al.
Fig. 5 Computing the feature
vectors of the collar spread

4 Designing and extracting feature vector
In order to adopt the RF approach and use the OPF algorithm
to search the items of preferred collar design, it is necessary
to build a feature vector space that effectively reflects thedesign features of collars. To describe the three importantelements, which are spread, turnover and front designs, thefeature vectors are designed and extracted as below.
4.1 Collar spread
For our study, we limit our scope to images of tops thatshowed the wearer’s upper body only. We also assume thateach image has a monochromatic background showing ahuman subject from the front and the color of which is at least
somewhat different from that of the subject’s skin. Incorporating more advanced image processing technologies would
make it possible to ease these restrictions, but doing so would
deviate from the main focus of our study. Figure 5 illustrates
the process of extracting feature vectors that represent thecollar spread (opening). We use the following steps to extract
features:
Step 1: Dividing the image into background, clothing andskin regions.
This step is the preparation work for identifying the collarline (the clothing-skin boundary) of Step 2 and determining
the outline of the clothing in the image (the clothingbackground boundary) of Step 5. The collar spread features
can be expressed according to the depth value for each angle
emanating from the subject’s neck, as indicated by the resultof Step 7 in Fig. 5.
We separate the background and the foreground of the
input image. To extract the skin color region, we use skincolor S, the dominant color of the area between the neckline
and the chin, as our learning data and extract the pixels with123

the shortest Mahalanobis’ generalized distances from skincolor S and use their color t as the skin color. And thus wecan define the collar line and clothing outline by locating theskin, clothing and background boundaries. We place single
seeds in the skin, clothing and background areas respectively
and then watershed algorithm was used to divide the image
into the three regions based on these seeds.
Step 2: Extracting collar line
With the image divided into three regions, we then extract
pixels from where the skin and clothing regions meet to identify the collar line.
Step 3: Getting the l, r, c points
This step involved extracting the coordinates of the left
end 1, center c and right end r of the collar line, which will
be used in Step 7 to compute the feature vectors representing
the spread of collar. We use chain codes to trace the contourof the entire collar line identified in Step 2, which allow us todetermine the coordinates of left end 1 and right end r. Thenthe coordinates of 1 and r are used to determine the center cof the collar.
Step 4: Obtaining the intersection points
Then we extend radial lines from the center c in the direction of αi = i ×π/36 (i = 0, . . . , 36) until the lines intersectwith the collar line that we extracted in Step 2. The intersections are defined as points pi (i = 0, . . . , 36).
Step 5: Extracting clothing outline
With the image divided into three regions in Step 1,
we extract pixels from where the clothing and backgroundregions meet to generate the clothing outline.
Step 6: Estimating shoulder width S
Using the clothing outline founded in Step 5, we next
calculate shoulder width. Its value serves as the basis fornormalizing the feature vector calculations in Step 7. By

Retrieval of clothing images based on relevance feedback with focus on collar designs
Fig. 6 Comparison of theresult by spread feature vectors
on clothes with V opening,square opening, turn over andfront design collar

normalizing the feature vectors, we ensure that the feature
quantities obtained are independent of image size.
To calculate shoulder width, we use the differential ofbody width. Denoting the body width at height y in the clothing outline as W (y), the shoulder width S is given as W (ys )at the height ys , where the second-order derivative of W (y)exceeds a given threshold.
S = W (ys )W  (ys ) > thw(1)

Step 7: Calculating feature vectors
With the intersection points pi (i = 0, . . . , 36) obtainedin Step 4 and the center c obtained in Step 3, the lengths|c − pi | (i = 0, . . . , 36) are computed and normalized basedon shoulder width obtained in Step 6. Finally, the normalized
values constitute the 36d feature vector of the collar.
Figure 6 shows the searching results using the above
spread feature vector only. We can see the collar spread feature is well captured (the first two rows), but the feature ofturnover and front design cannot be distinguished (the last
two rows).
4.2 Turnover and front design
To capture the features of turnover and front design, we
design three different feature vectors, which are described in
the following as SIFT-Based Feature (SBF), Saliency-BasedFeature (SABF) and Saliency & SIFT-Based Feature (SSBF).

SBF
SIFT is the most commonly used size and orientation invariant feature. Using a gradient histogram around arranged
points makes it is possible to capture local details of image
contents. It can be expected to capture ribbon shapes andother design details. To accelerate the feature matching inimage retrieval, we combined the bag-of-visual-word method[17–19] with the SIFT feature extraction. First we computethe 128d local SIFT feature. Then K-means method is used tocluster the SIFT feature vectors into 500 clusters (500 here isempirically given). The centers of clusters are used as codewords. Finally, each SIFT feature vector is mapped to theclosest codeword to obtain the histogram of the codewords,
which is a 500d feature vector. We compute the above feature
vector for the region from the collar to the chest (Collar-SIFT)
and the whole image (Whole-SIFT).
SABF
Figure 7 illustrates the comparison of searching results
between a piece of plain front design clothing and a striped V
collar one using SIFT feature. While this approach manages
to find relatively good matches (the first row) for the plain
clothing image, it fails to produce the same quality results(the second row) for a striped one. The second row resultsshows that images with similar clothing textures—not collar
designs—appear in the top search results. In other words, theSIFT features can be too dominated by texture features.

123

H. Li et al.

Fig. 7 Comparison of search results between a piece of plain clothing
and a striped one with Sift feature

In order to reduce the effects of the texture features on thecollar designs, we propose a new feature extraction method
using saliency map proposed by Xiaodi et al. [20]. Drawing
on the mechanisms of human visual attention, the saliency
map method allows users to determine the area of an image
that observers are most likely to focus on. A saliency map
calculates the degree of attention not based on the presenceof patterns, but on the differences between a given location of
an image and its surroundings. This method can be expected
to be effective for obtaining features from the front designs
as well as turnover, which are the decorative embellishments
serving for attracting attention. As long as the design is distinct in some way from its surroundings, the saliency map
method can get good matches even if the clothing consists oftextures.
Figure 8 shows the results from two examples consistingof textures. The frill area of the example shown in Fig. 8aexhibits prominent differences from its surroundings on thesaliency map. Even though the clothing consists of large
checked textures, the turnover can still be captured by the
Saliency Map (Fig. 8b).
The appearance of a front design, meanwhile, depends
heavily on the size, length and breadth of their attentiondrawing elements. Various shapes and sizes of ribbons andFig. 8 The results of Saliency
Map acting on two clothing
images. a An example withfront design. b An example withturnoverFig. 9 Feature vector designfor capturing front designs. a
Examples of ribbons. b 3 × 5
grid arranged on saliency map
for capturing the spatial
distribution of front designs

123

frills have an impact on personal preference. Frills that covera considerable area on a piece of clothing, for example, create quite a different visual impression from minimal, dainty
frills on the top of the collar. One can also locate areas withthe highest concentrations of elements that diverge from thegeneral look of a given clothing item. The frills in the imagesof Fig. 9a, for example, spread out across the width of thewearer’s chest to create a relatively showy impression. The
ribbons in the images of Fig. 9b, meanwhile, give the clothing a more extravagant but vertically oriented appearance.
To capture these design-related differences, the proposed
method lays a given grid (3 × 5) over the saliency map asshown in Fig. 9b. We obtain a 15d feature vector, where thepixels values’ sum in each grid cell represents a dimensionof the feature vector.
SSBF
This approach, however, can capture the overall shape offront design only. If the clothing in the input image featuresa small group of frills, the design details of the frills cannot
be captured. We thus propose a method that combines thesaliency map and SIFT feature. Fortunately, Saliency Map
proposed by Xiaodi et al. [20] allows controlling the level

Fig. 10 Saliency map computed with the method given in [20]. a Input
image. b Saliency Map computed using low frequency band. c Saliency
Map computed using high-frequency bandRetrieval of clothing images based on relevance feedback with focus on collar designs

Fig. 11 Extracting turnover features in a high-detailed Saliency Map

Fig. 13 Generating the OPF classifierFig. 12 Results by SBF (top) and SSBF (bottom)

of detail. Using the high-frequency band to compute the difference from the average, we obtain results that retain thefrill details (Fig. 10c). Figure 11 shows the result of using
a Saliency map over a striped shirt with a turnover collar. If
we are to apply normal edge detection to this image, with its
prominent pattern, it will be difficult to determine where theturnover is. Using a Saliency Map with the proper level ofdetail, however, allows us to limit the impact of the pattern toa certain degree. Then by computing the SIFT features fromthe saliency map image, we are able to obtain the detailsof collar design while eliminating the influence of texture
features of the clothing item.
In Fig. 12, the images of the top row are the search results
obtained using SIFT feature only. The clothing items withround neck are also included in the results as those items havethe gather pattern similar to the strip texture of the input clothing. Most of those items are eliminated in the results shownin the bottom row by combining Saliency Map and SIFT feature. In addition, more items with turnover collar or similar
collar spread are included in the search results. However, thequery results are still not good enough. So a prototype combining Relevance Feedback approach with OPF classifiers
is proposed to improve the qualities of the query results asfollows.

5 The RF-OPF prototype
Relevance feedback (RF) method is a critical component in
our CBIR prototype, which makes it possible for users tointeract with the system and thus reflects their design preference in the query. The classifiers, another critical componentof our CBIR system, are used for processing queries. Their
efficiency (related with response time) and effectiveness(related with users’ satisfaction) are very important for evaluating the quality of this CBIR system. In our prototype, we
use the optimum-path Forest (OPF) [11,21,22] classifier for
query and classification. OPF works by modeling the classification as a graph partition in a given feature space. It starts as

a complete graph, whose nodes represent the feature vectorsof all images in the database. All pairs of nodes are linked byarcs which are weighted by the distances between the feature
vectors of the corresponding nodes (referred as costs here andafter). As illustrated by Fig. 13, given a set of training nodes,
a minimum spanning tree (MST) can be generated from thecomplete graph. Then the adjacent training nodes are marked
as prototypes if they belong to different classes, which are
relevant and irrelevant in our case. The partition of the graphis carried out by the competition process among prototypes,
which offer optimum paths to the remaining nodes of thegraph. The optimum paths from the prototypes to the othersamples are computed by the algorithm of the image foresting transform (IFT), which is essentially Dijkstra’s algorithm
modified for multiple sources and more general path-valuefunctions. At last, all the non-prototypes are connected witha prototype directly or indirectly with the minimum costs.
With the prototypes as the roots and the non-prototypes as theintermediate and terminal nodes, the optimum trees are built,
which constitute the optimum-path forest (OPF). Comparedwith SVM, ANN-MLP and K-NN, OPF is usually superior to
ANN-MLP and K-NN in accuracy and significantly outperforms SVM in computation time [11,21,22], which is very
important in a prototype based on RF approach that generates
results in a dialogic fashion.
All the images in the database are represented by the feature vectors extracted with one of the three methods referredin the Sect. 4. The first is Spread + Collar-SIFT + WholeSIFT represented by 1036d (36d + 500d + 500d). The second
is Spread + Saliency represented by 51d (36d + 15d). The
last is Spread + Saliency-SIFT represented by 1036d (36d
+ 500d + 500d).
Based on the OPF classifier described above, we build our
RF-OPF prototype, using the following steps:
1. The initial training set containing images of ten different collar types plus front design type is presented touser. When the users choose one desired collar type fromthe initial training set, the five images with smallest L2
distance to the chosen image in the feature spaces are
returned to the users.

123

H. Li et al.

Fig. 14 The operating mechanism of the RF-OPF prototype

2. The user evaluates the results. If he/she is satisfied with
the initial query result, the RF ends up without using the
OPF classifier. If not satisfied, he/she should mark theimages with × representing irrelevant or O representing
relevant.
3. The first five images marked with × or O constitutethe original training set for building an OPF classifier
mentioned above. The procedure is illustrated as Fig. 14
below and the detail of the OPF algorithm was describedin [21]. Then we use the OPF classifier to sort the unclassified images of the database into two classes, relevant
and irrelevant.
4. The RF-OPF prototype then chooses five images withthe maximum degree of relevance as the query results
and five images with the minimum degree of relevanceas the training samples from the relevant class and returns
them to the users. The new marked training samples will
be merged into the former training samples to build a
new OPF classifier for the next RF phase if the users are
not satisfied. This procedure continues until the users are
satisfied.
The framework and operating mechanism of our RF-OPF
prototype are shown in Figs. 4 and 14, respectively.
When selecting the query results and the training samples,
we compare the costs of paths from all non-training images toall relevant and irrelevant prototypes. The five images which
belong to relevant class with the largest ratio of costs to therelevant prototypes over costs to the irrelevant prototypes arechosen as the query results. The training samples are the fiveimages which belong to relevant class and have the smallest
ratio of costs to the relevant prototypes over costs to theirrelevant prototypes. In our implementation, the cost of thearc connecting two adjacent nodes of the OPF feature spaceis calculated with the L2-norm. And the cost of a path is the
maximum value of the costs of all arcs constituting the path.
Assuming the number of relevant prototypes and irrelevant prototypes to be k and m and denoting the k relevant
prototypes and m irrelevant prototypes as pi (i = 1, 2. . .k)
and q j ( j = 1, 2. . .m), we consider k × m pairs of ( pi , q j )123

Fig. 15 Number of images for the ten collar types and clothing withfront designin computing the ratio of the costs of paths to the relevant
and irrelevant prototypes. Let CRU→ pi and CIU→q j represent the costs of the path from a non-training sample U
to the relevant prototype pi and the irrelevant prototype q j ,respectively. RelevanceU→( pi ,qj ) , which represents the ratioof CRU→ pi over CIU→qj , is computed as:
RelevanceU→( pi ,qj ) = CRU→ pi − CIU→qj (2)We use subtraction instead of ratio to avoid encountering theoverflow problem when CIU→qj is very small.

6 Experiment and discussion
6.1 Experiment
The images used for experiments are gathered from the Internet. All images have monochromatic background and show
upper bodies from the front. Totally there are 274 images
including ten different collar types and the clothing with front
design as shown in Figs. 1 and 3. The number of images foreach of the ten collar types and the clothing with front design
is shown in Fig. 15. Because it should be easier to find a certain collar type if the number of it in the database is high, wetake into consideration the proportion of image numbers ofdifferent collar types in evaluating the efficiency of proposed
methods.
For all images, we pre-computed their Spread, SBF,
SABF, SSBF vectors. The Spread feature vector has 36
dimensions representing the distance from the neck center tothe collar line measured every π/36 as shown in Fig. 5. The
SIFT feature vector is computed for the collar area and thewhole upper clothing area, which consists of 500 dimensions
separately (Fig. 7). When being used alone, the SABF vector

Retrieval of clothing images based on relevance feedback with focus on collar designs

Fig. 16 Comparison of the average scores of the three methods on the collar types

has 15 dimensions (Fig. 9). When combined with SIFT, it
has 500 dimensions.
As described in Sect. 5, our RF procedure starts by letting
the users choose one desired image out of an initial training
set containing images of ten different collar types plus front
design type. This step is used to build the initial OPF classifier
and has big impact on the results of the following RF steps.
In other words, the type of collar the users choose at theinitiation step should be an important parameter in designing
the experiment for two reasons. One is its relationship withthe feature vectors; the effectiveness of the proposed feature
vectors should be different for different collar types. The
other is its number used in the experiment as it should beeasier to find a certain collar type if there are more imagesof such type in the database. The experiments presented in[12], however, ignored these factors completely. To solve theproblems, in our new experiments, each subject is asked totest all the ten collar types plus one front designs for the threedifferent combinations of feature vectors: Spread + SBF,
Spread + SABF and Spread + SSBF. Therefore, each subject
performed 33 tests in total and we compared the effects ofthe three feature vectors for each collar type, respectively. To
eliminate the bias caused by the population of collar types inthe database, the number of images of each collar type is used
to weight the score inversely when compare the average scoreof the three feature vectors for all images. While previous
experiments used only four subjects [12], we improved thereliability of experiment results by expanding the number ofsubjects to 10. The ten subjects are female college students
from school of nursing. For each of the three feature vectors,
they were asked to initiate the RF with each collar type in theinitial training set. At each step of RF, the subjects were asked
to mark each of the five training images as “relevant” (O)and “irrelevant” (×) and evaluate each of the query results
as “satisfactory” (O) and “unsatisfactory” (×). Then a score
ranging 0–5, which corresponds to the number of satisfactory
images, is automatically computed for the query results ofeach step.
6.2 Comparisons of feature vectors (Spread + SBF,
Spread + SABF, Spread + SSBF)
Figure 16 shows the evaluation scores (averaged by ten subjects’scores) of the initial query results for the three feature
vectors. We can observe that SBF and SSBF perform well at
the first six collar types, especially for the Square and the Vtypes. SSBF works well for the Crew type also and SABF
is excellent for the Off shoulder type. We use the weighted(inversely proportional to their number percentage of total)
average scores of the collar types to compare the overall qualities of the three methods, which will reduce the bias causedby the different number of images of different collar types.
As shown in Fig. 17, the weighted score of the SSBF method
is higher than those of the other two. One-tailed paired t-test
reveals that the average score of SSBF in Fig. 17 is significantly higher than that of SBF and SABF at significance level
5 % ( p = 0.05).
6.3 Improvement by RF-OPF
Although the results shown in Fig. 17 demonstrate that SSBF
is more effective than the other two feature vectors, the initialquery results of each feature vector is not good enough. It isbecause that the weighted average score of even the bestcase(SSBF) is just 3.08 against the full score of 5. We expect123

H. Li et al.

Fig. 17 Comparison of the weighted average scores of the three methods on the collar types

Fig. 18 The improvement of the RF-OPF prototype on the methods

that continuing the RF iterations illustrated in Sect. 5 canfurther improve the qualities of the query results.
So if the subjects get few satisfactory images from thefirst query results, they are asked to continue the query andevaluation process until they are satisfied with the numberof the satisfactory images returned by the RF-OPF prototype. The evaluation scores at each RF step are also averagedby the ten subjects and inversely weighted by the numberof collar types as we have done in dealing with the initialquery results. Since the subjects always terminated the RF
when the query results contain four or five desired images(the evaluation scores are correspondingly 4 or 5), the timesof RF iteration they took also reflects the efficiency of our
RF-OPF prototype. To validate whether RF-OPF is useful for
improving the query results, we compare the average evaluation scores of all RF steps with the first query scores. Please
note that not all collar types are used in computing the average evaluation scores of all RF steps. It is because the resultsof the Square and the V collar types with the SBF and the
SSBF methods satisfy the subjects at the initial query phase,and the Crew collar type with the SSBF method satisfies the
subjects at the initial query phase too. On the other hand,
Square collar type with the SABF cannot get any suitable
image at the initial query phase, which generates a scoreof zero that cannot be compared in this experiment. Therefore, as is shown in the left title of Fig. 18, there are ten,eight and nine collar types with the above three feature vector
combinations to be compared. Figure 18 shows that the RFOPF prototype makes great improvement on the initial query
results for all the three feature vectors, which are 107.69,
43.28 and 95.47 %, respectively. But for those collar types
not involved in the RF phases, the effectiveness of the RFOPF prototype cannot be validated in this way. We partiallyaddress this problem with another experiment describedlater.
The average RF iteration times the subjects take to reach
their objects are the evidences to validate the efficiency of
the RF-OPF prototype directly and the corresponding feature vectors indirectly. We compare the average RF times of
the three methods from the ten subjects on each collar type in
Fig. 19. From this figure, some useful information about theefficiency of the RF-OPF prototype on the collar types canbe observed, such as the Square collar type being still very

Fig. 19 Comparison of the RF times of the three methods on the collar types

123

Retrieval of clothing images based on relevance feedback with focus on collar designs

Fig. 20 Comparison of the average RF times of the three methods

Fig. 21 Scores trends of five RF steps on the three feature vectors

difficult to be found with the SABF vector even with the helpof RF-OPF prototype. Figure 20 shows that the SSBF vector
needs the least average RF times (1.30) to make the users satisfied, which demonstrates its efficiency. A one-tailed pairedt test for the RF times in Fig. 20 shows that the difference
between the SBF and SSBF and that between the SBF and
SSBF are statistical significance ( p = 0.05). The difference
between the SBF and SABF is statistical significance at someextent ( p = 0.1).
Since the experiment results of Fig. 18 do not contain theentire collar types, the effectiveness of the RF-OPF prototype
cannot be validated completely. To solve this problem, we
ask all the subjects to perform five times RF steps no matter
what the results are at each step. Since the highest average RF
times of the three feature vectors shown by Fig. 20 is 3.21, 5
times is considered to be enough for this experiment. With theevaluation scores of the initial query and the five RF steps’
results, we get Fig. 21. It shows that the RF-OPF prototype
improves the first query scores of the three feature vectors
from 2.15, 2.82 and 1.53 to 4.54, 4.77 and 4.28, respectively.
The improvements are significant and the scores trends are
incremental in general.
At last we use the SSBF feature and the RF-OPF prototype to query similar collar images for the two given images

Fig. 22 a The top five results for a striped V collar image with the
SSBF method by the second RF phase. b The top five results for a front
design image with the SSBF method by the second RF phasementioned in Fig. 7. The results are shown in Fig. 22, whichshows that the striped V collar type image can get five imagesof similar collar design at the second RF step and the image
with front design can get four images of similar design at
the second RF step too. Note that, although the initial query
results or even the first RF results may be very unsatisfactory, we can always get better results after several RF
iterations.

7 Conclusion
Collar design plays an important role when people choose
the desired clothing. In this paper, we focus our research on
the retrieval of clothing image based on the collar design.
Three different feature vectors SBF, SABF and SSBF combined with Spread for capturing the detailed features of collar
design and a CBIR prototype based on RF method and OPF
classifiers are proposed. Through experiments, it is provedthat SSBF is the best among the three feature vectors in123

H. Li et al.

terms of both effectiveness and efficiency. Our experimentresults also demonstrated that the proposed RF-OPF prototype improves the qualities of the query results significantly.
But there are still some technical problems to be solved.
1. The SSBF feature vector still does not react well to somecollar types, such as the Off shoulder and the Regular collar types. 2. The subjects may be confused by some similar
collar types, such as the Round and the U collar type, theCrew and the Boat collar type during the RF phases in theexperiment. In addition, the collars combing multiple design
features, such as a collar combing turnover with V spread or
front design, are difficult to be classified by the subjects. 3.
The score of the query results is computed as the number ofsatisfactory images without considering the degree of satisfaction of each image in the query results. To solve the firstproblem, we plan to develop and experiment with other new
feature vectors for capturing the collar design. The second
problem can be solved by improving the spread feature vectors for better describing the roundness, width and angle, so
as to better discriminate collar types of similar shape, likethe Round, Crew, Boat and U collar types. The 3rd problem can be solved by asking subjects to score each of theretrieved images and analyze the trend of highest and averagescores of RF iterations. The current image database is relatively small. We need to gather more images for improving
the reliability of the experiment results. Another important
issue is that the initial image sets for letting user to selectone collar type to initiate the RF have large impact on the
results of succeeding steps. We need to explore some new
methods which can always avoid misleading the building of
the classifier.
3D garment design is a rapidly developing field [7,8]. Presenting 3D clothing images in clothing-shopping websites
will help people find the desired clothing more easily. So
it is very important for us to extend our research to deal
with 3D collar design features in the future. Although collar
is very important for choosing clothing, Color, texture andthe design of other parts of garments also affect how peoplechoose clothing at different extent. We are going to extend
our retrieval system by considering other features. It shouldbe easy to capture color and texture features with existing
computer vision technologies. The technology provided in[13] can be employed for capturing the overall shape of garments. Moreover, the proposed RF-OPF system should havehigh potential to deal with more design factors as it is reported
that OPF is superior to conventional learning algorithms suchas ANN-MLP, K-NN and SVM in both computation time andaccuracy, especially in complex situations, i.e., with a large
amount of overlapped regions [21,23].
Acknowledgments We would like to thank Prof Takami Yamamotofor her valuable comments. This work was partially supported by JSPS
KAKENHI (16H05867, 25280037).

123

References
1. Veltkamp, R.C., Tanase, M.: Content-based image retrieval systems: a survey. Technical Report UUCS-2000-34, Utrecht University, The Netherlands (2000)
2. Hsu, E., Paz, C., Shen, S.: Clothing image retrieval for smarter
Shopping. EE368, Department of Electrical and Engineering, Stanford University (2011)
3. Hackler, N.: UK Cooperative Extension Service. University of
Kentucky-College of Agriculture, CT-LMH, pp. 1–9 (1997)
4. Onuma, J.: Coordinate Technique-Appealed, 1st edn. Bunka Publishing, Bunka Fashion College (2001)
5. Pundir, N.: Fashion Techology-Today and Tomorrow. A Mittal Publication, New Delhi (2007)
6. Wang, L., Tian, B.: An analysis of factors determining the shapeof collar. J. Panzhihua Univ. 2, 87–89 (2008)
7. Fang, J.: 3D collar design creation. Int. J. Cloth. Sci. Technol. 15(2),88–106 (2003)
8. Liu, Y., Zhang, D., Yuen, M.: A survey on CAD methods in 3D
garment design. Comput. Ind. 61(6), 576–593 (2010)
9. Zhang, X., Wong, L.Y.: Virtual fitting: real-time garment simulationfor online shopping. In: ACM SIGGRAPH Posters (2014)
10. Hauswiesner, S., Straka, M., Reitmayr, G.: Virtual try-on throughimage-based rendering. IEEE Trans. Vis. Comput. Graph. (TVCG)
19(9), 1552–1565 (2013)
11. Silva, A.T., Falcao, A.X., Magalhaes, L.P.: Active learning paradigms for CBIR systems based on optimum-path forest classification. Pattern Recogn. 44, 2971–2978 (2011)12. Shimizu, K., Yang, W., Toyoura, M., Mao, X.: Relevance feedback based retrieval of cloth image with focus on collar design.
In: 2015 International Conference on Cyberworlds, pp. 137–144.
IEEE, Visby (2015)
13. Liu, S., Song, Z., Liu, G., Xu, C., Lu, H., Yan, S.: Street-to-shop:
cross-scenario clothing retrieval via parts alignment and auxiliary
set. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 3330–3337 (2012)
14. Bossard, L., Dantone, M., Leistner, C., Wengert, C., Quack, T.,
Gool, L.V.: Apparel classification with style. ACCV12 4, 321–335(2012)
15. Kondo, S., Toyoura, M., Mao, X.: Skirt image retrieval basedon sketches. Sketch-based interfaces and modeling, pp. 11–16(2014)
16. Tekawa, M., Hattori, M.: Improvement of reuse of classifiersin CBIR using SVM active learning. Proc. ICONIP 2, 598–605(2010)
17. Fei-Fei, L., Fergus, R., Torralba, A.: Recognizing and learning
object categories. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR, shortcourse (2007)
18. Qiu, G.: Indexing chromatic and achromatic patterns for contentbased color image retrieval. Pattern Recogn. 35(8), 1675–1686(2002)
19. Nister, D., Stewenius, H.: Scalable recognition with a vocabularytree. In Proceedings of IEEE Computer Society Conference on
Computer Vision and Pattern Recognition, CVPR, pp. 2161–2168(2006)
20. Hou, X., Zhang, L.: Saliency detection: a spectral residual
approach. In: IEEE Conference on Computer Vision and Pattern
Recognition, pp. 1–8 (2007)
21. Papa, J.P., Falcao, A.X., Suzuki, C.T.N.: Supervised pattern classification based on optimum-path forest. Int. J. Imaging Syst. Technol.
19(2), 120–131 (2009)22. Silva, A.T., Falcao, A.X., Magalhaes, L.P.: A new CBIR approachbased on relevance feedback and optimum path forest classification. J. WSCG 18(1–3), 73–80 (2010)

Retrieval of clothing images based on relevance feedback with focus on collar designs
23. Papa, J.P., Falcao, A.X.: Optimum-path forest: a novel and powerful framework for supervised graph-based pattern recognition
techniques. Institute of Computing University of Campinas, pp.
41–48 (2010)

Honglin Li received the B.Sc.
degree in Automation from Central South University and M.Sc.
in Computer Science from
Huaqiao University in 2002 and2012 respectively in China. He
began to study as a Ph.D candidate in Computer Science in University of Yamanashi from 2015.
His research interests include
computer vision, pattern recognition and data mining.

Masahiro Toyoura
received
the B.Sc. degree in Engineering, M.Sc. and Ph.D. de-green
in Informatics from Kyoto University in 2003, 2005and 2008
respectively. He is currently an
Assistant Professor at Interdisciplinary Graduate School, University of Yamanashi, Japan. His
research interests are augmented
reality, computer and humanvision. He is a member of ACM
and IEEE Computer Society.

Wei Yang received the B.Sc.
degree in Engineering, M.Sc.
and Ph.D. degrees in Computer Science from University
of Yamanashi in 2010, 2012and2015 respectively. Her re-searchinterests include non-photo realistic rendering and computer aesthetics.

Xiaoyang Mao received her
B.Sc. in Computer Science from
Fudan University, China, M.Sc.
and Ph.D. in Computer Sciencefrom University of Tokyo. She iscurrently a Professor at Interdisciplinary Graduate School, University of Yamanashi, Japan.
Her research interests include
texture synthesis, non-photorealistic rendering and theirapplication to scientific visualization. She is a member of ACM
and IEEE Computer Society.

Kazumi Shimizu received her
B.Sc. degree in engineeringfrom computer media department, University of Yamanashi.
She is currently a master student
at University of Yamanashi. Her
research interests include image
processing and machine learning.

123