Synthesis of Facial Images
Based on Relevance Feedback
Caie Xu

Shota Fushimi

Masahiro Toyoura

University of Yamanashi
Yamanashi Japang17dhl01@yamanashi.ac.jp

University of Yamanashi
Yamanashi Japan

University of Yamanashi
Yamanashi Japanmtoyoura@yamanashi.ac.jp

Jiayi Xu

Honglin Li

Xiaoyang Mao

Hangzhou Dianzi University
Hangzhou China
xujiayi@hdu.edu.cn

University of Yamanashi
Yamanashi Japanlihonglin79@qq.com

University of Yamanashi
Yamanashi Japan
mao@yamanashi.ac.jp

A Abstract—We propose a dialogic system based on arelevance feedback strategy that allows for the semiautomaticsynthesis of a facial image that only exists in a user’s mind. The
user is presented with several facial images and judges whether
each one resembles the face that he or she is imagining. Basedon the feedback from the user, a set of sample facial images areused to train an Optimum-Path Forest classifying the relevanceof facial images. An interpolation method is then employed tosynthesize new facial images that closely resemble the imagined
face. A series of experiments are conducted to evaluate andverify the effectiveness and efficiency of the proposedtechnique.
Keywords—face image synthesis; relevance feedback;
optimum-path Forest

I. INTRODUCTION
Face recognition technique benefits a wide range ofapplications, especially in assisting law enforcement [1]. For
instance, automatic face retrieval from police dataset canhelp the police seek and identify suspects quickly. However,in many situations, the face images of suspects are notavailable. If there were a technique for computer to create
physical images of the faces that people picture in theirminds, it would be possible to develop more accurate
caricatures of criminals.
Our study proposes a semiautomatic method ofsynthesizing facial image in a user’s mind through relevance
feedback (a form of dialogic searching). A set of sample
facial images are used to train an Optimum-Path Forest (OPF)
classifying the relevance of facial images based on users’
feedbacks. Based on the trained OPF, new facial images thatclosely resemble the imagined facial image are created.
The remainder of the paper is organized as follows:
Section II reviews the related works. Section III presents thedetails of the proposed method. Section IV describes thepreparing of the training dataset. Section V demonstrates
some results and describes the evaluation experiment.

II. RELATED WORK
While face recognition has been one of the most active
research field of computer vision, there are few studies havebeen conducted on the synthesis of facial images.
W. Di and D. Qionghai [2] and N. Wang et al. [3][4]proposed methods for synthesizing facial images based on
sketches. The way introduced by W. Di, and D. Qionghai
searches a facial image database for the parts with thehighest degree of resemblance and then patches the results
together to form a facial image [2]. N. Wang et al. presented
Bayesian face sketch synthesis methods [3][4], which uses
the spatial neighboring constraint in both the neighborselection and weight computation. However, both methods
need to draw sketch that not everyone is blessed with thisgift.
Face hallucination is the technique for synthesizing highresolution facial images from low resolution ones. The
method by C. Liu et al. [5] assumes that low resolution
images still manage to reveal general facial characteristics
and uses a high resolution sample database for obtaining theinformation required for the synthesis. Recently, advanced
approach using deep learning technology has been proposed
for hallucinating faces of unconstrained poses [6]. Whilehallucination method requires low resolution images to infer
the information for reconstructing the high resolution images,
our technique uses relevance feedback to acquire theinformation required for synthesizing the face a user images.
III. PROPOSED METHODAs Figure 1 shows, extracting primary feature, training
OPF based on relevance feedback and synthesizing facialimages are three major components of the system.
In our study, we used 1,000 images for building thetraining database. These images are converted to feature
space for training an OPF classifier. The ultimate purpose of
our method is not to classify those sample facial images or toretrieve a particular face from database, but to synthesize anew image resembling a face in the user’s mind. The trained
OPF is used to define the positions in feature spacecorresponding to the desired facial images.

coordinates of the image that best matches the corresponding
face in mind.

To train the OPF, the system defines an initialclassification boundary by letting the user to evaluate an
initial dataset consisting of facial images of different sex andages. Then, the system shows the user multiple unevaluatedimages (have not been judged) that lie near the classification
boundary and has the user label them according to whether
they resemble or do not resemble the face in his or her mind.
Based on these feedbacks, the system updates theclassification boundary.

B. Training Optimum Path Forest based on relevance
feedback
Relevance feedback, a process that shows synthesis
results to the users and updates classifiers based on users’
feedback, is often used in image retrieval. Several
researchers have proposed methods that employ various
classifier types and reuse past classification result to obtain
good results based on relatively minimal amounts offeedback [8].

Then, the system finds a position farthest from theclassification boundary on the positive side, and produces thesynthesized image. If the result satisfies the user, the wholeprocess is complete; otherwise, the user repeats the labelingprocess on unlabeled cases near the classification boundary.

Our study used Optimum Path Forest (OPF) [9,10,11] forclassification. OPF works by modeling the classification as agraph partition in a given feature space. It starts as acomplete graph, whose nodes represent the feature vectors of
all images in the database. All pairs of nodes are linked byarcs which are weighted by the distances between the featurevectors of the corresponding nodes (referred as costs hereand after). Given a set of training nodes, a minimum
spanning tree (MST) can be generated from the completegraph. Then the adjacent training nodes are marked asprototypes if they belong to different classes, which arerelevant and irrelevant in our case. The partition of the graphis carried out by the competition process among prototypes,
which offer optimum paths to the remaining nodes of graph.
The optimum paths from the prototypes to the other samples
are computed by the algorithm of the image forestingtransform (IFT), which is essentially Dijkstra’s algorithm
modified for multiple sources and more general path-valuefunctions. At last, all the non-prototypes are connected witha prototype directly or indirectly with the minimum costs.
With the prototypes as the roots and the non-prototypes as
the intermediate and terminal nodes, the optimum trees arebuilt, which constitute the OPF. OPF is very important in a
prototype based on RF approach that generates results in a
dialogic fashion. It shows good performance for samples
represented in complex and high dimensional feature space.
As depicted by Figure 1, the OPF is trained based on users’
relevance feedbacks in the following 4 steps:

A. Constructing feature space
Various feature representations have been studied in thecontext of face recognition in the past few decades. Recent
research results have demonstrated that deep learning can beused to learn the facial representation which is effective forboth face identification and verification [7]. However, since
our purpose is to synthesize a target facial image, we need afeature representation not only can discriminate faces but
from which a facial image can be generated. For this purpose,
we use the pixel level image feature used in Face
Hallucination method [5].
The first step of the Face Hallucination method is toprepare a database of high resolution facial images. The
method then separates high resolution facial image IH into aglobal face image I Hg , which expresses the overall feature ofthe image, and local face image I Hl .

IH  I Hg  I Hl

(1)Face Hallucination creates a global face image featurevector space by applying Principal Component Analysis(PCA) to the facial images in the database and finding theprincipal components with large eigenvalues. Formula (2)expresses global face image I Hg in terms of global facefeature space basis B, coordinate vector X, and average
facial image 

I Hg  BX  (2)Our study uses the global feature space as the search
space for facial image synthesis. We apply PCA to 1000
sample facial images to define the average facial image
 and basis B of global face feature space. Given acoordinate vector X, a facial image can be generated withequation (2). Our method use trained OPF to locate theStep 1: The system presents the user 5 male facial images
and 5 female facial images of different ages and waits for theuser to select one he/she thinks to be closest to the face in
his/her mind. Since none of those 10 images may resemble
the target face, the user is likely to select the image thatsimilar to what they desire according to sex and age, whichacts as the initial classification boundary.
Step 2: Four images closest to the user selected facial
image in feature space are returned to the user. User starts toevaluate and labels the images with positive (○) or negative(X), which serve as the prototype for the OPF classifier. Thisevaluation phase ends up if the user is satisfied with at least
one of the 4 facial images.
Step 3: An OPF classifier is built based on this set. Then,
we use the OPF classifier to divide the unlabeled images ofthe database into two classes, relevant and irrelevant.

Best nodeX

Constructing
feature space
Database

Synt
hesizeFeature space

SelectResultrelevant prototype
best position image

X

input for initiation

relevant

irrelevant prototype

irrelevnat

resultPresent samples
around borders

Update borderXXD

isplayFace in mind

non-labeled

Display
Relevance
FeedbackLabel

Training images

Figure 1: Overview of the proposed system

Step 4: Four border nodes are selected and thecorresponding images are presented to the user. The user
evaluates and labels the images with positive (○) or negative(X), and the new marked training images will constitute andreplace the former training samples to rebuild new OPF
classifier.
At every iteration before Step4, best positive nodes
located far from the negative prototype and close to thepositive prototype are selected and used to create theresulting facial image. If the user is satisfied with the createdimage, the whole relevance feedback procedure ends.
C. Synthesizing virtual facial images using interpolation
The traditional relevance feedback approach is forsearching actual images in the given database. However, in
our study, synthesizing non-existent facial images is theultimate goal. In principle, any point near the best positive
node should correspond to a desired facial image.
As a practical solution, we propose to select the top k (inthis paper k=3) best positive nodes as shown in Figure 1 andcalculate the result according to the following formula (3)kkiix   w( xi ) xi /  w( xi )IV. DATABASE
For our sample image set, we used 1,000 images of Asianfaces from the CAS-PEAL database [12] and Cartoon Face
database [13]. We made all the images monochrome, and set
the size to 96  128. The database comprised only frontal
facial images, but the positions and sizes of the faces differed.
We aligned, resized, and trimmed the images so that thepoints between the eyes in the facial images would all line up.
Our study is concerned only with general facial features, sowe used a low resolution of 96  128 for all the images.
V. EXPERIMENT
A prototype system for synthesizing faces in mind has
been built based on our proposed method and experiments
were conducted to validate the effectiveness of the proposed
method. We recruit 10 volunteers (7 male and 3 female
university students in their 20s). The subjects are asked toperform two different kinds of task. In the first task, thesubject is given a facial image as a reference and asked togenerate a facial image similar to the reference image. In thesecond task, the subjects are asked to imagine a face andcreate a facial image similar to the one in his/her mind.

(3)Here, x and xi (i=0,1,2) are the coordinate vectors of theresulting facial images and the 3 best positive images,respectively. w( xi) is the weight assigned to xi based on thedistance given by the classifier. In this paper, the value ofw( xi) is assigned using two kinds of weights which areaverage weight and disproportionately weight. The average
weight means all 3 images with equal weight. The
disproportionately weight is inversely proportional to its
distance from each of the 3 images.

(a)Reference(b) 1st(c) 4th(d) 10th(e) 14th

Figure 2: Results at nth iteration during the relevance feedback
procedure

A. Task 1：Synthesizing a facial image by referring to agiven facial image
Figure 2 shows the synthesized results during therelevance feedback process. 2(b) ~ 2(e) are the output image
after first, fourth, tenth, fourteenth iterations of our relevancefeedback algorithm. We can see the resulting facial image
becomes more resemblance to the reference image graduallywhen the number of iteration increases.
In Figure 3, the final results (user satisfied faces) for 2
reference facial images are shown. Synthesized image 1 has
the similar face shape as the reference face and presents asimilar side hairstyle, although the eyes opened wider than
the reference image. Synthesized image 2 presents similar
hairstyle as reference image. However, it has wider
foreheads with less frontal hair then the reference image andhas a mole at the lower left of the nose. Such detailed
information cannot be synthesized with the proposed method.
Manual attachment of the mole can easily improve thequality of synthesized image.

Technology Department of Zhejiang Province, China (No.
2016C33G2040006/2016C33012).

Figure 4: Scores of final results

Figure 5: Number of iterations required for achieving satisfied result(a)Reference
image 1(b)Synthesizedimage 1(c)Reference
image 2(d)Synthesizedimage 2REFERENCES
X. Wang, and X. Tang, “Face Photo-Sketch Synthesis andRecognition,” IEEE Transaction on Pattern Analysis and MachineIntelligence, vol. 31, no. 11, pp. 1955-1967, 2009.
[2] D. Wu, and Q Dai, “Sketch Realizing: Lifelike Portrait Synthesis
from Sketch,” Computer Graphics International Conference, pp. 1320, 2009.
[3] Q. Liu, X. Tang, H. Jin, H. Lu and S. Ma, “A nonlinear approach forface sketch synthesis and recognition,” IEEE Computer Society
Conference, vol. 1, pp. 1005-1010. 2005.
[4] N. Wang, X. Gao, L. Sun and J. Li, “Bayesian Face Sketch
Synthesis,” IEEE Transactions on Image Processing, vol. 26, no 3, pp.
1264-1274, 2017.
[5] C. Liu, H. Y. Shum, W. T. Freeman, “Face Hallucination: Theory andPractice,” International Journal of Computer Vision, vol. 75, no. 1, pp.
115-134, 2007.
[6] S. Zhu, S. Liu, C. Loy, and X. Tang, “Deep Cascaded Bi-Network forFace Hallucination,” European Conference on Computer Vision, pp.
614-630, 2016.
[7] Y. Sun, Y. Chen, X. Wang and X. Tang, “Deep learning face
representation by joint identification-verification,” Advances in
Neural Information Processing Systems, pp. 1988-1996, 2014.
[8] I. Ruthven, M. Lalmas, “A survey on the use of relevance feedbackfor information access systems,” The Knowledge Engineering
Review, vol. 18, no. 2, pp. 95–145, 2003.
[9] A. T. Silva, A. X. Falcao, L. P. Magalhaes, “Active learning
paradigms for CRIR systems based on optimum-path forestclassification,” Journal of WSCG, vol.18, no. 1-3, pp. 73-80, 2010.
[10] J. P. Papa and A. X. Falcao, “Optimum-path forest: a novel andpowerful framework for supervised graph-based pattern recognition
techniques,” Institute of Computing University of Campinas, pp. 4148, 2010.
[11] H. L. Li, M. Toyoura, K. Shimizu, W. Yang and X. Y. Mao,
“Retrieval of clothing images based on relevance feedback with focus
on collar designs,” Visual Computer, vol. 32, no. 10, pp. 1351-1363,2016.
[12] H. L. Li, G.H. Liu, and K. N. Ngan, “Guided Face CartoonSynthesis,” IEEE Transactions on Multimedia, vol. 13, no. 6, pp.
1230-1239, 2001.
[13] W. Gao, B. Cao, S. G.Shan, X. L. Chen, D. L. Zhou, X. H. Zhang
and D Zhao, “The CAS-PEAL Large-Scale Chinese Face Databaseand Baseline Evaluations,” IEEE Transactions on Systems, Man, andCybernetics (Part A), vol. 38, no. 1, pp. 149-161, 2008.
[1]Figure 3: Results using reference images

B. Task 2: Subjective grade of synthesized faces
In this experiment, we ask subjects to imagine a face, andlet them to use the system to create the images accordingly.
Then, the subjects are asked to grade the similarity of theresulting images to their imagined images with a five-pointsystem (1: No resemblance; 2: very weak resemblance; 3:
Neither weak nor strong resemblance; 4: Somewhat strong
resemblance; 5: strong resemblance).
Figure 4 shows the final scores of this evaluation. The
average final score was 4.0. Figure 5 shows the number ofiterations it took to reach the final result and thecorresponding number of subjects, with the number ofiterations on the x-axis and the number of subjects whoreached their final results at that number of iterations on they-axis. On average, it took 10.9 iterations to reach the final
results. Our method thus earned rating of 4 or higher fromthe majority of experiment participants and produced final
results in a feasible number of iterations.
VI. CONCLUSION AND FUTURE WORKIn this paper, we propose a method for the semiautomaticsynthesis of a facial image in a user’s mind. By training an
OPF based on the user’s feedback, we successfully create
synthesized images that resembled the facial images thatusers had in mind. Currently, the face image is aligned only
by eye distance and caused the blurring. In the future, wewill design more robust alignment algorithm to replacecurrent approach. Also, we will conduct more experiments toverify the system performance.
ACKNOWLEDGEMENT
This work was supported by JSPS KAKENHI Grant
Number 17H00737 and the Public Projects of Science