Journal of Information Processing Vol.24 No.6 897–907 (Nov. 2016)[DOI: 10.2197/ipsjjip.24.897]

Regular Paper

Texture Transfer Based on Energy Minimization
for Painterly Rendering
Yuta Tsunematsu1

Norihiko Kawai1,a)Tomokazu Sato1,b)

Naokazu Yokoya1,c)Received: April 13, 2016, Accepted: August 10, 2016

Abstract: Non-photorealistic rendering (NPR) creates images with artistic styles of paintings. In this field, a numberof methods of converting photographed images into non-photorealistic ones have been developed, and can be categorized into filter-based and exemplar-based approaches. In this paper, we focus on the exemplar-based approach andpropose a novel method which transfers a style of a reference pictorial image to a photographed image. Specifically,
we first input a pair of target and reference images. The target image is converted by minimizing an energy function
which is defined based on the diﬀerence in intensities between an output image and a target image, and the patterndissimilarity between an output image and a reference image. The proposed method transfers structures and colorsof textures in the reference image and generates continuous textures by minimizing the energy function. In experiments, we demonstrate the eﬀectiveness of the proposed method using a variety of images and examine the influenceof parameter changes and intensity adjustment for pre-processing on resultant images.
Keywords: texture transfer, non-photorealistic rendering, painterly rendering, exemplar-based rendering, energy minimization

1. Introduction
Non-photorealistic rendering (NPR), which creates images
with artistic styles of paintings such as oil paintings and pencil
drawing, has been investigated in the field of computer graphics. Such rendering approaches enable various expressions according to the preference and sensibility of creators even if thetarget scene is the same. In this field, a number of methods forconverting photographed images to non-photorealistic styles havebeen developed [16], and can be categorized into filter-based andexemplar-based approaches.
Many of the filter-based methods focus on strokes and convert
sets of pixel values to various strokes by imitating actual paintingscomposed of multiple strokes. A number of filter-based methods
have been proposed so far [10], [11], [12], [15], [22]. For example, Haeberli [10] proposed a method which selectively converts
a part in a target image into a painterly style until the entire image
is converted. In this method, users manually select a direction anda size of a brush, and the color of the brush is automatically determined according to the intensities of the target image. In [10],a method that automatically determines the positions and directions of strokes using intensity gradients is also proposed. Hertzmann [11] proposed a method that draws strokes based on an actual drawing manner that first draws a rough sketch with a large
brush and then draws details with smaller brushes. Among relatively recent works, Zeng et al. [22] proposed a semantics-driven
method. This method first parses a target image using methods
1
a)
b)
c)

Nara Institute of Science and Technology, Ikoma, Nara 630–0192, Japan
norihi-k@is.naist.jp
tomoka-s@is.naist.jp
yokoya@is.naist.jp

c 2016 Information Processing Society of Japan

in Refs. [9], [19], [20] and represents the image by multiple layers. The method then generates strokes in each layer considering
an orientation field obtained by anisotropic diﬀusion and object
categories. Such filter-based methods can generate various expressions of paintings by adjusting parameters for strokes. However, available painting styles are limited according to methods.
In addition, much knowledge and many experiences of users are
required to obtain their desired results.
There have also been a number of exemplar-based methods
that transfer painterly styles in reference images to target images [2], [8], [13], [17], [18], [21]. These methods just require
users to input reference images with their favorite styles andcan convert target images more easily than filter-based methods.
However, most of the conventional methods evaluate the local
similarity and copy pixel values of the most similar patches in
the reference image to the target image, and the process is iterated from the region with already determined pixel values to thatwith pixel values to be determined by raster scan. Such successive copy often does not preserve the spatial coherence for theentire image. For example, in the case where the reference image does not contain appropriate patches for some local area inthe target image, once an implausible patch is copied there in thetarget image in a step during the raster scan, implausible textures
tend to be continuously copied in subsequent steps. Although themethods iterate the raster scan to try to obtain more natural images, the number of iterations is very heuristic because there isno criterion to represent the plausibility for the entire image. In
addition, while some methods only transfer the structures of textures, users often desire the transfer of both structures and colorsof textures.

897

Journal of Information Processing Vol.24 No.6 897–907 (Nov. 2016)

For these issues, this paper focuses on an exemplar-based approach, and proposes a method to transfer structures and colorsof textures in an energy minimization framework. The contribution of this paper is optimally determining all pixel values byminimizing an energy function that evaluates the plausibility forthe entire image based on both the relationship between output
and target images and the relationship between output and reference images, rather than evaluating the similarity only for local
regions and successively determining pixel values. The proposed
approach transfers both structures and colors in the reference image to the target image with fewer texture disconnections than the
successively copy approach while preserving the characteristicsof the target image.
The rest of this paper is organized as follows. The next section
reviews related work. Section 3 describes the proposed method.
Section 4 shows three kinds of experiments. Finally we conclude
this paper in Section 5.

2. Related Work
This section reviews exemplar-based texture transfer methods. Hetzmann et al. [13] proposed a texture transfer method thatrefers a pair of a photo and its converted image with a painterly
style. Tang et al. [18] improved the method with respect to thecomputational cost by employing CUDA. These methods convert
a target image by learning the relationship between the photo andits converted image. These methods also work with color transfer methods such as the method in Ref. [5]. In Ref. [5], a target
image is converted so that the image has colors that are used ina reference image. In the method, colors in target and reference
images are classified into 11 basic ones according to the BasicColor Terms (BCTs) [4]. Within each category, colors in the target image are transformed to ones used in the reference image.
Although the combination of Refs. [13], [18], and [5] transfers
both textures and colors, users have to prepare a pair of a photoand its converted image according to their preference.
For this problem, texture transfer methods that need only asingle reference image in addition to a target image have been
proposed [2], [8], [17], [21]. Efros et al. [8] proposed a texture
synthesis method and applied it to texture transfer. The method
firstly copies similar texture patches in a reference image to a target image successively while overlapping the partial regions of
the patches. In the overlapping regions, the optimal boundaries
between the patches are calculated using the method in Ref. [7]to achieve the natural connection of textures. Ashikhmin [2] proposed a texture transfer method by extending his texture synthesis
method [1], which successively determines pixel values by rasterscan. During the raster scan, the correspondences of pixels thatwere already made in the raster scan are used for determining the
pixel in a reference image corresponding to a target pixel in atarget image for texture coherence and speeding up of synthesis.
Wang et al. [21] and Lee et al. [17] have focused on directionsof strokes, which are important characteristics in artistic paintings and are not considered in the methods in Refs. [2], [8], [13].
The method in Ref. [21] segments a target image and creates adirection field based on a medial axis of each region. Users thenmanually cut out patches from a reference image and they arec 2016 Information Processing Society of Japan

used for converting the target image. Lee et al. [17] proposed
an automatic transfer method, which is a method extended from
Ref. [2], considering directions of strokes. The method creates asmooth direction field using the method in Ref. [14], which preserves edge directions smoothly around important features. The
transferred image is then generated in a similar way to Ref. [2].
The methods in Refs. [17], [21] mainly transfer structures of textures and do not consider color changes of target images.
In summary, it should be noted that existing exemplar-based
texture transfer methods basically successively transfer partial
textures. Therefore, the order of transfer largely aﬀects results. In
addition, the method often does not preserve the spatial coherence
for the entire image. On the other hand, the proposed method optimally determines the intensity of each pixel independently in anenergy minimization framework, resulting in generating continuous textures in any part in the output image.

3. Texture Transfer by Energy Minimization
Figure 1 shows the overview of the proposed method. The
proposed method converts target image T into output image O
considering both the relationship between output and target images for preserving the structure of textures in target image T andthe relationship between output and reference images for reflecting the characteristics of textures and colors in reference image R
to output image O. Note that the resolution of output image O isthe same as that of target image T but is not necessarily the sameas that of reference image R.
In the following, Section 3.1 defines an energy function, Section 3.2 explains a method for minimizing the energy, and Section 3.3 describes a coarse-to-fine framework for energy minimization.
3.1 Definition of Energy Function
We define an energy function E based on the weighted sum ofS S DOT (Sum of squared diﬀerences between intensities in output image O and target image T ) for preserving the structure of
textures in target image T and S S S DOR (Sum of SSD between
patches in output image O and reference image R) for reflecting
the characteristics of textures and colors in reference image R as
follows:
E = S S DOT + αS S S DOR ,Fig. 1(1)

Overview of the proposed method.

898

Journal of Information Processing Vol.24 No.6 897–907 (Nov. 2016)

Fig. 2Illustration of pairs of similar patches.

where α is a weight for controlling how much textures in the reference image are reflected to the output image. S S DOT is defined
based on the diﬀerence in values of all pixels in output image O
and target image T as follows:S S DOT =||IO (xi ) − IT (xi )||2 ,
(2)
xi ∈O

where IO (xi ) and IT (xi ) are vector representations of RGB values
at pixel position xi of pixel i in the output image O and target image T , respectively. This term preserves the structure of the target
image from largely changing.
S S S DOR totalizes SSD values for all the pairs of patches centered at all the pixels in output image O and some patches in reference image R and is defined as follows:||IO (xi + p) − IR (x j + p)||2 ,(3)
S S S DOR =xi ∈O p∈W

where IR (x j + p) is a vector representation of RGB values at pixel
position x j + p in a reference image R. Pixel j is an arbitrary pixel
in reference image R, but automatically determined when the energy function is minimized. p is a shift vector to indicate a pixel
in a square patch W. Here, if xi + p is out of the image region,
the calculation is skipped. If x j satisfies the condition that x j + p(∃p ∈ W) is out of the image region, such pixel j is excluded
from the candidate list for corresponding pixels in R in advance.
It should be noted here that since the neighboring patches in
O overlap each other and the energy minimization described in
Section 3.2 makes a patch centered at an arbitrary pixel in output
image O similar to some patch in reference image R as shown in
Fig. 2, continuous textures are usually generated at an arbitrary
part in output image O in spite of not explicitly considering the
spatial coherence of textures between adjacent patches.

Fig. 3 Pixel relationship to calculate pixel value IO (xi ).

f(xi ) = arg min
x j ∈Rc 2016 Information Processing Society of Japan||IO (xi + p) − IR (x j + p)||2 .

(4)

p∈W

Here, since it takes much time if we exhaustively search for themost similar patch, we employ PatchMatch [3], which limits therange of search for similar patches, to speed up the searching process by following the state-of-the-art methods in the field of texture synthesis such as Ref. [6].
The process (ii) updates each pixel value vector IO (xi ) of output
image O independently so that energy E can be minimized while
keeping all the pairs of similar patches between output image O
and reference image R determined by process (i) fixed. In thefollowing, we describe a method to determine pixel value IO (xi ).
Here, E can be resolved into each partial energy Exi , whichcontains only pixel value vector IO (xi ) for pixel i as a parameter.
As illustrated in Fig. 3, pixel positions related to xi in the output
image are xi in the target image and f(xi + p) − p in the reference
image, and the partial energy can be represented as follows:||IO (xi )−IR (f(xi +p)−p)||2 , (5)Exi = ||IO (xi )−IT (xi )||2 +αp∈Wwhere the relationship between the total energy E and each partial
energy Exi is represented as follow:

Exi .
(6)
E=
xi ∈O

Since each partial energy Exi is independent with respect to eachpixel value in output image O, we can minimize energy E by minimizing each partial energy Exi independently. Pixel value IO (xi )that minimizes partial energy Exi can be calculated by diﬀerentiating Exi and finding an extremal value as follows.
IO (xi ) =3.2 Energy Minimization
Energy function E in Eq. (1) is iteratively minimized by aframework of greedy algorithm to find a good local minimum.
Here, after initializing pixel values in output image O and pixels j corresponding to ∀i(∈ O) in Eq. (1) (how to initialize themis described in detail in Section 3.3), energy function E is minimized by iterating the following two processes until a condition
for energy convergence is satisfied: (i) searching the reference
image for similar texture patches to determine j in Eq. (3) and (ii)updating pixel values IO in the output image.
The process (i) basically searches the reference image for such
central position of the patch that satisfies the following equation
with respect to each pixel xi in output image O while all the pixel
values in output image O fixed.

IT (xi ) + αp∈W IR (f(xi + p) − p)

1 + αNW,(7)

where NW is the number of pixels in patch W.
3.3 Coarse-to-fine Approach
Resultant images generated by the proposed method largely
depend on the patch size. If the patch size is fixed through theentire process, only textures with a specific scale are reflected on
resultant images. For this problem, we employ a coarse-to-fineapproach, which generates image pyramids and iterates energy
minimization while changing the resolution of the target, reference, and output images. This approach can also avoid bad local
minima and reduce computational cost as well as reflect textureswith diﬀerent sizes on resultant images.
Specifically, we first generate image pyramids with M levels,

899

Journal of Information Processing Vol.24 No.6 897–907 (Nov. 2016)

Fig. 4 Initialization of pixel correspondences.

Algorithm 1 Proposed method
1: Input: T , R, O
2: Generate image pyramids with M levels consisting of T k , Rk and Ok(k = 0, 1, . . . , M − 1)
3: for k = M − 1 to 0 do
4:
if k = M − 1 then5:
Copy T k to Ok
6:
Initialize f(xi ) for ∀i in a random manner
7:
else
8:
Initialize f(xi ) for ∀i using correspondences for the k + 1-th level9:
Ok is initialized by Eq. (7)10:
end if11:
while a condition for energy convergence is not satisfied do
12:for ∀i ∈ Ok do
13:
Find f(xi ) that satisfies Eq. (4) using PatchMatch
14:
end for15:for ∀i ∈ Ok do
16:
Update pixel value IOk (xi ) by Eq. (7)17:
end for18:
end while
19: end for
20: Output: O(= O0 )which contain images whose width and height are recursively reduced by half. Here, we represent the k-th level images of the target, reference, and output images as T k , Rk , and Ok , respectively.
The width and height of k-the level image are 1/2k of those of theoriginal image. We then iterate energy minimization from M − 1th level to 0-th level layers successively using a certain size ofpatch. In the M − 1-th level, pixel values in output image O M−1
are initialized by copying target image T M−1 to O M−1 , and initial
pixel positions f(xi ) = x j corresponding xi are determined in arandom manner by following PatchMatch algorithm [3]. In theother levels k, initial values of Ok are determined by Eq. (7) after
initializing pixel positions f(xi ) = x j corresponding xi . Here, as
shown in Fig. 4, since pixel (u, v) in Ok+1 is the combination offour pixels (2u, 2v), (2u + 1, 2v), (2u, 2v + 1), and (2u + 1, 2v + 1)
in Ok , given that (û, v̂) in Rk+1 is corresponding to (u, v) in Ok+1 ,we set initial pixels corresponding to the four pixels as (2û, 2v̂),(2û + 1, 2v̂), (2û, 2v̂ + 1), and (2û + 1, 2v̂ + 1), respectively.
Finally, we summarize the specific procedure of the proposed
method in Algorithm 1.

4. Experiments
We conducted experiments using four photographs as target
images and four paintings with various characteristics as reference images (Fig. 5) with a standard PC (Core i7-3930K 3.2 GHz
CPU and 16 GB memory). The resolutions of target and referencec 2016 Information Processing Society of Japan

Fig. 5

Target (T1 to T4) and reference (R1 to R4) images.

images were resized so that the short sides become 300 pixels forexperiments to unify the scales of textures in target and reference
images. The characteristics of the target images (T1 to T4) andreference images (R1 to R4) are as follows:
T1 Landscape with detailed textures of buildings in one areaand nature.
T2 City landscape with many detailed textures of buildings andcloud texture.
T3 Flower meadow.
T4 Winter mountain with little texture and low contrast.
R1 Landscape with small buildings and nature with long strokes
and a small number of colors.
R2 Photorealistic countryside landscape.
R3 Landscape with some buildings and nature with dark colors.
R4 Landscape with nature with clear brush strokes with natural
colors.
For the coarse-to-fine approach, we generated four-layer image
pyramids (M = 4). We considered the energy to be converged
when the ratio of energy decrease is less than 0.3% per layer.
In the experiments, we first examine the eﬀectiveness of the
proposed method and the influence of parameters on resultant
images. We then examine transfered results with and without
intensity adjustments in pre-processing for energy minimization
because intensity adjustments may be eﬀective in the proposed
framework especially when the distribution of intensities between
target and reference images are largely diﬀerent.

900

Journal of Information Processing Vol.24 No.6 897–907 (Nov. 2016)

Fig. 6

Results with diﬀerent parameters for target image T1 and reference
image R1.

Fig. 7

Results with diﬀerent parameters for target image T1 and reference
image R2.

Fig. 8Results with diﬀerent parameters for target image T1 and reference
image R3.

In the following, Section 4.1 describes experiments with different patch sizes and weights without intensity adjustment andSection 4.2 describes experiments with intensity adjustment. Section 4.3 compares experimental results obtained by the proposedand conventional methods using the target and reference images
in paper [17].
4.1 Experiments with Diﬀerent Parameters
This section discusses experimental results obtained whilechanging patch sizes and weights for balancing two cost terms.
Specifically, we employed 3 × 3 and 5 × 5 pixels as patch size and0.2, 0.5, and 0.8 as weight α. In this paper, we show converted
images and processing times for some combinations of target andreference images as shown in Figs. 6 to 16. In the following, we
first discuss individual results and then describe the overall tendency with diﬀerent parameters and various combinations of target and reference images.
Figures 6 to 9 show results for target image T1, which has detailed texture in one part and a high color contrast for the entire
image. From the results, we confirmed that the target image wassurely converted to painterly style, reflecting the characteristics
of textures in the reference images, but the amount of reflectancelargely depended on the parameter values. For example, whenW = 3 × 3 and α = 0.2, the conversion was slight but the colorof the reference images were surely reflected. When W = 3 × 3and α = 0.5, the strokes in the reference images were also reflected. However, as for reference images R1 and R4, which haveclear strokes, the reflectance of strokes is insuﬃcient as shown in
Figs. 6 and 9. When W = 5 × 5, the long strokes and relatively
large textures were reflected to the results; however, as a side effect, some detailed textures such as buildings got lost as weight α
became larger. For the combination of T1 and R1 with W = 3 × 3

c 2016 Information Processing Society of Japanand α = 0.5, we show the relationship between the normalizedenergy and the number of iterations in Fig. 10. From the graph,
we confirmed that the energy is steadily reduced by each iteration.
Figures 11 and 12 show results for target image T2, which
has detailed textures of buildings and cloud textures. Since thereference images do not contain textures corresponding to buildings in the target image, the detailed textures of buildings became
destroyed as the patch size and the weight got larger. However,when W = 3 × 3 and α = 0.5, the reflectance of the strokes in the901

Journal of Information Processing Vol.24 No.6 897–907 (Nov. 2016)

Fig. 9Results with diﬀerent parameters for target image T1 and reference
image R4.

Fig. 11

Results with diﬀerent parameters for target image T2 and reference
image R1.

Fig. 12Results with diﬀerent parameters for target image T2 and reference
image R4.

Fig. 10 Relationship between energy and number of iterations.

reference images is slight but actually noticeable while keeping
the rough structure of textures in the target image.
Figures 13 and 14 show results for target image T3, which has
detailed textures of flowers and grass. From the results, we cansee that the color tones of reference images are well reflected tothe target image, respectively. When W = 3 × 3 and α = 0.8
in Fig. 14, the strokes of the reference image are also surely reflected. However, we also confirmed that the detailed texture ofgrass became lost when W = 5 × 5.
Figures 15 and 16 show results for target image T4, which has
little texture and low contrast because of snowy scene. From theresults, we can confirm that the structure of textures in the target
image is not preserved when the patch size and the weight become even slightly large. Since the contrast of the target image
is low and the image contains similar texture patterns throughout
the entire region, some texture patterns in the reference images
are intensively used as exemplars. As a result, the conversion lostdetailed structure in the target image.
As for the overall tendency with respect to patch size andweight α, as the patch size and weight α become larger, the strokein a reference image gets reflected more, but the details of texture
in a target image get lost. Considering an ideal conversion thatc 2016 Information Processing Society of Japanpreserves the structure of textures in target images and reflects
the paintings style of reference images as much as possible, it
may be good to employ a large patch size for regions with little
and homogeneous texture in target images such as sky, sea andmountains for reflecting the strokes in reference images and employ a small patch size for regions with detailed textures in target
images such as buildings for preserving the structure of textures.
To achieve this, we should consider a framework to adaptively
change patch sizes according to the characteristics of textures in902

Journal of Information Processing Vol.24 No.6 897–907 (Nov. 2016)

Fig. 13

Results with diﬀerent parameters for target image T3 and reference
image R2.

Fig. 15

Results with diﬀerent parameters for target image T4 and reference
image R3.

Fig. 14

Results with diﬀerent parameters for target image T3 and reference
image R4.

Fig. 16

Results with diﬀerent parameters for target image T4 and reference
image R4.

target images in future work. Currently, although the quality of
the results depend on the patch size and weight, we could obtain averagely good results with 3 × 3 pixels patch and α = 0.5.
Therefore, we employ these values for the next experiments withintensity adjustment.
As the processing time for each conversion is shown in eachfigure, we confirmed that the processing time becomes shorter as
weight α becomes larger. This is because larger weights speed
up the convergence of energy. We also confirmed that the time is

c 2016 Information Processing Society of Japan

suﬃciently short for practical use.
4.2 Experiments with Intensity Adjustment
This section discusses experimental results obtained by the
proposed method with intensity adjustment. In this experiment,
we employed 3 × 3 pixels as patch size and 0.5 as weight α asmentioned above.
For intensity adjustment, we modified the intensities of the target image so that the histogram of the target image can correspond903

Journal of Information Processing Vol.24 No.6 897–907 (Nov. 2016)

Fig. 17

Intensity adjustment of target image.

to that of the reference image. Specifically, we first convert thecolors of target and reference images from RGB to HSV color
space and sort pixels of target and reference images in ascendingorder of their V values, respectively. We then change the V valueof k-th pixel in the target image to that of l-th pixel in the reference image, as shown in Fig. 17, where the relationship between
k and l is represented as follows:


NR
l=
k ,
(8)
NT
where [·] rounds down a value to an integer, and NT and NR indicate the resolution of images T and R, respectively. Finally, thecolor space of the target image is converted from HSV to RGB. In
this experiment, we regarded the intensity-adjusted target image
or the original target image as target image T .
Figures 18 to 21 show target images with intensity adjustment
and converted results for target images T1 to T4 with two reference images, respectively. In the following, we discuss the results.
Figure 18 shows the results of target image T1 converted withreference images R1 and R2. The left images in Fig. 18 wereobtained by intensity adjustment so that the histogram of the target image becomes almost the same with those of the reference
images. From the results, we confirmed that the results with intensity adjustment were worse than those obtained using the same
parameter W = 3 × 3 and α = 0.5 without intensity adjustment
as shown in Figs. 6 and 7. Since the adjustment enhanced thecontrast of intensities in sky and mountain regions, the high contrast caused inappropriate texture correspondences, resulting in
implausible converted images.
Figure 19 shows the results of target image T2 converted withreference images R1 and R4. When reference image R1 wasused, the target image after intensity adjustment became dark.
As a result, the converted result provides users with an impression diﬀerent from the result without intensity adjustment shownin Fig. 11. When reference image R4 was used, the intensity adjustment brightened the entire target image. The converted image
with the brighter target image has low contrast and pale colors.
Since the color tones and strokes in the reference images are reflected to the results for both cases when intensity adjustment isapplied or not, the evaluation may depend on the user’s preference.
Figure 20 shows the results of target image T3 converted withreference images R2 and R4. The target image was brightenedwhen reference images R2 and R4 were used. The strokes in
the reference images were reflected to the converted results thanc 2016 Information Processing Society of Japan

Fig. 18 Results of T1 with intensity adjustment.

Fig. 19 Results of T2 with intensity adjustment.

Fig. 20 Results of T3 with intensity adjustment.

those without intensity adjustment shown in Figs. 13 and 14;
however, the blurry eﬀects were also generated for both results.
Figure 21 shows the results of target image T4 converted withreference images R3 and R4. Since target image T4 has low con-904

Journal of Information Processing Vol.24 No.6 897–907 (Nov. 2016)

Fig. 21 Results of T4 with intensity adjustment.

trast, the intensity adjustment enhanced the contrast. As a result,the color tones and strokes in the reference images R3 and R4 arewell reflected and more natural resultant images were obtained
compared with the results without intensity adjustment shown in
Figs. 15 and 16.
In summary, when the contrast of the target image is low, theintensity adjustment by making the histogram of target image
comparable to that of the reference image often enhances the contrast and tends to reflect the characteristics of textures in a reference image to the converted result more clearly. In other cases,
the intensity adjustment does not always make converted results
better; however, it may generate additional converted results thathave diﬀerent strokes and brightness from those obtained by the
proposed method without intensity adjustment. For practical use,since we cannot know the users’ preferences, a system in whichusers can select their desired results by outputting converted results both with and without intensity adjustment may be useful.
4.3 Comparison
This section compares the results obtained by the proposed
method with those obtained by one of the state-of-the-art methods [17]. Figure 22 shows a target image and three reference
images, and Fig. 23 shows results by the proposed method andthe method in Ref. [17].
Reference image A is an oil painting with clear strokes anddistinct use of colors. In the resultant image converted by the
proposed method, the reflectance of strokes is slight but actuallyconfirmable, and the colors of the entire resultant image are surely
influenced by those in the reference image. In particular, the combination of blue and brown strokes appears in the entire resultant
image. On the other hand, in the resultant image obtained by theconventional method, clear stroke shapes similar to those in thereference image are generated in the resultant image. However,the color tones and the combination of blue and brown strokes are
not reflected because the conventional method does not intend toreflect color tones.
Reference image B is a water painting of fruits that are drawn
on a piece of bumpy paper. In the resultant image converted by

c 2016 Information Processing Society of Japan

Fig. 22

Target and reference images for comparison.

the proposed method, color tones used in the reference image appears over the entire image. The bumps on the paper are also
reflected. However, since the proposed method does not have theeﬀect of preserving edges, the bleeding of black ink can be observed. On the other hand, the bumps also seem to be reflected to
the resultant image by the conventional method, but it may not beeasy to clearly find the eﬀect.
Reference image C is an image with pastel colors. The proposed method reflects the strokes observed in the surface of thewater and the bird to the sky and cloud regions and also reflects
color tones to the entire target image. In the resultant image obtained by the conventional method, it may be diﬃcult to recognize the eﬀect of the reflectance because of the diﬀerence in color
tones.
From the overall results, it seems to be easier to recognize theeﬀect of the proposed method than the conventional method because the proposed method reflects not only strokes but also color
tones. However, for detailed textures such as the framework of
the tower, the proposed method tends to generate blurrier textures than the conventional method. The diﬀerence is remarkableespecially for the results for Reference A. Since pixel values are
determined by the uniform average of pixel values of patches forenergy minimization, the generated textures tend to blur if thereference image does not contain the texture whose structure isquite similar to that in the target image. In addition, the reflection of textures with large scales is still insuﬃcient. Although
we may reflect textures with large scales if we use larger patches,large patches often do not preserve the structure of the target image. To overcome these problems, we should consider a methodthat automatically determines the position-dependent patch size
according to the fineness of the texture in the target image andthe existence of similar textures in the reference image because

905

Journal of Information Processing Vol.24 No.6 897–907 (Nov. 2016)

because starting from larger level M (smaller resolution of the image) with a certain patch size is equivalent to using patches whichhave a larger size, and vice versa.

5. Conclusion
This paper has proposed an exemplar-based method that transfers a painterly style in a reference image to a photograph. The
texture transfer is achieved by minimizing an energy functionthat consists of two terms: intensity diﬀerences between target and converted (output) images, and pattern dissimilarity between reference and converted images. In experiments, we examined the influence of parameters and intensity adjustment ontransfered images. We also compared converted images obtainedby the proposed method with those obtained by a conventional
method. From the results, we confirmed that the proposed method
does not always reflect the strokes suﬃciently, but handling bothstrokes and color tones is eﬀective to convert a target image into apainterly style. In future work, we should develop a method that
adaptively adjusts the patch size in energy definition considering
the fineness of textures and the existence of similar textures.
Acknowledgments This work was partially supported by
JSPS KAKENHI No. 15K16039.
References
[1]
[2]
[3]

[4]
[5]
[6]
[7]
[8]
[9]
[10]
[11]
[12]
[13]
[14]
Fig. 23 Comparison with the method in Ref. [17].

[15]large patches reflect textures with large scales and small patches
preserve the detailed structure of the target image without blurs
even if the reference image contains few similar textures. Considering the level M of the coarse-to-fine approach also may improve the reflection of textures and the preservation of structures[16]

c 2016 Information Processing Society of Japan

[17]

Ashikhmin, M.: Synthesizing Natural Textures, Proc. Symposium on
Interactive 3D Graphics, pp.217–226 (2001).
Ashikhmin, M.: Fast Texture Transfer, IEEE Computer Graphics andApplications, Vol.23, No.4, pp.38–43 (2003).
Barnes, C., Shechtman, E., Finkelstein, A. and Goldman, D.B.: PatchMatch: A Randomized Correspondence Algorithm for Structural Image Editing, ACM Trans. Graphics, Vol.28, No.3, pp.24:1–24:11(2009).
Berlin, B. and Kay, P.: Basic Color Terms: Their Universality and
Evolution, University of California Press (1969).
Chang, Y., Saito, S., Uchikawa, K. and Nakajima, M.: Example-based
Color Stylization of Images, ACM Trans. Applied Perception, Vol.2,
No.3, pp.322–345 (2005).
Darabi, S., Shechtman, E., Barnes, C., Goldman, D.B. and Sen, P.:
Image Melding: Combining Inconsistent Images Using Patch-based
Synthesis, ACM Trans. Graphics, Vol.31, No.4, pp.82:1–82:10 (2012).
Davis, J.: Mosaics of Scenes with Moving Objects, Proc. IEEE Conference on Computer Vision and Pattern Recognition, pp.354–360(1998).
Efros, A.A. and Freeman, W.T.: Image Quilting for Texture Synthesis
and Transfer, Proc. SIGGRAPH, pp.341–346 (2001).
Guo, C.-E., Zhu, S.-C. and Wu, Y.N.: Primal Sketch: Integrating
Structure and Texture, Computer Vision and Image Understanding,Vol.106, No.1, pp.5–19 (2007).
Haeberli, P.: Paint by Numbers: Abstract Image Representations,
ACM SIGGRAPH Computer Graphics., Vol.24, No.4, pp.207–214(1990).
Hertzmann, A.: Painterly Rendering with Curved Brush Strokes ofMultiple Sizes, Proc. SIGGRAPH, pp.453–460 (1998).
Hertzmann, A.: Paint by Relaxation, Proc. Computer Graphics International 2001, pp.47–54 (2001).
Hertzmann, A., Jacobs, C.E., Oliver, N., Curless, B. and Salesin, D.H.:
Image Analogies, Proc. SIGGRAPH, pp.327–340 (2001).
Kang, H., Lee, S. and Chui, C.K.: Coherent Line Drawing, Proc. International Symposium on Non-photorealistic Animation and Rendering,pp.43–50 (2007).
Kovács, L. and Szirányi, T.: Painterly Rendering Controlled by Multiscale Image Features, Proc. Spring Conference on Computer Graphics, pp.177–184 (2004).
Kyprianidis, J.E., Collomosse, J., Wang, T. and Isenberg, T.: State of
the “Art”: A Taxonomy of Artistic Stylization Techniques for Images
and Video, IEEE Trans. Visualization and Computer Graphics, Vol.19,
No.5, pp.866–885 (2013).
Lee, H., Seo, S. and Yoon, K.: Directional Texture Transfer with Edge
Enhancement, Comput. Gr., Vol.35, No.1, pp.81–91 (2011).

906

Journal of Information Processing Vol.24 No.6 897–907 (Nov. 2016)

[18]

[19]
[20]
[21]
[22]

Tang, Y., Shi, X., Xiao, T. and Fan, J.: An Improved Image Analogy Method Based on Adaptive CUDA-accelerated Neighborhood
Matching Framework, The Visual Computer, Vol.28, No.6, pp.743–
753 (2012).
Tu, Z., Chen, X., Yuille, A.L. and Zhu, S.-C.: Image Parsing: Unifying Segmentation, Detection, and Recognition, International Journalof Computer Vision, Vol.63, No.2, pp.113–140 (2005).
Tu, Z. and Zhu, S.-C.: Parsing Images into Regions, Curves, andCurve Groups, International Journal of Computer Vision, Vol.69,
No.2, pp.223–249 (2006).
Wang, B., Wang, W., Yang, H. and Sun, J.: Eﬃcient Example-based
Painting and Synthesis of 2D Directional Texture, IEEE Trans. Visualization and Computer Graphics, Vol.10, No.3, pp.266–277 (2004).
Zeng, K., Zhao, M., Xiong, C. and Zhu, S.-C.: From Image Parsing to
Painterly Rendering, ACM Trans. Graphics, Vol.29, No.1, pp.2:1–2:11(2009).

Naokazu Yokoya received his B.E.,
M.E., and Ph.D. degrees in informationand computer sciences from Osaka University in 1974, 1976, and 1979, respectively. He joined the ElectrotechnicalLaboratory (ETL) of the Ministry of International Trade and Industry in 1979. He
was a visiting professor at McGill University in Montreal in 1986–87 and has been a professor at Nara
Institute of Science and Technology (NAIST) since 1992. He has
also been a vice president at NAIST since April 2013. He is afellow of IPSJ, IEICE and VRSJ and a member of IEEE, ACM
SIGGRAPH, JSAI, JCSS and ITE.

Yuta Tsunematsu received his B.E. degree in information and computer sciencesfrom Osaka University in 2012. He received his M.E. degree in information science from Nara Institute of Science andTechnology (NAIST) in 2015. He has
been working for Ricoh IT Solutions Co.,
Ltd. from 2015.

Norihiko Kawai received his B.E. degree in informatics and mathematical science from Kyoto University in 2005. Hereceived his M.E. and Ph.D. degrees in information science from Nara Institute of
Science and Technology (NAIST) in 2007
and 2010, respectively. He was a research
fellow of the Japan Society for the Promotion of Science and a postdoctoral fellow at the University ofCalifornia at Berkeley in 2010–2011. He has been an assistantprofessor at NAIST since 2011. He is a member of IEEE, IEICE,IPSJ and VRSJ.

Tomokazu Sato received his B.E. degree
in computer and system science from Osaka Prefecture University in 1999. He received his M.E. and Ph.D. degrees in information science from Nara Institute of
Science and Technology (NAIST) in 2001
and 2003, respectively. He was an assistant professor at NAIST in 2003–2011.
He was a visiting researcher at Czech Technical University in
Prague in 2010–2011. He has been an associate professor at
NAIST since 2011. He is a member of IEEE, IEICE, IPSJ, VRSJ
and ITE.

c 2016 Information Processing Society of Japan

907