PARTIAL STYLE TRANSFER
USING WEAKLY SUPERVISED SEMANTIC SEGMENTATION
Shin Matsuo

Wataru Shimoda

Keiji Yanai

Department of Informatics, The University of Electro-Communications, Tokyo
1-5-1 Chofugaoka, Chofu-shi, Tokyo 182-8585 JAPAN
Email: {matsuo-s,shimoda-k,yanai}@mm.inf.uec.ac.jp

ABSTRACTIn this paper, we propose a partial texture style transfer method by combining a neural style transfer methodwith segmentation. A style transfer technique based on
Convolutional Neural Network (CNN) can change appearance of an image naturally while keeping its structure. We extend this algorithm for changing appearance
partly in a given image. For example, changing a ballmade of “leather” in the image to one made of “metal”.
The original algorithm changes the style of an entire image including the style of background even though wewant to change only object regions. Therefore, we need
information of target object position, in order to transfer
texture styles to only object region in an image. We segment target object regions using a weakly supervised segmentation method and transfer a given texture style to
only the segmented regions. As results, we achieved partial style transfer for only specific object regions, whichenables us to change materials of objects in a given image
as we like.
Index Terms— style transfer, weakly supervised
segmentation
1. INTRODUCTION
In 2015, Gatys et al. proposed an algorithm on neuralartistic style transfer [1, 2] which synthesizes an image
which has the style of a given style image and the contents of a given content image using Convolutional Neural Network (CNN). This method enables us to modify
the style of an image keeping the content of the image easily. It replaces the information which are degraded while the signal of the content image goes forward through the CNN layers with style information extracted from the style image, and reconstructs a newimage which has the same content as a given content images and the same style as a given style image as shownin Figure 1. In this method, they introduced “style matrix” which was presented by Gram matrix of the featuremaps, that is, correlation matrix between feature maps
in CNN.
978-1-5090-6067-2/17/$31.00 ©2017 IEEE

Fig. 1. An example result of the neural style transfer.
In this work, we apply this neural style transfer algorithm to changing of the material of objects in an image. As a material image dataset, we use Flickr Material
Database (FMD) [3] which is widely used for material image analysis, and we realize material change of images.
The method proposed by Gatys et al. [1, 2] the style ofan entire image including the style of background eventhough we want to change only object regions. Therefore, we need information of target object position, inorder to transfer texture styles to only object regions in agiven image. We segment the regions of the target materials using a weakly supervised segmentation method andtransfer the style of the given materials to only the target
regions. As results, we achieved partial style transfer for
only specific material regions, which enables us to changematerials of objects in a given image as we like. In theexperiments, we apply material semantic segmentation
to the synthesized images the style of the target materials of which was replaced with the style of the othermaterial to confirm if material changes were correctly
done. If material change becomes possible, various kindsof applications can be possible such as interior designand clothing design.
2. PROPOSED METHOD
2.1. Overview
In the proposed method, we combine neural style transfer
with semantic segmentation.

Fig. 3. The algorithm of neural style transfer.
Fig. 2. Processing flow.
Figure 2 shows the processing flow of the proposed
method.
1. Change the style of the material images by Gatys
et al.’s neural style transfer method [1].
2. Estimate the regions corresponding to the given
material by a semantic segmentation method.
3. Synthesize the image in which the material ischanged by integrating the material regions of thetransferred images and the background regions of
the original images into the final output images.
4. Confirm if the material is changed correctly by applying semantic material segmentation.
2.2. Neural Style Transfer
First, we transform the style of an image using the neuralstyle transfer method proposed by Gatys et al. [1, 2]. We
represent an input image to be transformed as xc , a given
style image the style of which is transferred as xs , andan output image which is a synthesized image with thecontent of xc and the style of xs as xg . In the algorithm,
we repeatedly modify xg so that the content features ofxg extracted from CNN becomes close to the content
features of xc and the style features of xg becomes close
to the style features of xs . After several tens of iteration,
we obtain a synthesized image.
Following [1], we use VGG19 [4] pre-trained with the
1000-class ILSVRC dataset as a base CNN for feature extraction, and we extract content features from conv4_2
and style features from five layers (conv1_1, conv2_1,
conv3_1, conv4_1, and conv5_1).
We use activations (a 3D tensor) F (x, l) of layer l as
content representation of layer l. The loss function regarding content features which is the difference between
F (xc , l) and F (xg , l) is represented by the following equation:
1X
2
Lc (xc , xg ) =(Fi,j (xc , l) − Fi,j (xg , l))
(1)
2 i,j

On the other hand, according to [1], we use Grammatrix G(x, l) of activations of layer l as a style representation. It is the original finding by Gatys et al.
that Gram matrix of CNN activations represents a styleof an image efficiently. The loss function regarding content features which is the difference between G(xs , l) andG(xg , l) is represented by the following equation:
G(x, l) = F (x, l)F T (x, l)(2)
Losss,l (xs , xg , l) =1 X
2(Gi,j (xs , l) − Gi,j (xg , l))4Nl 2 i,jLosss (xs , xg ) =X

wl Losss,l

(3)
(4)

l

The loss function is represented by the following
equation:
Loss(xc , xs , xg ) = wc Lossc + ws Losss(5)
where wc and ws are weighting constants. We estimate
xg so as to minimize this loss function with the L-BFGS
method. The estimated xg was the image with the content of xc and the style of xs .
2.3. Weakly Supervised Segmentation
In this paper, as a method on semantic segmentation,
we use CNN-based weakly-supervised semantic segmentation proposed by Shimoda et al. [5]. With a weaklysupervised semantic segmentation method, we can train
a segmentation model from training images having onlyclass labels without pixel-level annotation. In thismethod, they improved class-specific saliency maps proposed by Simonyan et al. [6] which is a back-propagationbased object region estimation method, and proposed amethod to obtain “Distinct Class-specific Saliency Maps
(DCSM)”. DCSM can be used as unary potentials ofdense CRF [7]. Figure 4 shows the procedure of the
DCSM-based weakly-supervised semantic segmentation.
2.3.1. Training CNN
For preparation of CNN-based semantic segmentation,
we need to train a CNN with a multi-label loss function. As an off-the-shelf basic CNN architecture, we usebicycle

bicyclepersonFig. 4. The procedure of CNN-based weakly-supervised
semantic segmentation.
the VGG-16 [6] pre-trained with the 1000-class ILSVRC
dataset. In this framework, a CNN is fine-tuned withtraining images with only image-level multi-label annotation.
Recently, fully convolutional networks (FCN) which
accept arbitrary-sized inputs are used commonly inworks on CNN-based detection and segmentation such as[8] and [9], in which fully connected layers with n units
were replaced with the equivalent convolutional layers
having n 1×1 filters. Following them, we introduce FCN
to enable multi-scale generation of class saliency maps.
2.3.2. Saliency Maps
Recently, a convolutional neural network (CNN) trainedwith only image-level annotation has been known to havethe ability to localize trained objects in an image. Simonyan et al. [10] proposed class saliency maps based on
the gradient of the class score with respect to the input
image, which showed weakly-supervised object localization could be done by back-propagation-based visualization. However, their class saliency maps are vague andnot distinct as shown in Figure 5. When different multiple kinds of target objects are included in the image, themaps tend to respond to all the object regions.
To resolve the weaknesses of their method, Shimoda
et al. [5] propose a new method to generate CNNderivatives-based saliency maps. The proposed method
can generate more distinct class saliency maps which discriminate the regions of a target class from the regionsof the other classes. The generated maps are so distinct
that they can be used as unary potentials of CRF directly.
To make class saliency maps clearer, they propose
three improvements [5]: (1) using CNN derivatives withrespect to feature maps of the intermediate convolutional layers with up-sampling instead of an input image;(2) subtracting saliency maps of the other classes fromsaliency maps of the target class to differentiate target

train

person

train

chair
sofa
Simonyan et al.

chair

sofa
DCSM

Fig. 5. Obtained class saliency maps (Left) by Simonyan
et al. [10] (Right) by the DCSM [5].
objects from other objects; (3) aggregating multiple-scaleclass saliency maps to compensate lower resolution of thefeature maps.
2.3.3. Dense CRF
Conditional Random Field (CRF) is a probabilistic
graphical model which considers both node priors andconsistency between nodes. By using CRF, we can obtain smoother regions from roughly estimated region
potentials. Because object class-specific saliency maps
(OCSM) represent only probability of the target classes
on each pixel and have no explicit information on object
region boundaries, we apply CRF to obtain more distinct object boundaries. In the framework, we use Dense
CRF [7] where every pixel is regarded as a node, andevery node is connected to every other node. The energyfunction is defined X
as follows: X
E(c) =
θi (ci ) +θi,j (ci , cj )
(6)
i

i,j

where ci represents a class assignment on pixel i. The
first unary term of the above equation is calculated fromclass saliency maps M̂ic . We defined it as θi (ci ) =c− log(M̂x,y).
In our work, we introduce background label extensionin addition to the method by Shimoda et al. [5]. Using
class saliency maps of the target classes, we estimate thebackground potential as follows:c
Mbg = 1 − max Mx,y(7)c∈target

Note that the classes the likelihood estimated by themulti-class CNN of which exceed the pre-defined thresholds are selected as target classes.
3. EXPERIMENTS
In the experiments, we use images in the Flickr Material
Database (FMD) [3] which contains ten kinds of material classes (fabric, foliage, glass, leather, metal, paper,ground-truth region, respectively.
Table 1 shows Pixel Acc and IU on the materialchanged results shown in Figure 7. Both measurements
on fabric, foliage and stone show relatively large values.
Note that in this experiments, we applied one style image per material category, while we applied twenty style
images per material category in the next experiments.

water

resultwater(red)plastic

resultfoliage(green)plastic(gray)fabricresultfabric(red brown)

paper(skyblue)stone(brown)

wood

resultleather(blue)

water(red)wood(lightgreen)

Table 1. The evaluation scores (Pixel-wise accuracy andmean IU) of the segmentation results after transformation shown in Figure 7.

Fig. 6. Examples of segmentation results and classspecific saliency maps of material images.
plastic, stone, water, and wood) and 100 images for eachclass. Figure 6 shows the obtained saliency maps of some
FMD matarial images by DCSM [5]. In the experiments,
we picked up 20 images from each class and totally 200
images as style images, and we used two images (leather
jacket and leather ball) shown in Figure 7 as base content
images.
As shown in Figure 2, we carried out style transfer
of whole images, semantic segmentation on ten material
classes, replacing materials, and re-segmentation of theimages in which materials were changed. We applied 200
style images to two kinds of content images. That is,
we repeated that procedures 400 times. Figure 7 shows
20 results out of 400 kinds of combinations. Note thatthe colors of the color bars in the left of the material
names correspond to the colors of the pixels in the resegmentation result images. This shows that changing
materials into fabric, foliage and stone were successful,while changing to glass, leather, metal, plastic, paper,
water and woods were only partly successful.
We evaluated accuracy of re-segmentation of stylechanged images over 400 combinations with the pixelwise accuracy (Pixel Acc.) and the mean of intersection
over union (Mean IU). Pixel Acc, accpixel_acc , represents
the ratio of correctly labeled pixels among all the pixel
in a given image, while Mean IU, accmean_IU , represents the ratio of the intersection of estimated pixels andground-truth pixels over their union which penalized if
the detected regions are either too large or too small.
That it, Mean IU is more strict measurement than Pixel
ACC.
tn
accpixel_acc =(8)Nwhere tn is the number of correctly labeled pixels and N
is the number of all the pixels.
#(Se ∩ Sgt )(9)accmean_IU =#(Se ∪ Sgt )where (S) represents the number of pixels of region S,
and Se and Sgt represents the estimated region and theclass
fabricfoliageglass
leathermetal
paper
plastic
stone
water
wood

pixel
acc.
0.64460.75560.44090.58580.30860.4491
0.32270.7186
0.46830.3550mean
IU
0.61320.73090.55200.5756
0.45630.49830.42120.72930.50730.4567pixel
acc.
0.80910.86680.43010.5706
0.34560.3196
0.1931
0.94310.47190.2701mean
IU
0.81470.71010.59990.68970.56160.53430.30120.90540.64540.5227

Figure 8 and Figure 9 shows the average statistics ofPixel Acc and mean IU on each changed material with20 style images per material class. These graphs show
the same tendency as Table 1.
4. DISCUSSION
From the results shown in Figure 7, we have confirmedthat in many cases the proposed method enabled partial
style transfer and changing materials of objects keeping
the shapes of objects. The results over 400 combinations
shown in Figure 8 and Figure 9 indicated that some materials were easy to transfer and some were not easy.
Especially, we found that fabric, foliage and stone wereeasy materials to transfer, while metal, glass and plastic were hard materials to transfer for the current styletransfer method.
The transferred results on foliage, fabric and stone
seems natural and easy to recognize into which material
an input image transferred even for human. The style
images of these styles have irregular and small-scale textures which are easy to transform, and it makes styletransfer easy.
In addition, it was turned out that we obtained morenatural results when the content of the style image isclose to the content of the content image. Figure 10
shows the example of this case.

Fig. 8. The accuracy of re-segmentation results regarding Pixel Acc.

Fig. 10. Advantage of Common contents between content images and style images.
ferred images using both results. These two independent CNN-based processing sometimes made unnatural
boundaries in the transferred images. For future work,
we plan to make an end-to-end network which realizes
partial style transfer including both processing of segmentation and style transfer.
Fig. 9. The accuracy of re-segmentation results regarding mean IU.

Acknowledgment This work was supported by JSPS KAKENHI Grant Numbers 15H05915 and 17H06026.

5. CONCLUSIONS[1] L. A. Gatys, A. S. Ecker, and M. Bethge, “A neural algorithm ofartistic style,” in arXiv:1508.06576, 2015.

In this paper, we examined if neural style transfer technique could change the material of objects. To do that,
we proposed a combination of neural style transfer andsemantic material image segmentation. In the experiments, we examined 400 combinations with two content
images and 20 style images for each material class. As results, in many cases, changing materials of objects weresuccessfully done, and we observed the tendency thatsome materials were easy to transfer and some were noteasy. Especially, we found that fabric, foliage and stone
were easy materials to transfer, while metal, glass andplastic were hard materials to transfer for the current
style transfer method. In addition, it was turned outthat we obtained more natural results when the content
of the style image is close to the content of the content
image.
For future work, considering the finding of this work,
we will propose a method to select better style images or
better part of style images automatically, and improve
the neural style transfer method so that it can transfer
the glossy materials such as metal, glass and plastics.
In this work, we carried out style transfer and segmentation independently, and synthesize partial trans-[2] L. A. Gatys, A. S. Ecker, and M. Bethge, “Image style transfer
using convolutional neural networks,” in Proc. of IEEE Computer
Vision and Pattern Recognition, 2016.

6. REFERENCES[3] C. Liu, L. Sharan, E. Adelson, and R. Rosenholtz, “Exploring
features in a bayesian framework for material recognition,” inProc. of IEEE Computer Vision and Pattern Recognition, 2010.
[4] K. Simonyan and A. Zisserman,“Very deep convolutional
networks for large-scale image recognition,”in Proc. ofarXiv:1409.1556, 2014.
[5] W. Shimoda and K. Yanai, “Distinct class-specific saliency maps
for weakly supervised semantic segmentation,” in Proc. of European Conference on Computer Vision, 2016.
[6] K. Simonyan, A. Vedaldi, and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in International Conference on Learning Representations, 2015.
[7] P. Krahenbuhl and V. Koltun, “Efficient inference in fully connected crfs with gaussian edge potentials,” in Advances in Neural
Information Processing Systems, 2011.
[8] M. Oquab, L. Bottou, I. Laptev, and J. Sivic, “Is object localization for free? -weakly-supervised learning with convolutional
neural networks,” in Proc. of IEEE Computer Vision and Pattern Recognition, 2015.
[9] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in Proc. of IEEE Computer
Vision and Pattern Recognition, 2015.
[10] K. Simonyan, A. Vedaldi, and A. Zisserman, “Deep inside convolutional networks: Visualising image classification models andsaliency maps,” in International Conference on Learning Representations, 2014.

Fig. 7. The results of material-changed images and their results of semantic material segmentation.