Style Image Retrieval for
Improving Material Translation
Using Neural Style Transfer
Gibran Benitez-Garcia1,2, Wataru Shimoda, and Keiji Yanai2
OMRON SINIC X Corporation1,
The University of Electro-Communications2
MMArt-ACM ‘201

UEC Yanai Lab. Tokyo.

2020.10.26ⓒ 2019 UEC Tokyo.

Motivation– Results of Neural Style Transfer methods to translate object materials
rely on the style picture chosen to modify the content image– Automatically find the ideal style image that better translates thematerial of an object

Content Image
2

Translated images with different materials
UEC Yanai Lab. Tokyo.

ⓒ 2020 UEC Tokyo.

Objective• Objective:– Automatically find the ideal style image based on its discrimination level
and its relation with the content image in terms of semantic information• Approach:– An image retrieval method based on the most discriminative candidate style
images, and evaluate the semantic similarity with the content using IN

NST
3UEC Yanai Lab. Tokyo.

ⓒ 2020 UEC Tokyo.

Neural Style Transfer– NST* exploits CNN feature activations to recombine the content of agiven photo and the style of artworks
Gaussian noise

Content ImageOutput

Content Feats

Pre-trainedCNNImageOptimizationStyle Feats

Style Image
2020 UEC
Tokyo. CVPR 2016.
4 * Gatys, Leon, et al. "Image style transfer using convolutional ⓒneuralnetworks."
UEC Yanai Lab. Tokyo.

Instance Normalization– IN* computes the mean/standard deviation and normalize across eachchannel in each training example– Generates a network agnostic to the contrast of the original images.
A.k.a. erases style information.
Search results with IN

Query

Search results with BN2020 UEC Tokyo.
5 * Ulyanov, Dmitry, et al. "Instance normalization: The missingⓒingredientfor fast stylization." arXiv 2016.
UEC Yanai Lab. Tokyo.General Proposal

Style Image Retrieval
Foliage images

VGG FEATS +IN WHITENING
material label

FOLIAGE
content image

NEURAL STYLE
TRANSFER+from wood
Input

SEMANTIC

SEGMENTATION
6

Material Translation…………………………
UEC Yanai Lab. Tokyo.

OutputStyle Image Retrieval
1) Search refinement– Choose the best scored material image per class (pre-trained CNN), and theimages with more extensive material regions (segmentation)2) Style removal– IN whitening from the pre-trained CNN, and L2 norm from query and styles

7

UEC Yanai Lab. Tokyo.

ⓒ 2020 UEC Tokyo.

Datasets of object materials• FMD* dataset:– Pixel labels– 10 class materials– 1,000 images in total• Extended-FMD** dataset:– Image labels– Same classes as FMD– 10,000 images in total* Sharan, Lavanya, et al. "Material perception: What can you see in a brief glance?." Journal of Vision, 2009.
8 ** Zhang, Yan, et al. "Integrating deep features for material recognition." ICPR, 2016.
UEC Yanai Lab. Tokyo.

Evaluation of IN-based style retrieval– Classification and segmentation metrics to evaluate generated results:
average accuracy (acc) and mean Intersection over the Union (mIoU).
– Baseline: fixed style images, all processes based on Gatys NST

9UEC Yanai Lab. Tokyo.

ⓒ 2020 UEC Tokyo.

original materials (material A)

Cross-classification of material translation

10translated materials (material B)
Fa Fo Gl Le Me Pa Pl St Wa Wo
Fabric - 64 88 27 68 62 80 72 23 62
Foliage 23 - 70 11 27 24 38 40 12 50
Glass 47 38 - 20 55 41 71 41 22 63
Leather 86 32 81 - 35 21 63 54 6 85
Metal 69 27 94 37 - 28 56 62 10 80
Paper 47 24 32 16 27 - 65 49 11 52
Plastic 68 33 86 45 73 26 - 72 30 48
Stone 71 66 87 7 49 72 74 - 23 94
Water 36 27 68 4 46 37 41 58 - 74
Wood 48 52 90 39 33 33 97 84 9 UEC Yanai Lab. Tokyo.

ⓒ 2020 UEC Tokyo.

Qualitative results– Wood object to different materials, using NST and IN-based style retrieval

11

UEC Yanai Lab. Tokyo.

ⓒ 2020 UEC Tokyo.

Comparison with different NST methods– We evaluated all methods using GAN metrics, i.e., Inception Score (IS),and the Frechet Inception Distance (FID).

12

UEC Yanai Lab. Tokyo.

ⓒ 2020 UEC Tokyo.

Qualitative comparison
NST-Base, NST-IN, WCT-Base, WCT-IN, MUNIT-Base, MUNIT-IN, and StarGAN
Content Image

13

UEC Yanai Lab. Tokyo.

ⓒ 2020 UEC Tokyo.

Conclusion• Conclusions:– We experimentally proved that by defining an image style search with IN,
the results of NST material translation are significantly better.
– Our style retrieval proposal can boost material translation results ofconventional NST methods, such as Gatys, WCT, and MUNIT.

• Future work:– Test and analyze different options for removing the style information.
– Integrate the NST, Segmentation, and Search process for resultsoptimization.
14

UEC Yanai Lab. Tokyo.

ⓒ 2020 UEC Tokyo.

Thank You

15

UEC Yanai Lab. Tokyo.

ⓒ 2019 UEC Tokyo.