Texture Synthesis Based on Aesthetic Texture
Perception Using CNN Style and Content
Features
Yukine Sugiyama1(B) , Natsuki Sunda1 , Kensuke Tobitani1,2, and Noriko Nagata1

1 Kwansei Gakuin University, Sanda, Hyogo 669-1337, Japan{ggs53875,nagata}@kwansei.ac.jp, tobitani@sun.ac.jp
2 University of Nagasaki, Nishi-Sonogi, Nagasaki 851-2195, Japan

Abstract. We propose a texture synthesis method that controls the desired visual
impressions using CNN style features and content features. Diversifying userneeds has led to the personalization of products according to individual needs. In
the custom made garment service, users can select and combine fabrics, patterns,
and shapes of garments prepared in advance to design garments that meet theirtastes and preferences. Furthermore, controlling the visual impressions will enable
the service to provide designs that better match the user’s preferences. In image
synthesis, controllable texture synthesis have been performed with style and content, however, few previous studies have controlled images based on impressions(including aesthetics). In this study, we aim to synthesize textures with desired
visual impressions by using style and content features. For this purpose, we first(1) quantify the affective texture by subjective evaluation experiments and (2)extract style features and content features using VGG-19 from pattern images
for which evaluation scores are assigned. The explanatory variables are style andcontent features, and the objective variables are evaluation scores. We constructan impression estimation model using Lasso regression for each of them. Next,(3) based on impression estimation models, we control the visual impressions
and synthesize textures. In (2), we constructed highly accurate visual impression
estimation models using style and content features. In (3), we obtained synthesis
results that match human intuition.
Keywords: Impression · Style · Content · Lasso Regression

1 Introduction
In product design, there is a growing interest in visual impressions. Visual impressions
refer to the impression evoked by the surface properties of materials and are considered
important in evaluating the quality and desirability of a product. In addition, the customization and personalization of products are becoming more common as the Internet
spreads and users need to diversify. One example is a custom made clothing service. In
this service, users can design clothes according to their tastes by selecting and combining fabrics, patterns, and shapes of clothes prepared in advance. However, developing© The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2023
I. Na and G. Irie (Eds.): IW-FCV 2023, CCIS 1857, pp. 1–15, 2023.
https://doi.org/10.1007/978-981-99-4914-4_9

2

Y. Sugiyama et al.

a system that supports users in creating their original designs is necessary to promotefurther personalization. Yet, creating original designs is difficult for users who do notknow design. This study proposes a method to automatically synthesize texture images
based on the user’s desired visual impressions information. These techniques will enable
design support based on human preferences, satisfaction, and other emotional values.

2 Previous Research
Research on texture analysis and synthesis has been conducted for many years, and various texture features have been proposed [1–3]. Gatys et al. proposed an image transformation algorithm focusing on style features and content features extracted from VGG-19[4], a convolutional neural network used for object recognition. The proposed method
produces images in which the style image’s style is transferred to the shape and structureof the objects depicted in the content image and shows highly accurate results. This study
suggests that style features retain more color and pattern information in the image, while
content features retain more shape information [5, 6].
In addition, although there are many studies that control for the aesthetics of texture
such as preference and emotion [7–11], there are few studies that control for impressions(aesthetic concepts) such as “flashy” and “slim.”

3 Proposed Method
In this study, we construct a model for estimating visual impressions using style andcontent features and synthesizing textures with the desired visual impressions’ control.
An overview of the proposed method is shown in Fig. 1. The first step is to extract styleand content features from the pretrained middle layer of VGG-19. Next, we construct a
visual impression evaluation model by formulating the relationship between the evaluation points assigned to the pattern image and the extracted style features and content
features, respectively, using Lasso regression. Finally, based on the constructed model,
style and content features are calculated and textures synthesized, which control visual
impressions.
This enabled us to construct the model while suppressing overlearning, and to formulate the relationship between texture and impression. Textures were generated basedon this relationship.

Texture Synthesis Based on Aesthetic Texture Perception Using CNN Style

3(a) Overview of feature extraction.

(b) Overall overview of the proposed method.

Fig. 1. Overview of the proposed method.

4 Quantification of Visual Impressions
4.1 Collection and Selection of Evaluation Terms
We collected and selected evaluation words related to the visual impressions evoked by
the patterns. For the experimental method, we conducted free description and goodnessof fit experiments based on the method of Tobitani [12]. Finally, a total of 10 evaluation words were selected for the subjective evaluation experiment: “cheerful,” “bright,”“colorful,” “complex,” “multilayered,” “cool-looking,” “free,” “cute,” “elegant,” and“sophisticated” [13].

4

Y. Sugiyama et al.

4.2 Subjective Evaluation Test
We conducted a subjective evaluation experiment to quantify the visual impressions
evoked by clothing patterns. Participants observed the stimuli presented on an LCD
monitor and rated the degree of each evaluation word on a 7point scale consisting of“strongly agree,” “agree,” “somewhat agree,” “neither agree nor disagree,” “somewhat
disagree,” “disagree,” and “strongly disagree.” The participants were undergraduate andgraduate students, male and female. We obtained rating data from 5 to 10 persons per
stimulus and per rating word, and scored each rating scale in 1-point increments, with -3
points for “strongly disagree” and 3 points for “strongly agree.” We defined the calculated
mean value based on the rating score (training data) for each stimulus and rating word
[14].

5 Extraction of Image Features
We extract image features to build a visual impression estimation model. To which evaluation points were assigned, image features were extracted from 1158 pattern images.
Style features are cross correlation matrices (Gram Matrix) of feature maps extracted
from the middle layer of VGG19. The content features are feature maps extracted
from the middle layer of VGG19. Style features are extracted from conv1_1, conv2_1,
conv3_1, conv4_1, and conv5_1 based on the work of Gatys [5, 15]. The feature dimensions are 64 × 64, 128 × 128, 256 × 256, 512 × 512, and 512 × 512, respectively.
Content features are extracted from Conv4_2. The number of feature dimensions is 28
× 28 × 512.

6 Construction of a Visual Impression Estimation Model
6.1 Construction of a Visual Impression Estimation Model
We formulate the relationship between visual impressions and style and content features.
Lasso regression is used in the formulation. Lasso regression is a penalized regression
model in which the L1 regularization term is used to construct a regression model while
preventing overlearning by setting the unselected variables to 0. Lasso regression ischosen (or used) because the explanatory variables are high dimensional, and excessive
learning is expected. The penalty parameter of Lasso regression is the value obtained
when K-split cross validation minimizes the mean squared error. K = 11 by Sturges’ rule.
Objective variables are the evaluation points, explanatory variables are style features andcontent features, and Lasso regression is used to construct visual impressions’ evaluation
models.
6.2 Results and Discussion
The determination coefficients for each model are shown in Tables 1 and 2. In the modelusing style features, the average coefficient of determination of the five models was morethan 0.5 for seven out of ten words, confirming that a highly accurate visual impressions

Texture Synthesis Based on Aesthetic Texture Perception Using CNN Style5

evaluation model could be constructed. In Table 2, the coefficient of determination was
0.5 or higher for 9 out of the 10 words. For the words with low coefficients of determination, “free,” “elegant,” and “refined,” the variation of evaluation scores is slight(Fig. 2). This means that the relationship between visual impressions and image characteristics cannot be modeled precisely. Therefore, in the following texture synthesis, wewill perform texture synthesis for the seven words, excluding these evaluation words.
Table 1. Coefficients of determination for impression estimation models.
(a) Coefficients of determination for impression estimation models constructed using stylefeatures.

evaluation term
cheerfulbrightcolorful
complexmultilayered
cool-looking
free
cute
elegant
sophisticatedconv1_1
0.582
0.711
0.330
0.229
0.167
0.699
0.172
0.372
0.229
0.138

conv2_1
0.699
0.784
0.565
0.530
0.488
0.775
0.386
0.550
0.317
0.198

conv3_1
0.628
0.760
0.608
0.543
0.570
0.776
0.408
0.501
0.393
0.212

conv4_1
0.694
0.801
0.695
0.623
0.673
0.809
0.487
0.568
0.460
0.305

conv5_1
0.648
0.695
0.603
0.642
0.661
0.716
0.332
0.549
0.411
0.393

average
0.650
0.750
0.560
0.513
0.512
0.755
0.357
0.508
0.362
0.249

(b) Coefficients of determination for impression estimation models constructed using content
features.

evaluation term
cheerfulbrightcolorful
complexmultilayered
cool-looking
free
cute
elegant
sophisticatedconv4_2
0.713
0.803
0.728
0.800
0.832
0.887
0.616
0.692
0.554
0.371

6

Y. Sugiyama et al.

Fig. 2. Distribution of evaluation points (vertical axis: number of images, horizontal axis:
evaluation points).

7 Texture Synthesis
7.1 Method
Based on the model constructed in Sect. 6, style features and content features with
controlled visual impressions are calculated, and textures are synthesized. The texture
synthesis process is completed in five different steps. (i) Extract style features and content
features from the input image. (ii) Exaggerate the extracted image features. Equation 1.
Does the control The extracted image features are denoted as P_original, the regression
coefficients obtained by building the model are transformed to fit the shape of the image
features as ωlasso and the weights are denoted as S. (iii) Style and content features are
extracted from the output images. (iv) Calculate the errors of style features and content
features from the features of input and output images. Hereafter, the error of the stylefeature is denoted as style loss (L_style) and that of the content feature as content loss
(L_content). (v) The sum of style loss plus weight α and content loss plus weight β isdenoted as L_total. Update the output image to minimize Eq. (2) L_total. Iterate (iii) to(v) up to 300 times.
Texture Synthesis Based on Aesthetic Texture Perception Using CNN Style

7

The image features are controlled by Eq. 1. By applying the weight parameter S to
the regression coefficients of the Lasso regression, the part of the image for which novariable is selected is kept at 0, and only the values for the part of the image strongly
related to the affective texture are changed.


P_controlled = P_original × (1 + ωlasso × S)

(1)

L_total = α × L_style + β × L_content(2)

As for the content features, since the total number of variables selected by this method
was small, we added variables by entering values 0.8 times the selected coefficient ofdetermination for variables with a high correlation (correlation coefficient of 0.8 orhigher) with the selected variables.
7.2 Results

Fig. 3. Synthesized results with varying weights α and β for the evaluation word “cheerful.”

8

Y. Sugiyama et al.

The results of synthesized images by varying weights α and β are shown in Fig. 3. The
input image size was set to 224 × 224, and S = 10 for the control of style features
and S =104 for the control of content features. These images are shown to change inaccordance with changes in the α and β weights.
Next, we observe the changes in the image when one of the values of α and β is
fixed. First, Fig. 4(a) shows the case where α = 106 and β = 1, and the style loss weights
increase. In the case of “cheerful,” the entire image is yellowish (Fig. 4(b)). In the caseof “bright,” the brightness seems to have increased, and in the case of “colorful,” thesaturation seems to have increased, respectively, as appropriate. The image in the “coollooking” group was tinted a bluish hue (Fig. 4(c)). In addition, “complex” emphasized theveins of leaves (Fig. 4(d)), and “multilayered” emphasized shadows and other elements
to give a visual impression of depth (Fig. 4(e)). In the “Cute” group, the brightness wasincreased while the saturation was decreased, giving the image a pastel tone.
On the other hand, Fig. 4(d) shows the case where the content loss weights are
increased to α = 1 and β = 106 . Changes in texture were observed in the “complex” and“multilayered” cases. In both cases, shading is emphasized, and light areas are especiallyemphasized in the “multilayered” case. However, for the evaluation terms in general, it
was found that the images with larger content loss weights showed more minor changes
than those with larger style loss weights.
In the next section, we examine the validity of the generated images. In this section,
we control only the style features found to be particularly effective and generate textures.

(a) Synthesized results for a=1, ß=10^6.

Fig. 4. Texture synthesis results.

Texture Synthesis Based on Aesthetic Texture Perception Using CNN Style(b) Synthesized results for “cheerful.”(d) Synthesized results for “complex.”

9(c) Synthesized results for “cool-looking.”(e) Synthesized results for “multilayered.”(f) Synthesized results for a=1, ß=10^6.
Fig. 4. (continued)

8 Verification
This section quantitatively verifies whether the emotional quality evoked by the synthesized images is significantly improved compared to the original images. In this section,
only the control of style features, which showed appropriate changes in the previous

10

Y. Sugiyama et al.

section, is subject to verification, and parameters of α = 106 , β = 1, and S = 10 are
employed.
8.1 Construction of Experimental Dataset
First, we constructed a dataset for the effectiveness experiment. Stimuli were selectedfrom 2878 unknown images in the dataset. We first estimated (i) the visual impressions of
the source images. Using the model constructed in Sect. 6, we calculated the evaluation
score for each pattern by inputting the style features extracted from the original images.
Next, (ii) patterns with high/medium/low evaluation points in common for all words (7words) were extracted. The patterns were arranged in the order of the highest score foreach word and divided into three groups: high, medium, and low. Then, the patterns in thehigh, medium, and low rank groups for all the evaluation terms were extracted, resulting
in 51, 7, and 14 patterns in this order, respectively. Finally, we selected patterns thatsatisfied (iii) “stability of synthesis” and “visibility of exaggeration.” We synthesized
35 images (5 times for 7 words = 35 images) and selected 10 patterns as stimuli thatsatisfied each criterion shown in Table 2.
Table 2. Criteria for “stability of synthesis” and “visibility of exaggeration.”
Stability of synthesis

・No image is blacked out
・The same quality is produced at least 4 out of 5 times for all
words
・The structure of the pattern is established

Visibility of exaggeration

・The change is easy to see compared to the original image

8.2 Effectiveness Verification Experiment
Next, we conducted an effect verification experiment. Participant participants were asked
to observe the stimulus pairs presented on an LCD monitor and to answer which of
the four evaluation words was true for each word using a fourtrial scale consisting of“left,” “more or less left,” “more or less right,” and “right.” The total number of trials
per participant was 280, using the experimental dataset constructed in Sect. 8.1. The
participants were 10 undergraduate and graduate students (5 males and 5 females, aged
23.3 ± 1.19 years). To eliminate the influence of the order effect, the order in which thestimulus pairs were presented was randomized for each participant, and the order of theevaluation words was randomized for each trial.
8.3 Results and Discussion
The validity of this method was verified by conducting a statistical analysis of the dataobtained in Sect. 8.2 and obtaining the psychological scale values. Multiple comparisonswere conducted using the yardstick method to evaluate whether there was a statisticallyTexture Synthesis Based on Aesthetic Texture Perception Using CNN Style11significant difference between each stimulus. As a result, we confirmed that the psychometric values of the synthesized images with exaggerated visual impressions qualities
were significantly higher than those of the original images in the proportions shown in
Table 3. The changes in the psychological scale values are shown in Fig. 5.
Table 3. Percentage of patterns with significantly increased psychological scale values.
Exaggerated visual
impressions

p <. 01(**)

p <. 05(*)

cheerful

0. 7

0. 8

bright0. 7

0. 8

colorful

0. 1

0. 1

complex

0. 3

0. 4

multilayered

0. 3

0. 4

cool-looking

0. 9

0. 9

cute

0. 0

0. 0

The three words “cheerful,” “bright,” and “cool-looking” significantly increased thepsychometric scale values for most patterns. Since these are low order visual impressions
qualities perceived from color information, they varied regardless of the pattern’s taste.
The psychometric values of “complex” and “multilayered” patterns increased significantly in about half of the patterns. The patterns that showed a significant increase were
those with delicate patterns, and the lines became thicker and more three dimensional
according to the patterns. On the other hand, the patterns with larger scales did not showsuch changes, which may explain the non-significance of the results.
Next, the psychological scale value of “colorful” increased for most patterns but was
non-significant for all of them. One of the reasons for this is that the degree of change
was smaller than the other words. At the present stage, the weight parameter in Eq. 7.4
is unified as S = 10 for all words, but for “colorful,” we confirmed that the degree ofchange approaches the other words by setting a more significant value of S (Fig. 6).
Therefore, further study is needed to adjust the parameters.
Additionally, none of the patterns significantly increased in the “cute” category, andhalf of the patterns significantly decreased in the “cute” category. One of the possible
reasons for this is the influence of the original images. It is assumed that patterns withlow saturation in the original images are faced with a decrease in saturation, and thebalance of the color scheme perceived as “cute” is lost. Furthermore, since “cute” is ahigher order sensory quality consisting of various elements, it is also considered affectedby individual differences. Therefore, we conducted a factor analysis of each participant’s
psychological scale of “cute” for each image. Due to the factor analysis, three factors
were extracted, with a cumulative contribution rate of 60.2%. Table 4 shows the factor
loadings matrix after rotation, and Table 5 shows the factor correlation matrix.

12

Y. Sugiyama et al.

Fig. 5. Change in psychological scale values.

Texture Synthesis Based on Aesthetic Texture Perception Using CNN Style

13(b) Synthesized image for =10 (c) Synthesized image for =20(a) Original

Fig. 6. Comparison of different values of the magnification parameter in “Colorful.”

Table 4. Factor loadings matrix after rotation.
participant

factor
F1

F2

F3

No. 1

1.178

−0.322

0.013

No. 9

0.704

0.240

−0.168

No. 7

0.550

0.285

0.031

No. 8

0.502

0.326

0.022

No. 10

0.462

0.061

0.403

No. 5

0.034

0.784

0.075

No. 6

0.175

0.655

−0.086No. 2−0.159

0.536

0.073

No. 4

−0.062−0.036

0.853

No. 3

−0.013

0.155

0.542

Table 5. Factor correlation matrix.
factor

F1

F2

F3

F1

1

0. 665

0. 494F2

0. 665

1

0. 466F3

0. 494

0. 4661Comparing the factor scores for each image revealed that participants had different
evaluation tendencies with each factor. Participants with the “F2” factor tended to rate“cute” highly for the synthesized images with “cheerful” and “bright” exaggerated.
Participants with the “F3” factor tended to rate the original and synthesized images withthe exaggerated “colorful” highly. Participants with the “F1” factor tended to rate the14

Y. Sugiyama et al.

original image, and the “cool-looking” exaggerated image lower than those with the“F2” factor and the “F3” factor. This suggests that the model should be expanded toconsider future evaluation tendencies differences among individuals.

9 Conclusion
In this study, we proposed a method for synthesizing texture images of clothing patterns with desired visual impressions. First, (1) subjective evaluation experiments wereconducted on pattern images to quantify the visual impression. We obtained evaluation
scores for 10 words that express the visual impressions for the image dataset collectedfrom floral patterns. Next, (2) style and content features were extracted from the patternimages used in the subjective evaluation experiment using the pretrained VGG19. Then,
we constructed a visual impressions evaluation model by formulating the relationshipbetween the quantified visual impressions, the extracted style features, and the content
features using regression. As a result, we could model visual impressions with highaccuracy while selecting features that are mainly strongly related to visual impressions.
Finally, (3) based on the obtained model, we calculated the image features when thedesired visual impressions quality is exaggerated and synthesized images by optimizing
the model to minimize the error between the features and the original images. (4) To
verify the validity of the proposed method, we synthesized unknown images with thedesired exaggerated visual impressions. It was found that the changes in the images synthesized using the content features were smaller than those synthesized using the stylefeatures. In addition, the experiment demonstrated the method’s effectiveness in which
the emotional quality evoked by the synthesized images was significantly improved
compared to the original images.
As future research topics, we will extend the model to a higher order visual impression
consisting of various elements, such as “cute,” to consider individual differences. In
addition, we will quantitatively verify the degree to which the degree of exaggerationof the visual impressions quality changes by adjusting the weight parameters set whenchanging the style features according to the taste of the words and patterns.

References
1. Julesz, B.: Textons, the elements of texture perception, and their interactions. Nature
290(5802), 91–97 (1981)
2. Portilla, J., Simoncelli, E.P.: A parametric texture model based on joint statistics of complex
wavelet coefficients. Int. J. Comput. Vision 40(1), 49–70 (2000)
3. Tobitani, K., Shiraiwa, A., Katahira, K., Nagata, N., Nikata, K., Arakawa, K.: Modeling of“high-class feeling” on a cosmetic package design. J. Jpn. Soc. Precis. Eng. 87(1), 134–139(2021)
4. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image
recognition (2014). arXiv preprint arXiv:1409.1556
5. Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition(CVPR), pp. 2414–2423 (2016)

Texture Synthesis Based on Aesthetic Texture Perception Using CNN Style

15

6. Wang, P., Li, Y., Vasconcelos, N.: Rethinking and improving the robustness of image styletransfer. In: Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR), pp. 124–133. IEEE, Nashville (2021)
7. Yu, N., Barnes, C., Shechtman, E., Amirghodsi, S., Lukac, M.: Texture mixer: a network
for controllable synthesis and interpolation of texture. In: Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12164–12173 (2019)
8. Li, Y., Fang, C., Yang, J., Wang, Z., Lu, X., Yang, M.: Diversified texture synthesis withfeed-forward networks. In: Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition (CVPR), pp. 3920–3928 (2017)
9. Yang, S., Wang, Z., Wang, Z., Xu, N., Liu, J., Guo, Z.: Controllable artistic text style transfer
via shape-matching GAN. In: Proceedings of the IEEE/CVF International Conference on
Computer Vision (ICCV), pp. 4442–4451 (2019)
10. Chen, H., et al.: DualAST: dual style-learning networks for artistic style transfer. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
pp. 872–881 (2021)
11. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition (CVPR), pp. 10684–10695 (2022)
12. Tobitani, K., Matsumoto, T., Tani, Y., Fujii, H., Nagata, N.: Modeling of the relation between
impression and physical characteristics on representation of skin surface quality. J. Inst. Image
Inf. Telev. Eng. 71(11), 259–268 (2017)
13. Mori, T., Uchida, Y., Komiyama, J.: Relationship between visual impressions and image
information parameters of color textures. J. Jpn. Res. Assoc. Text. End-uses 51(5), 433–440(2010)
14. Sunda, N., Tobitani, K., Tani, I., Tani, Y., Nagata, N., Morita, N.: Impression estimation model
for clothing patterns using neural style features. In: Proceedings of the Springer International
Conference on Human-Computer Interaction, pp. 689–697 (2020)
15. Takemoto, A., Tobitani, K., Tani, Y., Fujiwara, T., Yamazaki, Y., Nagata, N.: Texture synthesis
with desired visual impressions using deep correlation feature. In: 2019 IEEE International
Conference on Consumer Electronics (ICCE), pp. 1–2. IEEE, Las Vegas (2019)