Impression Estimation Model for Clothing
Patterns Using Neural Style Features
Natsuki Sunda1 , Kensuke Tobitani1 , Iori Tani1 , Yusuke Tani1 ,Noriko Nagata1(B) , and Nobufumi Morita2
1Kwansei Gakuin University, 2-1, Gakuen, Sanda, Hyogo, Japannagata@kwansei.ac.jp
2
Couture Digital Ltd, Sankyo Sakaisujihonmachi Building F9 1-8-2, Bakurou-machi,Chuo-ku, Osaka, Osaka, Japan

Abstract. In product design, the impressions evoked by surface properties of objects attract attention. These impressions are called aﬀective
texture and demand is growing for technologies that quantify, index,and model it. Even in the fashion industry, the diversiﬁcation of userneeds necessitates the customization and personalization of products.
Consequently, there has been a focus on custom-made services. However,enormous amounts of time and human costs are needed to ﬁnd clothing
patterns to suit one’s own preferences and ideas from among the countless patterns available. This research focused on aﬀective texture related
to visual impressions, and we proposed a method for automatically estimating the aﬀective texture evoked by clothing patterns. To this end,
we conducted the following steps: (1) quantiﬁed the visual impressions
for patterns; (2) extracted style features as physical characteristics; and(3) modeled the relationships between visual impressions and physicalcharacteristics. Afterward, based on the obtained models, we estimatedthe impressions for unlabeled patterns. Then, we veriﬁed their validity
through relative evaluation and absolute evaluation, and we conﬁrmedthat our models estimated the impressions corresponded to the impressions that people actually felt. In addition, we implemented a system toenable users to intuitively search for patterns.
Keywords: Fashion · Texture · CNN · Lasso regression

1Introduction

In product design, not only designs, such as colors and shapes, but also theimpressions (e.g., “luxurious” or “antique”) evoked by surface properties ofobjects attract attention. These impressions are called aﬀective texture, which
is an important factor in preference for or the quality judgment of an object.
Therefore, demand is growing for technologies that quantify, index, and modelaﬀective texture.
Even in the fashion industry, the diversiﬁcation of user needs necessitates thecustomization and personalization of products. Consequently, there has been ac Springer Nature Switzerland AG 2020
C. Stephanidis and M. Antona (Eds.): HCII 2020, CCIS 1226, pp. 689–697, 2020.
https://doi.org/10.1007/978-3-030-50732-9_88

690

N. Sunda et al.

focus on services that allow customers to order custom-made clothing on theInternet. However, enormous amounts of time and human costs are needed todesign unique clothing. For example, it is diﬃcult to ﬁnd clothing patterns tosuit one’s own preferences and ideas from among the countless patterns available.
This research focuses on aﬀective texture related to visual impressions, andwe propose a method for automatically estimating the aﬀective texture evoked byclothing patterns. To this end, we model the relationships between visual impressions and physical characteristics. In addition, based on the obtained models, weimplement a system to enable users to intuitively search for patterns.

2Previous Research

Research on texture analysis has been conducted for a long time, and various
texture features have been proposed [1,2]. In recent years, Gatys et al. [3] hasbeen proposed an algorithm for transferring an image style using VGG-19 [4],which is a convolutional neural network used for object recognition. In theirresearch, it is suggested that content features are necessary for object recognition and that style features represent the detailed appearance in images, suchas multi-scale color information and the pattern information, and are not asimportant for object recognition.
Thereby, Takemoto et al. thought that style features were strongly related
to aﬀective texture based on unique texture properties (uniformity, density,coarseness, roughness, regularity, linearity, directionality, direction, frequency,and phase [5]). Thus, they proposed a texture synthesis method with the desiredaﬀective texture using style features and showed high-precision results [6].
In this research, we think that clothing patterns are also part of texture.
Therefore, based on Takemoto et al., we use style features as physical characteristics representing patterns and clarify the relationships with the visual
impressions they evoke.

3

Proposed Method

In this research, we propose a method for automatically estimating the aﬀective texture evoked by clothing patterns. Figure 1 shows an overview of ourapproach. First, we conducted a subjective evaluation experiment and a factor
analysis, and quantiﬁed the visual impressions of a small number of patterns.
Next, we extracted style features as physical characteristics using a VGG-19
network. Afterward, we conducted a lasso regression and constructed impression
estimation models. Finally, using our models, we estimated the impressions fora large number of unlabeled patterns.

4

Quantification of Visual Impressions

4.1

Subjective Evaluation Experiment

We conducted a subjective evaluation experiment in order to quantify the visualimpressions evoked by patterns.

Impression Estimation Model for Clothing Patterns

691

Fig. 1. Overview of our proposed method

First, it was necessary to gather and select comprehensive and representative words for evaluating visual impressions. Based on Tobitani et al. [7], we
conducted a free writing experiment using 10 pattern images. Then, we veriﬁed
whether the obtained words were suitable for visual impressions. As a result,
we selected 28 adjectives with high ﬁtness. We also added 12 adjectives that
previous research used for clothing materials and the feel of materials [8–10] and“cute” from the viewpoint of actual use. Finally, we used 41 adjectives as theevaluation words in this experiment.
Next, we speciﬁcally focused on ﬂower patterns and collected 1,158 such
images. Then, in order to select comprehensive and representative stimuli, weconducted clustering based on style features extracted from the images. Basedon the divided clusters, we selected 75 images as the stimuli. The extraction ofstyle features is described in Sect. 5.1.
A total of 40 graduate and undergraduate students (18 men; 22 women;
average age: 21.9 ± 2.8 years) participated in this experiment. The participants
observed stimuli shown on an LCD monitor and evaluated their impressions
on a seven-point Likert scale. In order to reduce the burden on them, we hadthem use 10 or 11 out of the 41 evaluation words. To take the order eﬀect
into consideration, the order of the evaluation words was randomized for eachstimulus, and the order of the stimuli was randomized for each participant.
Through this experiment, we obtained data for 10 participants per stimulus and per evaluation word. Then, we scored each scale from −3 to 3 in onepoint increments. For each evaluation word, we calculated the average score, anddeﬁned it as the evaluation score for the impression of each stimulus.
4.2

Factor Analysis

In order to extract and quantify latent factors that contributed to the impressions evoked by the patterns, we conducted a factor analysis of the evaluation
scores using the maximum likelihood method and promax rotation. As a result,six factors were extracted, and the cumulative contribution ratio was 81.1%

692

N. Sunda et al.

(Table 1). We interpreted each factor as follows: Factor 1 was “pop”; Factor 2
was “elaborate”; Factor 3 was “refreshing”; Factor 4 was “novel”; Factor 5 was“tidy”; and Factor 6 was “stylish”.
Table 1. Factor analysis results

Finally, we deﬁned the evaluation scores obtained from the subjective evaluation experiment and the factor scores obtained from the factor analysis as thequantiﬁed impressions (impression values) for the patterns.Impression Estimation Model for Clothing Patterns

5

Modeling the Relationships Between Visual
Impressions and Physical Characteristics

5.1

Extraction of Style Features

693

We used the style features proposed by Gatys et al. [3] as physical characteristics
representing patterns. The style features were Gram matrices of feature maps
output from hidden layers of a pretrained VGG-19 network. Since they represent
the detailed appearance in images, such as multi-scale color information andpattern information, they can be considered to be strongly related to visual
impressions for clothing patterns. We extracted style features from pooling layers
1, 2, 3, and 4 of VGG-19 using the images labeled in Sect. 4. Their dimensions
were 64 × 64, 128 × 128, 256 × 256, and 512 × 512, respectively.
5.2

Lasso Regression

We considered a regression problem to model the relationships between visual
impressions and physical characteristics. In this research, over-ﬁtting wasexpected since the style features were high-dimensional features with respect tothe number of labeled patterns. Therefore, we adopted a lasso regression basedon Takemoto et al. [6]. In a lasso regression, signiﬁcant parts are selected fromindependent variables, and regression models are constructed, while over-ﬁtting
is prevented.
We conducted a lasso regression by setting the impression values as thedependent variables and the style features extracted from each pooling layer
as the independent variables. Then, we used the penalty parameters obtained
when mean square errors were minimized during K-fold cross validation (K =7). For each factor and adjective, we constructed a regression model for eachpooling layer and adopted the model with the greatest coeﬃcient of determination among the four models as the impression estimation model. The impression
estimation models were as follows: “bright,” “breezy,” and “cool-looking” werepooling layer 1; all six factors were pooling layer 2; “rustic” and “cool” werepooling layer 3; and the other 36 adjectives were pooling layer 4. The averagecoeﬃcient of determination for all of the impression estimation models was 0.84
for six factors and 0.70 for 41 adjectives, and we conﬁrmed that our models hadhigh precision.

6

Validity of the Proposed Method

6.1

Impression Estimation for Unlabeled Patterns

Based on the models constructed in Sect. 5.2, we estimated the impressions forthe remaining 1,083 unlabeled images. First, we extracted style features fromthe pooling layer adopted as each impression estimation model used them. After
that, we input style features into the models and had the models calculate
impression values for each pattern. In addition to the right side of Fig. 1, Fig. 2
shows the top 20 patterns with high calculated impression values for several of
the factors and adjectives.

694

N. Sunda et al.

Fig. 2. Examples of the estimated results

6.2

Verification Experiment by Relative Evaluation

In order to conﬁrm whether the results estimated in Sect. 6.1 corresponded toimpressions that people actually felt, we ﬁrst conducted a pairwise comparison experiment using Thurstone’s paired comparison. Then, we psychologicallyscaled the actual impressions and obtained their ordinal structure. Finally, wecompared it with that of the estimated impressions.
Among the evaluation words, after considering the number of adjectives
included in each factor and their factor loadings, we selected the following 10adjectives: “cheerful,” “bright,” and “colorful” in the “pop” factor; “complex”
and “multilayered” in the “elaborate” factor; “cool-looking” in the “refreshing”
factor; “free” in the “novel” factor; “cute” and “elegant” in the “tidy” factor;and “sophisticated” in the “stylish” factor.
Then, we selected the top two, the two around zero, and the bottom two of
the calculated impression values for each evaluation word, and we used a totalof 60 images as the stimuli in this experiment.
A total of 10 graduate and undergraduate students (5 men; 5 women; averageage: 22.3 ± 1.1 years) participated in this experiment. The participants observed
two stimuli shown on an LCD monitor and evaluated which one was applicablefor each evaluation word. They used six images per evaluation word, and eachparticipant conducted 150 trials. To take the order eﬀect into consideration,
the order of the stimulus pairs and evaluation words was randomized for eachparticipant.
Figure 3 shows the psychological scale value of each pattern for each evaluation word. For “elegant,” “bright,” and “cool-looking,” we conﬁrmed that
our models accurately estimated the ordinal structure, including for patterngroups whose calculated impression values were around zero. In addition, statistically signiﬁcant diﬀerences existed between groups with high and low calculated
impression values in seven out of 10 words.
6.3

Verification Experiment by Absolute Evaluation

We conducted a subjective evaluation experiment with the unlabeled images
used in Sect. 6.1, and had people actually give them impression values. AfterImpression Estimation Model for Clothing Patterns

695Fig. 3. Pairwise comparison results

that, we calculated the correlation coeﬃcients between the impression values
given in this experiment and the impression values calculated in Sect. 6.1, andthen veriﬁed the validity of our models.
A total of 25 graduate and undergraduate students (15 men; 10 women;
average age: 22.7 ± 3.3 years) participated in this experiment. The participantsevaluated in the same procedure as in Sect. 4.1 with the evaluation words usedin Sect. 6.2. However, in order to reduce the burden on them, we had them use
one of the ﬁve stimuli sets into which the 1,083 images were divided.
Through this experiment, we obtained data for ﬁve participants to per stimulus and per evaluation word. Then, we obtained impression values in the sameprocedure as in Sect. 4.
Table 2 shows the correlation coeﬃcients between the given impression values and the calculated impression values. There were extremely strong positive
correlations for “cool-looking” and “bright,” as well as positive correlations for“colorful,” “cheerful,” “complex,” “cute,” and “multilayered.” “Free,” “elegant,”and “sophisticated” had weak positive correlations.
Table 2. Correlation coeﬃcients results

6.4

Discussion

From the results in Sect. 6.2 and Sect. 6.3, we conﬁrmed that our models had highgeneralizability. In particular, models such as the “bright” and “cool-looking”
models had high precision. On the other hand, models such as the “free” and“sophisticated” models had lower estimation precision than the other adjectives.
We considered individual diﬀerences in evaluations caused by ambiguities
of words. Therefore, for each evaluation obtained in Sect. 4.1 and Sect. 6.3, wecalculated the standard error between the participants for each stimulus andcalculated its average between stimuli (Table 3). “Bright,” “colorful,” and “coollooking” had a smaller average standard error than the other adjectives did.
This suggests that these simple impressions with high estimation precision are696

N. Sunda et al.

less aﬀected by individual diﬀerences. On the other hand, “free,” “elegant,”and “sophisticated” had a larger average standard error was larger than theother adjectives did. This suggests that these complex impressions involving
multiple factors with low estimation precision are greatly aﬀected by individual
diﬀerences. Based on these observations, for complex impressions, it is necessary
to extend the models so that they not only use the average of the given scores
but also consider the dispersion of individual evaluations.
Table 3. Standard error results

6.5

Implementation a Pattern Search System

Since we conﬁrmed the validity of our proposed method, we implemented apattern search system using the estimated results. This system enables users tointuitively search for patterns with the words used in Sect. 4 as queries. It wasapplied to the fashion-on-demand app COUTURE made by digital fashion ltd.
and is already being used in practical application (Fig. 4).

Fig. 4. The app COUTURE

7Conclusion

We have proposed a method for automatically estimating the aﬀective texture
evoked by clothing patterns and conducted the following steps: (1) quantiﬁed theImpression Estimation Model for Clothing Patterns

697

visual impressions of a small number of patterns with a subjective evaluation
experiment and a factor analysis; (2) extracted style features as physical characteristics using a VGG-19 network; and (3) modeled the relationships between
visual impressions and physical characteristics with a lasso regression. Afterward, based on the obtained models, we estimated the impressions for a large
number of unlabeled patterns. Then, we veriﬁed their validity through relativeevaluation and absolute evaluation, and we conﬁrmed that our models had highgeneralizability. In addition, we implemented a system to enable users to intuitively search for patterns.
In the future, we will extend our models to consider individual diﬀerences
and correspond with general patterns and shapes.

References
1. Julesz, B.: Textons, the elements of texture perception, and their interactions.
Nature 290(5802), 91 (1981)
2. Portilla, J., Simoncelli, E.P.: A parametric texture model based on joint statistics
of complex wavelet coeﬃcients. Int. J. Comput. Vis. 40(1), 49–70 (2000). https://doi.org/10.1023/A:1026553619983
3. Gatys, L.A., Ecker, A.S., Bethge, M: Image Style Transfer Using Convolutional
Neural Networks. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2414–2423 (2016)
4. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale
image recognition (2014). arXiv preprint arXiv:1409.1556
5. Tuceryan, M.A., Jain, A.K.: Handbook of Pattern Recognition and ComputerVision. Texture Analysis, pp. 235–276. World Scientiﬁc (1993)
6. Takemoto, A., Tobitani, K., Tani, Y., Fujiwara, T., Yamazaki, Y., Nagata, N.:
Texture synthesis with desired Visual impressions using deep correlation feature.
In: IEEE International Conference on Consumer Electronics, pp. 739–740 (2019)
7. Tobitani, K., Matsumoto, T., Tani, Y., Fujii, H., Nagata, N.: Modeling of therelation between impression and physical characteristics on representation of skinsurface quality. J. Inst. Image Inf. Telev. Eng. 71(11), 259–268 (2017)
8. Doizaki, R., Iiba, S., Okatani, T., Sakamoto, M.: Possibility to use product imageand review text based on the association between onomatopoeia and texture. Trans.
Jpn. Soc. Artif. Intell. 30(1), 57–60 (2015)
9. Mori, T., Uchida, Y., Komiyama, J.: Relationship between visual impressions andimage information parameters of color textures. J. Jpn. Res. Assoc. Text. end-uses
51(5), 433–440 (2010)
10. Mouri, C., Ueda, E.S., Terauchi, F., Aoki, H.: Relationship between Mimetic words
and kansei and sensory characteristics. Bull. Jpn. Soc. Sci. Des. 58, 209 (2011)