Using Deep Learning to Estimate User
Impressions of Designs for 3DFabrication
Koichi Taguchi, Manabu Hashimoto, Kensuke Tobitani and Noriko Nagata

Abstract This paper proposes a method for applying three typical human impressions directly into product designs to fabricate products using 3D printers. The
method automatically estimates human impressions of the three-dimensional shapeof an item in terms of three representative sensibilities: “hard–soft,” “ﬂashy–sober,”and “stable–volatile.” This technique can be used for new 3D fabrication processes
that reﬂect the designer’s intentions directly into the shapes of products. To estimate the impressions of the shape of an object, we need to draw strong correlations
between impressions, which are psychological factors, and the aspects of the shapeof the object, which are physical factors. The method uses deep learning effectively
to address this issue. The object being evaluated is ﬁrst converted to a set of imagesby photographing it from 20 surrounding directions. This image set is used as input
data for deep learning with parameters of human impressions of the object as supervisory signals. In experiments, we used original dataset of three-dimensional objectsof a car with assigned impressions that had been quantiﬁed using the semantic differential (SD) method. The correlation coefﬁcients between impressions estimatedusing this method and the supervisory signals for all the datasets were about 0.70 for“hard–soft,” about 0.61 for “ﬂashy–sober,” and about 0.67 for “stable–unstable.”K. Taguchi (B) · M. Hashimoto
Chukyo University, 101-2 Yagoto Honmachi, Showa-ku, Nagoya-shi 466-8666, Aichi, Japane-mail: taguchi@isl.sist.chukyo-u.ac.jp
M. Hashimotoe-mail: mana@isl.sist.chukyo-u.ac.jp
K. Tobitani · N. Nagata
Kwansei Gakuin University, 2-1 Gakuen, Sanda-shi 669-1337, Hyogo, Japane-mail: tobitani@kwansei.ac.jp
N. Nagatae-mail: nagata@kwansei.ac.jp© Springer Nature Singapore Pte Ltd. 2019
A. Martínez-García et al. (eds.), Progress in Optomechatronic Technologies,
Springer Proceedings in Physics 233, https://doi.org/10.1007/978-981-32-9632-9_8

65

66

K. Taguchi et al.

1 Introduction
Various kinds of three-dimensional printers have been developed, and their cost has
decreased. Thus, we can now use them not only in factories but also at home. Furthermore, a useful database of three-dimensional models has been prepared, andeveryone can utilize it via the Internet. “Personal fabrication (production using
three-dimensional printers by individuals)” will likely change the traditional wayof manufacturing by mass production at factories. Enabling “personal fabrication”
will require skillful modeling techniques using CAD systems to make the desired
three-dimensional objects. However, not everyone has such skills. Therefore, we propose a method for using impression factors to provide support in modeling complex
three-dimensional objects. To design a shape of models based on impressions, weneed to associate the impressions and shape of three-dimensional models. As such,
we propose a method for estimating human impressions of an object to be used insupporting such modeling technologies like an expert of CAD.
Normally, we would consider that impressions of an object would be determined
holistically based on shape, color, and material, but for this research, we have assumed
that the shape as a dominant factor in determining the impressions of objects basedon a method from Tobitani et al. [1]. We also deﬁned three types of impressions ofobjects as particularly important: “hard–soft,” “ﬂashy–sober,” and “stable–unstable.”
Taguchi et al. [2] calculated the relationship between features and impressions
using multiple regression analysis. However, it can be used only for the “hard–soft”
impression factor because the features have been optimized for a speciﬁc impression.
Designing which features and classiﬁers to optimize for various impressions is difﬁcult. In this research, we propose a method to estimate impressions of “hard–soft,”“ﬂashy–sober,” and “stable–unstable” automatically using a deep neural network(DNN) for only the shape of a three-dimensional model. Two types of convolutional
network (CNN) are used currently in the ﬁeld of object classiﬁcation. One type ismulti-view architectures, while the other is volumetric architectures. Qi et al. [3] analyzed the challenges of object classiﬁcation on three-dimensional data using these
two types of CNNs. As a result, the multi-view CNN architectures were proven to
be more accurate than volumetric in object recognition. In addition, a person canonly recognize an object from a certain viewpoint, one at a time. Therefore, we use
a multi-view CNN architecture to estimate impression factors in this method.
The remainder of this paper is organized as follows. The quantiﬁcation of impression factors to the objects is described in Sect. 2. The proposed method to estimate
the impression factors is presented in Sect. 3. Experimental results and analysis areprovided in Sect. 4. Finally, the conclusion and discussion are given in Sect. 5.

Using Deep Learning to Estimate User Impressions …
Table 1 Various bipolar
adjectives (impressionfactors)

67

Bipolar adjectives (impression factors)
ordered–chaotic

connected–disconnected

stable–unstable

dynamic–static

active–passive

healthy–unhealthy

excitable–calm

relaxed–tensesoft–hardsmooth–roughdistinct–vague

weak–strong

blunt–sharp

intense–milddelicate–rugged

cheerful–cheerlessﬂashy–sober

heavy–light

2 Quantiﬁcation of Impression Factors
The relationship between impression factors and three-dimensional shape needs tobe clariﬁed in order to enable intuitive manipulation of three-dimensional shapes
using an impression. Tobitani et al. [1] deﬁned the rating scale for three-dimensional
shapes using the semantic differential method to quantify sensitive bipolar adjectives(impression factors). The average values of 10 people were calculated in the SD
method for every 18 bipolar adjectives. The 18 bipolar adjectives for this experimentare shown in Table 1.
In this experiment, conditions such as background, materials, lighting, and so on
were kept constant, and the range of impression coefﬁcients was −3.0 to 3.0. We
analyzed using major factor analysis (PFA) and Varimax rotation. According to theliterature, the impression coefﬁcients of the three-dimensional shape can be expressedby three bipolar shapes: “hard-soft,” “ﬂashy-saw,” and “stable-unstable.” The experimental dataset was abstract objects manufactured by a professional designer. Whengenerating these three-dimensional shapes, the bipolar adjectives mentioned above
were presented as guidelines for shape design. Therefore, we focused on these mostrelevant impression factors for the three-dimensional shapes.

3 Impression Estimation Method
The ﬂow of our method is shown in Fig. 1.
Our method uses a deep learning convolutional network with multi-viewpoint
images to estimate human impressions of three-dimensional object. This method
includes both the learning and estimation modules. The method is assumed to havesupervised learning, so the three-dimensional models and ground-truth (impressionfactors) need to be made to correspond. In the learning module, we used the semantic
differential (SD) method [4] to estimate values for the impressions of each object,and we used these values as supervisory signals. Incidentally, they are based on the68

K. Taguchi et al.

VRIWQHVV'HHS
1HXUDO1HWZRUNIODVKLQHVV

VWDEOHQHVVPXOWLYLHZLPDJHV&RQYHUVLRQRI'PRGHOVWRPXOWLYLHZLPDJHV(VWLPDWLRQRIKXPDQLPSUHVVLRQVE\GHHSQHXUDOQHWZRUN

Fig. 1 Flow of our method for estimating impression factors

contents of Sect. 2. The input data for the network were the multiple images taken
across viewpoints. Figure 2 shows an example of virtual viewpoints set around athree-dimensional object.
The multi-view images represent three-dimensional shapes using multiple viewsof three-dimensional objects that were generated by rendering. The virtual viewpointsof plurality were installed around the objects. In this study, the input data consisted ofa set of images of the objects taken from 20 directions spaced equally around them.
Therefore, the input data were grayscale images taken from 20 viewpoints. Here, anarea in which a three-dimensional model did not exist was deﬁned as the backgroundand supplemented with a numerical value. The network consisting of impression
estimations was achieved using this image set as the input for deep learning and thetraining to minimize the difference between the estimated results and the supervisory
signals. However, in our method, there is a scale invariance due to transformation of9LHZSRLQWV YHUWH[RIUHJXODU GRGHFDKHGURQUHQGHUHG PXOWLYLHZ LPDJHV

Fig. 2 Virtual viewpoints set around a three-dimensional object and the rendered images

Using Deep Learning to Estimate User Impressions …

69&11

³VRIWQHVV´
9LHZ
SRROLQJ&11&11

³IODVKLQHVV´³VWDEOHQHVV´&11YLHZSRLQWV(VWLPDWLRQUHVXOWVRILPSUHVVLRQV0XOWLYLHZLPDJHV

Fig. 3 Constitution of the network

three-dimensional objects to multi-view images. Therefore, we uniﬁed the size of thethree-dimensional object presented to the subject when quantifying the impression ofa person with respect to a three-dimensional object by SD method [4]. The network
is shown in Fig. 3.
Our method consists of two stages: the ﬁrst stage is with ﬁve convolutional layers
and three pooling layers and the second stage is with a view-pooling layer, which
integrated the multi-viewpoint image, and three fully connected layers. The error
function used was softmax, and the optimization method used was Adam [5].

4 Experimental Results of Impression Estimations
In our experiments, we used the SD method [3] to associate supervisory signals
with three-dimensional objects. For the objects, we used ModelNet40 [6] in speciﬁcclasses: a car, vase, and chair. Each three-dimensional object was evaluated by 20–40
people, and the −3 to 3 was used for the supervisory signals. In the experiments,
the proposed method was evaluated using correlation coefﬁcients. The experimental
datasets were evaluated using 20 three-dimensional objects randomly. Figure 4 shows
two distributions of relationships between the estimated results and the supervisory
signals. The blue dots represent the results of the previous method, and the red dots&DU'DWDVHWV(VWLPDWLRQUHVXOWV(VWLPDWLRQUHVXOWV(VWLPDWLRQUHVXOWVVXSHUYLVRU\VLJQDOVVXSHUYLVRU\VLJQDOVVXSHUYLVRU\VLJQDOV

VRIWٕKDUGIODVK\ٕVREHU

Fig. 4 Relationships between the estimated results and supervisory signals

VWDEOHٕXQVWDEOH70
Table 2 Correlation
coefﬁcients between
estimation results andsupervisory signals

K. Taguchi et al.
Datasets
Car

Correlation coefﬁcients for impressions
Hard-soft

Sober-ﬂashy

Unstable-stable

0.70

0.61

0.67

represent the results of our new method. Also, Table 2 shows the estimation results
and correlation coefﬁcients of the supervisory signals.
The supervisory signal in Fig. 4 is an average value evaluated by 20–40 people.
The correlation coefﬁcients between impressions estimated using this method andthe supervisory signals for all the datasets were about 0.70 for “hard–soft,” about 0.61
for “ﬂashy–sober,” and about 0.67 for “stable–unstable.” As a result, the proposed
method conﬁrmed a strong correlation with each impression. When we analyzed
three-dimensional objects with large difference between estimate results and supervisory signals, the variation in supervisory signals for the three-dimensional object
was large.

5 Conclusion
We proposed a method for automatically estimating three typical human impression factors, “hard–soft,” “ﬂashy–sober,” and “stable–unstable,” which were obtained
from objects by analyzing the shapes of three-dimensional models. The results ofthe experiment show that the correlation coefﬁcients between the impressions estimated using this method and the supervisory signals for the original datasets of car
used were about 0.70 for “hard–soft,” about 0.61 for “ﬂashy–sober,” and 0.67 for“stable–unstable.”
In future work, we will clarify the impression structure of people and estimate
using the RGB image, because our method cannot consider the texture and material.
Acknowledgments This research was partially supported by the Center of Innovation Programfrom the Japan Science and Technology Agency, JST.

References
1. K. Tobitani, S. Akizuki, K. Katahira, M. Hashimoto, N. Nagata, A comparison study on 3Dfeatures in terms of effective representation for impression of shape, in The 2nd International
Conference on Digital Fabrication, No. 22 (2016)
2. K. Taguchi, K. Sasaki, M. Hashimoto, K. Tobitani, N. Nagata, A proposal of 3D local feature for
estimating human’s impression factor to shape of object, in International Workshop on Advanced
Image Technology (2017)
3. C.R. Qi, H. Su, M. Niesner, A. Dai, M. Yan, L.J. Guibas, Volumetric and multi-view CNNs forobject classiﬁcation on 3D data, in IEEE Conference on Computer Vision and Pattern Recognition (2016)

Using Deep Learning to Estimate User Impressions …

71

4. C.E. Osgood, G.J. Suci, P.H. Tannenbaum, The Measurement of Meaning (University of IllinoisPress, Oxford, England, 1957)
5. D.P. Kingma, Adam: a method for stochastic optimization, in International Conference on
Learning Representations (2015)
6. Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, J. Xiao, 3D ShapeNets: a deep representationfor volumetric shapes, in IEEE Conference on Computer Vision and Pattern Recognition (2016),
pp. 1912–1920

Mr. Koichi Taguchi received his B.E. degree from Chukyo University, Aichi, Japan, in 2017. Heis now a graduate student of the university and is very interested in technologies about object
recognition and its application to sensitivity information processing. He is a member of theJapanese societies of IPSJ Japan and JSPE.
Manabu Hashimoto received his B.E. and M.E. degrees from Osaka University, Osaka, Japan, in
1985 and 1987. He joined the Mitsubishi Electric Corporation in 1987, and he has been engaged inresearch on robot vision, image recognition, pattern recognition, and human sensing in the Manufacturing Development Laboratory, Industrial Electronics and Systems Development Laboratory,and the Advanced Technology R&D Center of the company. He received his Ph.D. degree fromOsaka University in research on 3-D object recognition in 2000. Since 2008, he has been a professor of Chukyo University, and he is the Dean of the School of Engineering. He received theTechnical Innovation Award in 1998 from the Robotics Society of Japan, the Excellent Academic
Award in 2012 from the Symposium on Sensing via Image Information, and the Best Paper Awardof IWAIT2017 and IWAIT2018. He is a member of the IEEE and Japanese societies of IEICE,IPSJ, IEEJ, and RSJ, among others.
Kensuke Tobitani received his doctoral degree in engineering from Gifu University, Japan, in
2009. He is currently a lecturer at Kwansei Gakuin University. His research interests include computer vision, computer graphics, and Kansei information processing.
Noriko Nagata received her BS degree in mathematics from Kyoto University in 1983 and herPh.D. degree in systems engineering from Osaka University in 1996. She was a researcher at theIndustrial Electronics and Systems Laboratory of Mitsubishi Electric Corporation from 1983 to2003. She joined Kwansei Gakuin University in 2003 as an associate professor. She is currently aprofessor in the Department of Human System Interaction and a director of the Research Center
for Kansei Value Creation. In 2009, she was a visiting scholar at Purdue University. Her researchinterests include Kansei (affective) information processing, computer graphics, and multimedia
systems. She is a member of the IEEE and ACM.