Bull. Soc. Photogr. Imag. Japan. (2022) Vol. 32 No. 1: 1–9

Original Paper

Shape Reconstruction based on Diffusion of Captured Images through Deep
Learning using Position Data of The Light Sources
Hiroto SUTO*, Ikumi HIROSE* and Norimichi TSUMURA

Abstract:

Three-dimensional reconstruction techniques are used in various situations such as visual inspections in a factory and digital archiving in a
museum. One such method is photometric stereo, in which a change in the direction of the light source is used to illuminate an object. Thismethod can reconstruct objects with high accuracy. By contrast, this approaches have certain limitations, such as the requirement for a darkroom environment with no more light sources than necessary, and the need for a Lambert diﬀuse reﬂection material that reﬂects the incident
light equally in all reﬂection directions. Therefore, a method for transforming the specular reﬂection components of an object into a diﬀuse
form using deep learning was previously developed. However, this method applies a network in which the object surface is not changed despite the change in the position of the light source. It is therefore not applicable to a shape reconstruction using photometric stereo, which applies a change of the position of the light source. In this study, we propose an improved learning network that transforms captured specular
images into a diﬀuse form while reﬂecting the changes in the surface luminance by applying images that diﬀer in the position of the lightsource and its corresponding data. In addition, we veriﬁed the accuracy and eﬀectiveness of a conventional method for both the transformed
2D images and the 3D shape model reconstructed from these images. As a result, we conﬁrmed that the accuracy of our method is improvedin both cases.

Key words:

3D reconstruction, Shape reconstruction, Photometric stereo, Deep learning, GAN

1. Introduction
There has been a need for 3D shape measurements of objects anda 3D reconstruction on a computer, which applies measurement information such as a visual inspection of industrial products and thedigital archiving of cultural properties. Photometric stereo1, 2) is oneof the most popular methods for a 3D reconstruction. This method
applies multiple images in which the direction of the illuminating
light source is changed, and utilizes the relational expression estimating the luminance by the inner product of the light source vector
and the angle of the normal of the object for reconstructing theshape. It therefore implements a highly accurate shape reconstruction because the normal is calculated for each pixel of the image. By
contrast, other approaches have certain limitations, such as the requirement of a darkroom environment with no more light sources
than necessary, and the need for a Lambert diﬀuse reﬂection material that reﬂects the incident light equally in all reﬂection directions.
Therefore, a method for transforming the specular reﬂection components of an object into a diﬀuse form using a generative adversarial
network (GAN), which is a leaning network developed by Wu et al.
However, this method does not change the object surface despite thechange in the position of the light source. Thus, it is inapplicable toa shape reconstruction using photometric stereo, which uses achange in the position of the light source.
In this study, we propose a new learning network that transforms

captured specular images into a diﬀuse form while reﬂecting changes in the surface luminance by using images that diﬀer in terms of
the position of the light source and position data of the light source.
This learning network utilizes the position data of the light source
described as category information of a one-hot vector in addition toimages that diﬀer in the position of the light source. Thus, it enables
a shape reconstruction of the specular surface from environments inwhich more than a single light source is illuminated, such as in photometric stereo. This network allows us to convert the specular partof the captured image into a diﬀuse form, which is necessary forphotometric stereo, and enables a shape reconstruction by applying
the photometric stereo for objects with specular surfaces using theseimages transformed through our learning network. In addition, it
can avoid the limitations of not only surface reﬂections but also thecapturing condition because the diﬀuse and specular images used inthis network are respectively darkroom and non-darkroom images.

2. Related work
This section describes the related research on GANs, which is thebasis of the proposed learning network. The basic structure is basedon the Cycle-GAN proposed by Zhu et al.3), which is a network thattrains a generator and a discriminator by using two sets of images
with features before and after the transformation, transforming one
image into the other, and then reconverting the transformed image

Received 6th October, 2021; Accepted 12th March, 2022; Published 24th March, 2022
Department of Imaging Sciences, Graduate school of Engineering, Chiba University
1-33, Yayoi-cho, Inage-ku, Chiba-shi, Chiba 263-8522, Japn
* Hiroto SUTO and Ikumi HIROSE contributed equally to this work.

2

Bull. Soc. Photogr. Imag. Japan. Vol. 32 No. 1 (2022)

to be closer to the image prior to the transformation. This networkfeatures a mechanism that returns the image transformed by thegenerator back into the image prior to transformation. It is possible
to train the learning network such that this loss decreases by calculating the cycle consistency loss, which is represented by the diﬀerence between the dataset image before the transformation and thedataset image that has been re-transformed to be closer to the image
prior to the transformation using the generator.
Our method was also inspired by the network developed by Wu etal., which uses this Cycle-GAN mechanism and transforms the image from a specular form into a diﬀuse form for a 3D shape reconstruction using images4). This method applies a learning networkthat specializes in converting the surface reﬂection of a specular reﬂecting object into a diﬀuse reﬂection and introduces cycle consistency loss similar to that of a Cycle-GAN. However, because thismethod is intended to be applied along with an approach combining
two types of shape reconstruction5, 6), i.e., Structure from Motion7)
and Shape from Shading8), it is unsuitable for a shape restoration inphotometric stereo simply because of a problem in which the imagetransformation cannot reﬂect the direction of the light source position as shown in Fig. 1.
In this study, we use a StarGAN9) learning network to achieve an
image transformation, which includes the information of the lightsource position to solve this problem. In this research, the feature ofadding category information describing the information of the lightsource position at the time of training makes it easier to determine
the change in the light source position from the image alone. In
StarGAN, the category information is used to update the weights
using the classiﬁcation information of the image transformation. In
addition, to transform images by manipulating the category information, StarGAN has the structure of a single generator. The categorical information in StarGAN prevents learning from proceedingby focusing on features that are not the target of the transformation,
which results in a higher accuracy than learning from images alone.
In addition, the discriminator in StarGAN not only identiﬁes
whether an image has been transformed, by applying the ACGAN10)
method as an auxiliary approach allowing it to discriminate ﬁne details, it also identiﬁes the category of the image classiﬁcation. Using
this method to classify images, it is able to discriminate small changes in comparison to a general GAN. In this study, we incorporate the
network structure of StarGAN into the method developed by Wu etal. and propose a new image transformation network that can transform captured images into images suitable for photometric stereo.

3. Training dataset creation
In this study, we prepared a custom dataset to implement an image transformation that can be used for a shape reconstruction inphotometric stereo. This dataset consists of two types of assets: images of objects with specular and diﬀuse surface reﬂections, and category information that describes the surface reﬂection and lightsource position within the images. In this section, we describe howthese datasets were constructed.
3.1 Image dataset creation
We created the dataset images by rendering a 3D CG model in acomputer. In this study, 10 diﬀerent simple primitive shapes wereapplied, which can be divided into two main types of surface features, planar and curved, at a 1:1 ratio. The shape models include
typical primitive shapes such as polyhedra, cones, or similar shapes
such as spheres, pyramids, cones, cylinders, regular polyhedra, andtorus. Some of the rendering images used are shown in Fig. 2.
As shown in Fig. 3, the captured object was set on the ﬂoor. In
addition, the camera was placed at a ﬁxed position in a direction
vertical to the object.
The material of the object surface was set through the renderingon the computer, as shown in Fig. 3. In this study, we use phisically
based rendering11), which can quantitatively set the position, angle,
and gloss of an object. We applied silver as the material for the specular reﬂection, and a matte material with a white color for the Lambert diﬀuse reﬂection. We set the parallel light to change the surface
luminance, which is necessary for photometric stereo, and set theenvironmental light from outside for the diﬀuse material.
We applied the parallel light by setting up a directional light at
inﬁnity. We set up the light sources at six locations where the angle
between the light sources and the ﬂoor surface is 60°, using the center of the object as the base axis, as shown in Fig. 4. We capture theimages by turning on one light per shot. A 3D reconstruction can beachieved using images with light sources located in at least threepositions using photometric stereo. However, we have frequently observed a collapsed estimation of the normal where there is no light
exposure, and thus we used six light positions to ensure that theentire object is illuminated.
The environmental light is simulated on a computer using ﬁve
diﬀerent HDR environment maps. These environment maps arelimited to 32-bit HDR images captured indoors. This is because, inthis study, we suppose the use of photometric stereo in an indoor

Fig. 1 Results of image transformation with learning network of Wu et al.4) (a) Original image.
(b) Transformed image. (c) Ground truth.

Hiroto SUTO, Ikumi HIROSE, Norimichi TSUMURA

Shape reconstruction based on diffusion of captured images through deep learning using position data of the light sources

environment. In this case, environment maps are rotated at a random angle based on the axis vertical from the ﬂoor to accommodate
images captured in a variety of situations. The rendered image is a
24-bit color image. Although an image containing specular reﬂection should normally be a 32-bit image, because the purpose of thisstudy is to convert a specular reﬂection image captured using an ordinary camera, we decided to apply a 24- bit color image.
3.2 Category information creation
In this study, we describe information regarding the surface reﬂection and light source positions as the category information. The
learning network applied in this study uses such information totransform and classify the images. The category information is described through a one-hot vector using "1"s and "0"s. A vector witha 1 indicates a corresponding category, whereas a vector with a "0"
represents a noncorresponding category. In addition, we classiﬁed
images into two types of categories: surface reﬂection and lightsource position. As an example, the category information under acertain situation is as shown in Fig. 5. The value on the left side describes the surface reﬂection, and the other values indicate the light

Fig. 3 Rendering environment.

3source position. In addition, the generator can switch the target category to a diﬀerent category by switching the "0" or "1" one-hot
vectors using such information. In this study, we use this structure totransform images with a generator.

4. Building of a learning network
The learning model used for this approach applies a GAN consisting of two networks: a generator for image transformation and adiscriminator for image discrimination. The architecture of our approach is shown in Fig. 6.
4.1 Generator and discriminator
The generator serves to transform image ‫ ݔ‬in the dataset into image ‫ݕ‬, corresponding to the transformation target. The training network is based on the StarGAN structure and uses two types of information input into the generator, i.e., the images contained in thedataset and the category information as the target of the transformation. The architecture of the generator is shown in Fig. 7.

Fig. 5 Example of describing surface reﬂections and light positions as category information.

Fig. 2 Example of rendering images with specular surface used. (a) Sphere. (b) Cylinder. (c) Cone. (d) Regular dodecahedron. (e) Torus.

Fig. 4 Position of light source at rendering. (a) View from the shooting direction. (b) View from the direction horizontal to the ﬂoor.

4Bull. Soc. Photogr. Imag. Japan. Vol. 32 No. 1 (2022)

Fig. 6 Architecture and loss function of the proposed method.

Fig. 7 Architecture of the generator using category information.

The target category information for the transformation is createdby switching the one-hot vectors of the category information associated with the images in the dataset, as shown in Fig. 7. By inputting the image into the generator, the image can be converted into
the target image. To achieve the purpose of this study in terms of theimage transformation, we set some constraints related to the condition when choosing the images and category information, including
always switching the one-hot vector of the surface reﬂection before
and after the input of the generator, and ﬁxing the light source position without such switching. In addition, as with a Cycle-GAN, thismethod introduces a consistency loss mechanism in which the image transformed by the generator is transformed again into the original image prior to the transformation, and the diﬀerences between
them are calculated. Therefore, even when the surface reﬂection is
diﬀuse, the transformation from a diﬀuse to specular surface is set up
in the same way as when an image with a specular surface is input.
This makes it possible to transform the image by changing the surface reﬂection while maintaining the information of the light source
position.
The discriminator inputs the image ‫ݔ(ܩ‬, ܿ) transformed by thegenerator and image ‫ ݕ‬contained in the dataset, and then diﬀerenti-

Fig. 8 Architecture of the discriminator with discrimination and classiﬁcation functions.

ates the input images. In this study, we discriminate the images in
two ways: using the image itself, and by classifying the image basedon category information. The architecture of the discriminator isshown in Fig. 8. The discriminator alternately inputs the image‫ݔ(ܩ‬, ܿ) transformed by the generator and dataset image ‫ݕ‬.
First, it identiﬁes whether the image input into the discriminatoris an image transformed by the generator or a dataset image, as witha conventional GAN. However, using this method alone makes it

Hiroto SUTO, Ikumi HIROSE, Norimichi TSUMURA

Shape reconstruction based on diffusion of captured images through deep learning using position data of the light sourcesdiﬃcult to discriminate ﬁne details such as changes in luminance.
Thus, as an auxiliary approach, we use the method applying an ACGAN10) to categorize the images. Such an approach is applied toclassify the image input into the discriminator corresponding to thelight source position. This makes it possible to discriminate smallchanges such as the position of the light source in comparison to a
general GAN.
4.2 Loss function
We calculate the loss function to ﬁnd the optimal weights forachieving a higher accuracy of the generator and the discriminator.
The learning applied in this study is based on four loss functions, i.e.,
adversarial loss, category classiﬁcation loss, and reconstruction loss
based on a conventional StarGAN method, and uniformity preservation loss, which was newly developed for this study. When theadversarial loss is ࣦadv ; the category classiﬁcation loss is ࣦ‫ܩ‬cls and ࣦ‫ܦ‬cls
for the generator and discriminator, respectively; the reconstruction
loss is ࣦrec; and the uniformity preservation loss is ࣦret. In addition,
the hyperparameters of each loss function are ߣadv, ߣcls, ߣrec, and ߣret,
respectively. The ﬁnal loss functions of the generator and discriminator, ࣦ‫ ܩ‬and ࣦ‫ܦ‬, respectively, are as shown in the following equations.
(1)(2)
Adversarial and category classiﬁcation losses are functions that
use the results of an image discrimination and classiﬁcation by adiscriminator. Adversarial loss is the loss of computing the probability distribution of whether image ‫ ݔ‬input into the discriminator iscontained in the original dataset or is transformed by the generator.
The category classiﬁcation loss is a function that calculates the probability distribution of whether image ‫ ݔ‬input into the discriminator
will be classiﬁed as category information ܿ or ܿԢ. Thus, if expectedvalue is ॱ, the adversarial and category classiﬁcation losses can be
expressed through the following equations. For example, if ‫ܦ‬adv(‫ )ݔ‬is
able to output data close to reality, ॱ‫[ݔ‬log ‫ܦ‬adv(‫ ])ݔ‬becomes larger. In
addition, if ‫ݔ(ܩ‬, ܿ) generates an output close to reality,
becomes larger. Thus, ࣦadv will be close
to zero.
(3)

5

(4)
(5)
As shown in Eqs. (4) and (5), the loss function of the generator
calculates the probability for the transformed image, and the discriminator calculates the probability for the original image to learn
separately in the generator and discriminator.
The reconstruction loss is a function for the generator. It computes the diﬀerence between images transformed by the generator
and those transformed again using the generator such that the original image ‫ ݔ‬prior to input is restored. This takes on the role of creating correspondence between the images before and after the transformation. When the target category information of thetransformation is ܿ, and the original category information containedin the dataset is ܿԢ, the reconstruction loss ࣦrec is expressed throughthe following equation.
(6)
We calculate the diﬀerence between the image transformed by thegenerator and the image prior to the transformation with the L1
norm. We train the generator such that the diﬀerence in the above
equation decreases, and then build a learning network that enables
an image transformation that maintains the consistency of the image.
There are problems with the loss function described above. First,
images are transformed with high or low luminance regardless ofchanges in the position of the light source. Second, the transformation fails considerably in areas with low luminance, such as areas
corresponding to shadows. Therefore, in this study, we introduce a
uniformity preservation loss as a new loss function for applying adiﬀuse image illuminated toward an object from the same direction
as the captured position, and then correcting the image from eachlight source position to be closer to this image. An image with auniformity preservation loss is shown in Fig. 9.
This is a function that corrects for changes in the luminance of thesurface to be transformed, with the goal of improving the accuracyof the transformation. The formula is computed using the diﬀerence
between images based on the L1 norm, as with Eq. (6) for the reconstruction loss. When the image in the dataset is ‫ݔ‬, the categoricalinformation targeted for transformation is ܿ, and the image illuminated from the same direction as the captured direction is xԢ, theFig. 9 Mechanism of uniformity preservation loss.

6

Bull. Soc. Photogr. Imag. Japan. Vol. 32 No. 1 (2022)

uniformity preservation loss ࣦret is expressed through the following
equation.
(7)
This makes it possible to transform an image reﬂecting the luminance transformation of the surface owing to changes in the lightsource position while correcting the luminance of the transformed
image.
4.3 Network structure
The network structure of the learning network is based on thatused in StarGAN. The network structure of the generator and discriminator are shown in Fig. 10. The structure of the generator isroughly divided into three layers: a downsampling layer, a bottleneck
layer, and an upsampling layer. The downsampling layer convolutes
images and extracts image features, the bottleneck layer transforms
features based on the weights, and the upsampling layer deconvolutes the features and restores the images from the feature map.
The ReLU function is used as the activation function for eachlayer. In addition, in the bottleneck layer, the structure of the residual network (ResNet)12) is adopted to eﬃciently learn more detailed
features. In addition, as with a general neural network, the structureof the discriminator consists of three layers: an input layer, a hidden
layer, and an output layer. The input layer obtains a feature map fromthe image, the ﬁve hidden layers convolute to extract the features,and the output layer classiﬁes and identiﬁes such features.
4.4 Learning parameters
Two types of images, specular and diﬀuse, are used along with 200
patterns. In the case of specular images, we prepared six light sourc-

es positions, whereas in the case of diﬀuse images, to calculate thediﬀerence between the image illuminated from the same capturing
direction from the same direction, seven images, including one additional image, were used for the calculation of the uniformity preservation loss. The resolution of each image was set to 256 × 256 owing
to the learning time. In addition, although 32-bit images should be
used for images containing specular reﬂections, 8-bit images areused for training in this study because the purpose is to convert images taken under general conditions. We set the learning rate to
0.0001 for both the generator and discriminator; the hyperparameters of each loss function to 1 for the adversarial and uniformity
preservation losses, and to 10 for the category classiﬁcation and reconstruction losses; and the number of iterations to 100,000, whichis the number of times the best accuracy was achieved when we repeated the training of the model and checked the accuracy of theimage transformation and shape reconstruction.

5. Verification of learning results using the proposed
methodWe transformed the images using the learning network developedin this study, reconstructed the shape, and compared the results withthe ground truth. In addition, we veriﬁed the eﬀectiveness of the
newly introduced uniformity preservation loss. This time, among thepolyhedra and cones included in the dataset for primitive shapes, weused pentagonal pyramids, faceted icosahedra, and faceted conesthat were not applied for training, as shown in Fig. 11.
The evaluation of the transformed images is carried out in two
ways: a qualitative evaluation based on the transformation results of
the images, and a quantitative evaluation based on the root meansquare error (RMSE) of the ground truth images. The ground truth
image is an image of the target shape of the object created in a ren-Fig. 10 Network structure. (a) Generator. (b) Discriminator

Hiroto SUTO, Ikumi HIROSE, Norimichi TSUMURA

Shape reconstruction based on diffusion of captured images through deep learning using position data of the light sources

7

Fig. 11 Shapes for veriﬁcation. (a) Pentagonal pyramid. (b) Faceted icosahedra. (c) Faceted cone.

dering environment in which the surface reﬂection is Lambert diffuse reﬂection, as in the training dataset, and with no environment
map used in PBRT. Because the normal is estimated using only thepart of the object that is not included in the target object in illuminance stereo, the RMSE is calculated by masking the part of theobject that is not the target object in the quantitative evaluation aswell.
As with the evaluation of the image transformation, the evaluation of the restored shape is carried out in two ways: a qualitativeevaluation using the estimated normal map showing the direction of
the normal ‫ ݖݕݔ‬as an RGB value, as well as a 3D model of the restored shape, and a quantitative evaluation using the RMSE between the point cloud and the restored shape from the ground-truth
image. The RMSE between point clouds is synonymous with thecalculation of the Euclidean distance between point clouds. To derive this Euclidean distance, we use the iterative closest point (ICP)
algorithm13), which is a method for aligning point clouds. This method is used to align the point cloud of the target shape by repeatedly
detecting the nearest neighbor points between the point cloud of theground truth shape and the point cloud estimated from the transformed image, and by estimating the rigid body transformation sequence such that the error in the distance between the nearest
neighbor points is minimized. The process is repeated until the error
in the distance between the point clouds is less than the thresholdfor convergence, and the average error in the distance between the
point clouds at the time of convergence is derived for each corresponding point cloud. In practice, if the points to be mapped are ‫݌‬,the number of point groups is ܰ, the points to be mapped are ‫ݍ‬, andthe rotation and translation of the rigid body transformation sequence are ࡾ and ࢚, respectively, the Euclidean distance between the
point groups is ݀211, which is calculated as follows:(8)

5.1 Qualitative evaluation
In Fig. 12, (1) is a captured image to be input into the learningmodel, (2) is an image used as the ground truth, (3) is an imagetransformed using S2Dnet as a conventional approach, (4) is an image transformed using the learning model without uniformity preservation loss, and (5) is the images transformed using our approach.
Nos. 1-6 indicate the horizontal direction.
The result of a shape reconstruction when applying photometricstereo using the transformed images is shown in Fig. 13. This resultof the shape reconstruction is a view from the horizontal direction.

Fig. 12 Images transformed from each light position. (a) Pentagonal pyramid. (b) Faceted icosahedra. (c) Faceted cone. ((1) Input images. (2)Ground truth. (3) Wu et al.'s approach.4) (4) Our method (with nouniformity preservation loss). (5) Our method.)

8Bull. Soc. Photogr. Imag. Japan. Vol. 32 No. 1 (2022)

Fig. 14 RMSE between ground truth and transformed images with Wu et
al.'s approach8) and our method without and with uniformity preservation loss.

Fig. 13 Ground truth shape and reconstructed shape with Wu et al.'s approach4) and our method without and with uniformity preservation
loss. (The upper level is a normal view. The lower level is a horizontal
view.) (a) Pentagonal pyramid. (b) Faceted icosahedra. (c) Faceted
cone.

Based on the results, we conﬁrmed that our method can transform images reﬂecting changes in the luminance of the surface owing to changes in the light source position better than the conventional method. Although we could not conﬁrm signiﬁcant changes
in the image transformation with or without a loss of uniformity
preservation, we conﬁrmed that the reconstructed shape was closer
to ground truth in terms of the height from the ﬂoor.

Fig. 15 RMSE between ground truth and reconstructed shapes with Wu et
al.'s approach8) and our method without and with uniformity preservation loss.

mity preservation, where the error in the reconstructed shape wasslightly smaller with the method that introduced our proposed loss.

6. Discussion
5.2 Quantitative evaluation
A graphical representation of the RMSE of the transformed images using the S2Dnet when applying the conventional method, themethod with no uniformity preservation loss, and our proposed
method, along with the ground truth image of each shape, are shownin Fig. 14. Each graph is the result of averaging six images formed
into each shape, and the average for each method is shown on theright. The standard errors based on the shape applied are also shownin the same graph along with error bars.
In addition, Fig. 15 shows a graphical representation of the
RMSE results when applying the conventional method, the method
without uniformity preservation loss, and our proposed approach,along with the ground truth images, using photometric stereo. These
results indicate the individual errors for the three validation geometries, and the mean and standard error based on the geometry arealso shown on the right side.
The results show that the proposed method has a smaller RMSE
error than the conventional method for both the transformed image
and the reconstructed shape, and the accuracy of the proposed
method has been improved. In addition, there were no signiﬁcantchanges in the transformed image with or without a loss of unifor-

From the veriﬁcation results described in section 5, it can be stated that our method can convert an image to reﬂect the changes inluminance caused by the change in the direction of the light source
and can restore the shape close to the ground truth in comparison to
the conventional method. In particular, as one of the major diﬀerences between our method and the conventional approach, thethickness of the shape from the ﬂoor surface can be restored using
our method. As the reason for this, the image used in this study,
which is applied as an index for calculating the loss of uniformity, isilluminated from the same direction as the shooting direction. Although the distance between the ﬂoor surface and the light source
was not changed in the images applied for this training, the training
was conducted such that the images were closer to those irradiatedfrom the same direction as the shooting direction. Because of this,the luminance was converted to be higher at positions closer to theshooting direction and lower at positions farther away. As a result, itis therefore highly possible that the luminance representation reﬂected the shape of the vertical direction from the ﬂoor surface.
From this perspective, it can be stated that the introduction of theuniformity retention loss is eﬀective.

Hiroto SUTO, Ikumi HIROSE, Norimichi TSUMURA

Shape reconstruction based on diffusion of captured images through deep learning using position data of the light sources

By contrast, it was conﬁrmed that the surface was distorted incomparison to the results of the ground truth, and that the incorrectconversion has not yet been suﬃciently improved. This problem isthought to be largely due to the fact that the calculation of the uniformity preservation loss is from a diﬀerence in subtraction. Therefore, it will be necessary to improve this problem by reducing thenumber of convolutional layers, introducing the structure of a
Self-Attention GAN14) that can generate images based on globalfeatures and allowing such features to be learned, improving the hyperparameter of the uniformity preservation loss, and increasing thevariation of the height of the light source from the ﬂoor.
7. Conclusions and future work
In this study, we proposed a new learning network that can reﬂect
the changes in luminance of a surface owing to changes in the lightsource position by adding the position data of the light source as
category information. The results of photometric stereo using theimages transformed by our learning network showed that it is possible to reconstruct a shape more accurately than through a conventional image transformation method.
In the future, we will solve the problem of distortions appearing
on the surface of the shape and improve the method to address theincorrectly transformed parts of the image. In addition, it is expectedthat the approach will handle 3D reconstructions with images undera wide range of conditions by increasing the number of light sources,materials, and shapes included in the dataset images. Moreover, thisresearch was aimed at converting specular images captured by popular cameras. Because of this, we used 24-bit images in the training
set. However, specular images require a very wide dynamic range,and 24-bit images may not be able to accurately represent specular9

reﬂection. Therefore, in order to construct a conversion network that
can handle specular reﬂection images more accurately, it will be necessary to conduct learning using HDR format images of approximately 32 bits in the future.

References
1)

W. M. Silver, "Determining shape and reﬂectance using multiple images," PhD thesis (Massachusetts Institute of Technology, 1980), 1980.
2) R. J. Woodham, Optical Engineering, vol. 19, pp. 139-144 (1980)
3) J. Y. Zhu, T. Park, P. Isola, A. A. Efros, Proceedings of IEEE ICCV, pp.
2223-2232 (2017)4) S. Wu, H. Huang, T. Portenier, M. Sela, D. Cohen-Or, R. Kimmel, M.
Zwicker, Proceedings of ECCV, pp. 183-200 (2018)5) F. Langguth, K. Sunkavalli, S. Hadap, M. Goesele, ECCV, pp. 469-485(2016)
6) R. Maier, K. Kim, D. Cremers, J. Kautz, M. Nießner, Proceedings ofIEEE ICCV, pp. 3114-3122 (2017)7) O. Özyeşil, V. Voroninski, R. Basri, A. Singer, Acta Numerica, vol. 26,
pp. 305-364 (2017)8) R. Zhang, P. S. Tsai, J. E. Cryer, M. Shah, IEEE TPAMI, vol. 21, pp.
690-706, 1999
9) Y. Choi, M. Choi, M. Kim, J. W. Ha, S. Kim, J. Choo, Proceedings ofIEEE CVPR, pp. 8789- 8797 (2018)10) A. Odena, C. Olah, J. Shlens, PMLR, vol. 70, pp. 2642-2651 (2017)11) M. Pharr, W. Jakob, G. Humphreys, "Physically based rendering: From
theory to implementation," 3rd ed., by J. Pierce, Morgan Kaufmann,
Massachusetts, 201612) K. He, X. Zhang, S. Ren, J. Sun, Proceedings of IEEE CVPR, pp. 770778 (2016) 13) P. J. Besl, N. D. McKay, SPIE, vol. 1611, pp. 586-606(1992)
14) H. Zhang, I. Goodfellow, D. Metaxas, A. Odena, PMLR, vol. 97, pp.
7354-7363 (2019)