Electronics and Communications in Japan, Part 2, Vol. 90, No. 12, 2007
Translated from Denshi Joho Tsushin Gakkai Ronbunshi, Vol. J89-D, No. 2, February 2006, pp. 381–392

Color Photometric Stereo and Virtual Image Rendering Using
Neural Networks
Haruki Kawanaka,1 Yuji Iwahori,1 Robert J. Woodham,2 and Kenji Funahashi3
1

Department of Computer Science, Chubu University, Kasugai, 487-8501 Japan2

Department of Computer Science, University of British Columbia, Vancouver, B.C. Canada V6T 1Z4
3

Graduate School of Engineering, Nagoya Institute of Technology, Nagoya, 466-8555 Japan

Key words: neural networks; photometric stereo;
mixed reality; model-based rendering.

SUMMARY
In this paper we extend the application of neuralnetwork-based photometric stereo founded on the principleof empirical photometric stereo to color images proposing
a method for computing both the normal vectors of a target
object and its color reflectance coefficients. This method isable to render objects that have non-Lambert reflectanceproperties without using any parametric reflectance function as a reflectance model. In addition, we propose a novel
neural network-based rendering method that allows thegeneration of realistic virtual images of an object with
arbitrary light source direction and from arbitrary viewpoints based on the physical reflectance properties of theactual object and perform a comparative evaluation withapproximations by existing models, the Phong model, andthe Torrance–Sparrow model. © 2007 Wiley Periodicals,Inc. Electron Comm Jpn Pt 2, 90(12): 47–60, 2007; Published online in Wiley InterScience (www.interscience.
wiley.com). DOI 10.1002/ecjb.20423

1. Introduction
In the fields of computer vision and mixed reality,
research was originally focused primarily on monochrome
grayscale images; in recent years, however, color images
have gradually become the focus of such work [1–4].
With a fixed single viewpoint, Woodham [5] proposed a method known as photometric stereo for estimating
local surface normal vectors for the surface of an object
from multiple grayscale images taken under different lightsource conditions; in addition, he proposed [6] empiricalphotometric stereo, a method that does not assume a function for the reflectance properties of the surface of an object
and does not require information of the light source directions. These methods also fix the viewpoint while theimages are taken under different directions of the lightsource.
Iwahori and colleagues have previously developed aneural network-based photometric stereo method [7, 8].
This method is based on empirical photometric stereo [6]by training a neural network on a sphere with the same
surface reflectance properties as the test object; it is possible
to obtain the normal vectors for the surface of the object
and monochrome reflectance coefficients without assum-

Contrat grant sponsors: Scientific Research Grant [Basic Research (C) (2)16500108], the Kayamori Foundation of Informational Science Advancement (Yuji Iwahori), the Hori Information Science Promotion Foundation(Kenji Funahashi), and the Institute for Robotics and Intelligent Systems
and the Natural Sciences and Engineering Research Council (Robert J.
Woodham).

© 2007 Wiley Periodicals, Inc.
47

ing any parametric function for the reflectance propertiesof the surface of the object nor explicitly using a value forthe lighting directions. However, since the method given in
Ref. 8 makes use of an assumption that the reflectancefactor is a constant parameter proportional to the reflectance property function, there is a problem remaining in thatareas of the sharp specular reflections give some errors inthe surface normal obtained due to the effects of this specular reflectance.
Against this background, in this paper we first extend
the neural network-based photometric stereo to color images and then such that it can be applied to objects that giverise to sharp specular reflections. Both Refs. 9 and 10 haveproposed photometric stereo for color images; however,there are problems and limitations such that the former uses
a particular color light source while the latter assumes
completely diffuse reflectance (Lambert reflectance). Our
proposed method uses a standard white light source and isapplicable to objects with non-Lambert reflectance properties. In addition to extending photometric stereo to color
images, the method also extends to objects on which sharp
specular reflections arise.
Model rendering, on the other hand, consists of thecreation of virtual images of an object given a mathematicalformulation of the reflectance properties on the object’s
surface as a parametric reflectance function (such as the
Phong model) and the geometric shape of the object. In Ref.
11 a range finder was used to reconstruct the three-dimensional shape of an object and by assuming a Torrance–Sparrow model as the surface reflectance function of the object,a method was proposed to estimate its parameters from acolor image of the object. In this paper we propose a neuralnetwork-based rendering method that generates the appearance of an object using the viewing direction vector, thelight source direction vector, the object’s surface normal
vector, and color reflectance coefficients as an inverse
system of neural network-based photometric stereo. Thismethod makes no use of any parametric reflectance function and can generate realistic virtual images of an object
from arbitrary viewpoints and based on arbitrary lighting
directions.

undergo a modulation according to the inherent reflectivity
rate of the substance and then the reflected rays are projected uniformly in all directions. The color of these reflected rays is treated as the standard color of the object.
Another type of reflection is specular reflectance; due torelations of the inflection rate between the media, the rays
do not enter the inside of the object and are reflected on thesurface. Since these incident rays do not enter the object,they do not undergo any absorption and the reflecting rays
are projected as a direct specular reflection along the surface normal vector. Since specular reflections do not undergo the influence of the reflecting substance, they are thesame color as the light source. The observed light is a linear
sum of these two types of ray and can be expressed asfollows [12]:(1)Here E is the grayscale value observed, Ew is the intensityof the light source, and ρ is the reflectivity rate (the reflection coefficient). The first term represents the diffuse reflection component while the second term represents the
specular reflection component. d and m are the mixing
proportions; these are used to express the differences incolor by location on the image. d and m are the coefficientsof the surface normal vector n = (nx, ny, nz), the light source
direction vector l = (lx, ly, lz) and the viewing direction
vector v = (vx, vy, vz). In other words we may write(2)
Here Rd and Rm represent the diffuse reflection component
and the specular reflection component respectively of thereflectance properties.
2.2. Neural network-based photometric stereo
This method proposed in Ref. 8 involves fixing thelocation of the camera and the object and estimating thegradients and the reflectance coefficients of the object’s
surface from four monochrome grayscale images taken
under four different light source directions. However, here
the normal vectors and reflectance coefficients are estimated without computing the precise directions of the lightsources.
From Eq. (2), when the light source direction andviewing direction are fixed, variation in brightness values
will be due to differences in the direction of the surface
normal vector. In other words, when the light source direction and viewing direction are fixed, then Eq. (2) becomes

2. Principles of Dichromatic Reflectance
Model and Neural Network-Based
Photometric Stereo
2.1. The dichromatic reflectance model(3)

With substances such as plastic, incident rays (theilluminating rays) can be reflected in one of two ways. First
they can result in a diffuse reflectance, the illuminating rays
enter the inside of the object from the object’s surface andand when we have four observations each taken with adifferent light source direction, then the following set ofphotometric image equations can be established:

48

(4)

The neural network-based photometric stereo method constructs a radial basis function neural network (RBF-NN)
that takes {E1, E2, E3, E4} as input and produces {n, ρ} as
output [13]. Here in using monochrome grayscale images,
ρ will be a scalar value from 0 to 1. RBF-NN is able toperform nonlinear nonparametric approximation. Therefore, we are able to estimate the surface normal vector n
and the reflectance coefficient ρ from the set of four brightness values without the need to assume a particular parametric form of the reflectance function. The observation
environment is shown in Fig. 1 and the structure of thisRBF-NN is shown in Fig. 2.
The neural network is trained by using images of asphere with the same reflectance properties as the object
that we intend to perform reconstruction for. The neuralnetwork is able to sufficiently learn the relationship between the surface normal vectors and the brightness levels
by using a sphere for training since in addition to the factthat the surface normal vectors of the sphere can be computed easily from its radius and the image coordinates, thesphere also possesses all different kinds of surface normal
vectors. For training data we select a single value for thereflectance coefficient ρ from the set of values 1.0, 0.8, 0.6,
and 0.4 and then combine this value of ρ with four images
consisting of a set of brightness levels for identical pixels{ρE1, ρE2, ρE3, ρE4} and take this as input and then set thesurface normal vector and reflectance coefficient{nx, ny, nz, ρ} as output and construct training data fromvarious different points in the image of the sphere. NN thenlearns the relationship between the set of brightness levels
and the corresponding surface normal vector for various
different settings of the reflectance coefficient. We construct the training set for the neural network by sampling
uniformly from the space on the hemisphere with various

Fig. 2. RBF-NN for monochrome photometric stereo.

surface normal vectors as the outputs. These training data
are used to train the neural network.
When estimating the surface normal vectors of theobject and the reflectance coefficients, we give the set offour grayscale values for the same pixel {E1, E2, E3, E4}from four images of the object that were taken with the samelight source directions as the sphere used in the trainedneural network. NN outputs the local surface normal vector
and the reflectance coefficient. By performing this procedure for all pixels of the object, we are able to obtain thesurface normal vectors and reflectance coefficients for theobject in the image.

3. Neural Network-Based Color
Photometric Stereo
3.1. Applying neural network-based
photometric stereo to color images
When photometric stereo is applied to the surface
with completely diffuse reflection (Lambert reflectance),
the reflectance coefficients and surface normal vectors canbe determined analytically from three grayscale images [6].
When the reflectance coefficients of the target object are
non-Lambert with nonuniform reflectance in general, thesecan be estimated using four or more images from theviewpoint of unknown parameters and nonlinear simultaneous equations. In Ref. 8, four images are used, based on
this reasoning. When applying photometric stereo to color
images in this work we also use four different images takenwith four different directions for the light source in order toestimate the reflectance coefficients and the surface normal
vectors as given in Ref. 8.
Equation (1) showing the reflection model for the
bicolor case is rewritten as the following equation whenapplied to color images:(5)
Here E = {ER, EG, EB} are the RGB values for an image,
Ew = {EwR, EwG, EwB} are the intensities of the light source,
and r = {ρR, ρG, ρB} are the color diffuser reflectanceFig. 1. Observation setup system 1.

49

coefficients. As in the monochrome case, when the lightsource directions and the viewpoint directions are fixed, we
can write

The training data are constructed using images of asphere as in the original method. We need to create thesetraining data to be a uniform sampling from the whole spaceof values that can be obtained, but it is physically impossible to actually give diffuse reflectance coefficients withdifferent RGB values to the sphere. In addition, it is notdesired to require the taking of images of spheres of various
different colors. Therefore, the sphere used for the training
is white (r = {1, 1, 1}) but we set the color reflectancecoefficients r as follows to be r′ whereby each pixel israndomly assigned a coloration:(6)
Here CL represents the RGB value for the light source color.
We can extend photometric stereo to color images on thebasis of Eq. (6).
Constructing a neural network that can compute thesurface normal vectors and color diffuse reflectance coefficients simultaneously from the set of brightness values asin the monochrome neural network-based photometric stereo, would require a large amount of training data and isunrealistic. If the training data consisted of images represented in 24 bits (full color), then there are already around
167 million possible color diffuse reflectance coefficient
combinations and the scope of training is 65,536 times
greater than that of the monochrome case. When we consider how these would combine with the surface normal
vectors, the training scope becomes extremely large and it
is clearly impractical to extend the monochrome method
directly to the case of color images in terms of both theamount of training data required and the amount of timeneeded for training. Therefore, in order to perform training
efficiently we introduce the two-step neural network andpropose a method whereby the color diffuse reflectancecoefficients and the surface normal vectors are computed
using separate neural networks.

(7)
Here rand is a uniform random variable (a real number)
between 0 and 1. Using Eqs. (5), (6), and (7), we alsotransform the set of brightness values into the following E′
that reflects the effects of randomly generated color reflectance coefficients:(8)
When applying this coloration, it is necessary first to
partition each of the brightness level values of the imagesof the sphere used for training into a diffuse reflectancecomponent Rd and a specular reflectance component Rm. In
the method presented here we partition the reflectancecomponents of the images of spheres in the following
manner. We make use of four color images taken under four
different light source directions and also four more color
images of the same sphere from a different viewpoint (atotal of eight images). Even if the light source directions are
not given explicitly as long as they are fixed, then if the
viewpoint is changed significantly the locations in which
specular reflections occur will also change significantly; in
addition, the diffuse reflectance component that does notdepend on viewing direction will remain constant. Basedon these conditions and prior knowledge of shape consisting of the sphere, we are able to determine the diffuse
reflectance component by observing corresponding points
in the two images between the viewpoint change, and takethe darker pixel value for each light source. Depending on
the surface material, specular reflectance components may
be observed over a fairly wide range but in order to performthe decomposition of components into diffuse reflectanceand specular reflectance as described in this paper, thespecular component must be somewhat sharp. This is acondition on the objects to which the method can be applied. Figure 17 shows an example of the appearance of thedecomposition of reflectance components for the images ofspheres used for training with a randomly selected RGB
reflectance coefficient added based on r′.
To train the neural network we make use of imagesof the sphere that have random reflectance coefficients. The

3.2. A neural network to estimate color
reflectance coefficients
As shown in Fig. 3 we use an RBF-NN that isconstructed such that the inputs are given as{E1R, E1G, E1B, E2R, E2G, E2B, E3R, E3G, E3B, E4R, E4G, E4B}
and the outputs obtained are {ρR, ρG, ρB} in order to compute the color diffuse reflectance coefficients. Both the
input and output values are color RGB values.

Fig. 3. RBF-NN for reflectance coefficient estimation.

50

g
g
g
training data consist of inputs {E1R
, E1g G, E1g B, E2R
, E2G
,
g
g
g
g
g
g
g
E2B, E3R, E3G, E3B, E4R, E4G, E4B} and outputs corresponding to the color reflectance coefficient for each of thesegpixels {ρRg , ρG
, ρBg }; the neural network is then trained on
the sphere with various different points.
When estimating the color reflectance coefficientsfor an object, we input the set of RGB values for each pixelin four images of the target object taken with the same four
light source directions as used for the sphere{E1R, E1G, E1B, E2R, E2G, E2B, E3R, E3G, E3B, E4R, E4G, E4B}
to the trained neural network. Color reflectance coefficient
will be obtained for each pixel as output. By applying thisprocedure to all pixels on the object, we can obtain all thecolor reflectance coefficients.

Fig. 4. RBF-NN for estimating vector surface normal
vector.

This neural network learns the mapping between theset of grayscale values and the surface normal vectors; there
we add the random monochrome reflectance coefficientswith 256 levels (8 bits) to a white sphere used for training.
Training is performed by creating training data that consistof sets of grayscale values of the image of the sphere{E1g , E2g , E3g , E4g } to which are randomly added monochromereflectance coefficients corresponding to the diffuse reflectance components as inputs and surface normal vectors{nx, ny, nz} that correspond to this pixel as outputs.
When estimating the surface normal vectors for an
object we transform four color images of the object inmonochrome images and input the set of brightness levels
for each pixel {E1, E2, E3, E4} into the trained neuralnetwork. This results in the estimate of the surface normal
vector corresponding to this pixel being output. By performing the procedure for all pixels on the object, we canobtain the distribution of surface normal vectors for theobject.

3.3. A neural network to compute surface
normal vectors
We now describe the neural network for computing
the surface normal vectors. When estimating the surface
normal vectors, the color image is converted into a monochrome image using the following equation. This equationis generally used when converting color images into monochrome (grayscale) images:(9)
However, the grayscale transformation resulting from thisequation makes the dynamic range of the blue and the redsmall. With neural network-based photometric stereo, thelarger the dynamic range is, the easier it is to learn thegrayscale variations. Therefore, for each light source direction, we assume that the grayscale value is proportional tothe maximum of the RGB values and transform the color
image to a grayscale image using the following equation:

3.4. Positioning of this method(10)

Our method implements empirical photometric stereo that does not make explicit use of information regarding
the direction of the light source using a neural network. In
order to create the training data for the neural network, we
partition the image of a sphere into the diffuse reflectancecomponent and the specular reflectance component andthen randomly add a reflectance coefficient to the diffuse
reflectance component fusing these together; here, we decompose the reflectance components using prior knowledge of shape of the sphere and features of the diffuse
reflectance. This method makes use of a general assumption, namely, the dichromatic reflection assumption, but
specifically does not assume a parametric characteristicreflectance function to describe the specular reflectance. In
general decomposing the reflectance components of an
object of unknown shape is difficult using only the assumption of dichromatic reflection when the light source direction is not given explicitly, and the diffuse reflectanceWe then train the neural network taking the maximum value
among the RGB components as the monochrome value forthe corresponding pixel.
We could consider a method that uses the original
RGB values for input to the neural network; however, byusing the color RGB values instead of the grayscale values,a larger amount of training data will be needed and consequently it may result in a lower precision.
Figure 4 shows the structure of a neural network thatuses the grayscale values to estimate surface normal vectors. The neural network has a structure that allows the setof grayscale values {E1, E2, E3, E4} to be given as input andthe surface normal vector {nx, ny, nz} to be obtained as
output. In this way, since we convert to monochrome images as were used in the original method, we are able tohave the neural network estimate surface normal vectors forcolor images without any hindrance.

51

coefficients and the surface normal vectors cannot be calculated analytically.
An empirical approach to solve this problem that is
not analytically solvable under the condition that the lightsource direction is not given explicitly and no parametricfunction is assumed for the reflectance properties has been
given in Ref. 6 whereby the relationship between the actual
observed intensities and surface normal vectors is mappedusing a look-up table. Neural network-based photometricstereo (Refs. 7, 8, and this paper) extends the method ofRef. 6 by training a neural network to allow generalization.

Fig. 5. Incident angle i, emittance angle e, and phaseangle g.

4. Neural Network-Based Rendering(15)
Since Rd can be computed analytically as the inner productn ⋅ l = cos i of the surface normal vector n obtained via
photometric stereo and the explicitly specified light source
direction vector l, the neural network is used simply toestimate the term Rm. We construct a neural network thattakes i, e, and g that depend on l, n, and v as input and giveas the output Rm consisting of the RGB value for each pointof the object. This neural network is shown in Fig. 6.

4.1. A neural network used in rendering
Here as an application to mixed reality, we describe
a method for generating virtual images of an object from anarbitrary viewpoint and with an arbitrary light source direction using obtained surface normal vectors and the color
reflectance coefficients.
The neural network-based photometric stereo described above is a method for estimating the diffuse reflectance coefficients and the surface normal vectors of an
object; however, the neural network can also be used for theappearance of the object with realistic images generated.
We refer to this as neural network-based rendering.
Neural network-based rendering is a method for generating realistic virtual images of an object from an arbitrary viewpoint and with an arbitrary light source direction
without assuming any parametric function characterizing
the reflections as in the Phong model or the Torrance–Sparrow model.
From Eq. (2) we see that the brightness of a given
pixel is determined by the angles formed by the light source
direction vector l, the surface normal vector n, and theviewpoint direction vector v. The incident angle i, theemittance angle e, and the phase angle g are each functionsof n, l, and v given respectively by the following equations:4.2. Creating training data and neuralnetwork-based generalization
Since only the specular reflectance component isestimated using the neural network, there is no need to train
the network for the diffuse reflectance component. Consequently, the neural network training procedure differs fromthat used for photometric stereo and is performed using
only white sphere images. The training data are created bysampling various different combinations of i, e, and g. Herethe ranges that these angles can take are as follows:(16)
In order to create sufficient training data, we use images ofthe sphere from seven different directions as shown in Fig.
7. The light source is on the xz plane and the angle between
the straight lines from the origin to the light source foradjacent locations is every 15°. In addition, as in the case

(11)
(12)
(13)
The reflectance property R can be expressed as afunction of i, e, and g shown in Fig. 5 and the diffuse
reflectance coefficients r:(14)
Here since the objects have a dichromatic reflection, we
estimate E using the following equation decomposing thereflection into the specular component Rm and the diffuse
component Rd and calculating each separately:

Fig. 6. RBF-NN used for neural network-based
rendering.

52

normal vectors and color reflectance coefficients for eachpixel of the object in the pose to be generated.
Therefore, we first compute the object’s height distribution by integrating the surface normal vectors obtained
from the photometric stereo. We then create another height
distribution in a different pose by rotating the height distribution over the x, y, and z axes. In addition, we compute thesurface normal vectors and color diffuse reflectance coefficients by determining correspondences between object
point during rotation. By giving the intended light source
direction and the values of i, e, and g computed from thesurface normal vectors after the rotation as input to theneural network, we obtain the specular reflection component for each corresponding pixel and then the RGB values
for each pixel from Eq. (15).

Fig. 7. Observation setup system 2.

of photometric stereo, we decompose the image of thesphere into a specular reflectance component and a diffuse
reflectance component.
The training data consist of sets of inputs {i, e, g} andthe corresponding specular reflectance component of the
pixel of the sphere {Rm} as output sampled for various
different points of the image of the sphere. It is clearlystraightforward to have i and e take a variety of different
values and varying the location of the light source as shownin Fig. 7 enables us to create a variety of different combinations of i and e with seven different types of g thereby
allowing sufficient combinations for training data. In general, g can take values in the range shown in Eq. (16);however, in this paper we created training data with g in therange 0 ≤ g ≤ 90. Although it is not a substantial problem,
if it is necessary to generate images with a value of g greaterthan 90°, then it would suffice to increase the number ofdistinct light source locations to 12. We use these training
data to train the neural network.
When generating a virtual image of an object using
the trained neural network, we use Eq. (13) to compute {i,e, g} from the light source direction vector l, the surface
normal vector n, and the viewing direction vector v for eachpixel in the image and then give these as input to the neuralnetwork. This results in the specular reflectance component
for the pixel Rm as output. By using this Rm, the corresponding Rd and the color reflectance coefficients {ρR, ρG, ρB}
obtained from the neural network in Fig. 4, we can obtain
the RGB value for the pixel via Eq. (15). By repeating thisprocedure for each pixel in the object, we are able to createa virtual image of the object.

5. Experiments and Discussion
In this work we also performed an evaluation of the
precision and merits of neural network-based color
photometric stereo based on the approach of Ref. 15. The
following experiments were performed on an AthlonXp2500+ machine with main memory of 512 MB; weused MATLAB to train the neural network.
5.1. Simulation experiments with neuralnetwork-based photometric stereo
5.1.1. Color diffuse reflectance coefficients
Figures 18(a) to 18(d) show four input images (thetarget object for reconstruction is a sphere). In order toperform a quantitative evaluation of the precision of the
proposed method, these images of spheres were createdusing the Torrance–Sparrow model. Without assuming thereflectance function or the light source direction information explicitly, neural network-based photometric stereo isused for evaluation. We set the number of training examples
for the neural network to approximately 2000, the numberof training iterations was 400, and the spread constant was
0.4. Training took approximately 15 minutes while diffuse
reflectance coefficients took approximately 10 seconds tobe estimated.
Figure 18(e) shows the color of the object displaying
the color diffuse reflectance coefficient estimated by neuralnetwork-based photometric stereo as RGB values. We compute the error between the RGB values of the estimatedreflectance coefficients (a real-valued vector normalized
over the range 0 to 1) and the theoretical value as

4.3. Generating virtual images from different
viewpoints
To generate images in which the viewpoint has beenchanged (these are equivalent to images in which the object
and the light source have undergone a rotation), it is not
sufficient simply to have the neural network generalize thereflectance function; rather, in order to generate images inwhich the object is rotated, it is necessary to obtain surface(17)
Here ERGB is the vector of the RGB values of the estimatedreflectance coefficients, TRGB is the vector of theoretical53

gradient of the object’s aspect. The estimation results using
the previous method are shown in Fig. 9.
We aimed for a higher precision value than that of theprevious method [8]. While the average angle of error forthe previous method was 2.310°, the proposed method
could estimate the angle with an error of 0.371°. The use of
the dichromatic reflection model resulted in an improvement in the precision for the object where sharp specularreflections occurred.
Errors were large for the previous method on theregion where sharp specular reflections occurred. The reflectance property model for the previous method isFig. 8. Results of the proposed method. (a) Slope;(b) aspect.

values, and N is the number of pixels in the target object.
The average RGB error with 256 levels is between 10 and
15; this corresponds to when the error values were aiming
for between 0.0391 and 0.0586, and the actual error was
0.0373. This result confirms that reconstruction for color
diffuse reflectance coefficients was performed with a highlevel of precision.

(18)

Unlike the previous method given in Ref. 8, theproposed method adopts a dichromatic reflection modelthat allows it to deal with sharp specular reflections. Therefore, we compare the estimation accuracy of the proposed
method and previous method over a region of sharp specular reflections.
Both methods make use of a neural network with thestructure shown in Fig. 4; in these experiments as with theprevious method we did not make use of a dichromaticreflection model and also used Eq. (9) to transform images
into monochrome. The previous method does not assume adichromatic color model. In addition, we performed training for both methods using a set of 2000 examples, thenumber of training iterations was 400, and the spread
constant was 0.4. The training time for both methods wasapproximately 10 minutes; the estimation of surface normal
vectors took approximately 10 seconds for both methods.
The estimation results using the proposed method areshown in Fig. 8. Panel (a) shows larger angles between thesurface normal vector and the viewpoint direction vector
with a brighter coloring. Panel (b) depicts the directionaland since this does not comply with the dichromatic reflection model, in regions where a sharp specular reflection
occurs, errors appear due to the influence of specular reflections on the corresponding surface normal vectors. In contrast, the proposed method is able to perform better
estimation in these regions as well without a large error
arising. With the proposed method the error angle for
estimated vectors in regions with specular reflections was
also around 0.4° giving a higher accuracy than the previous
method.
We next conducted estimation experiments using
color images themselves. The estimation results are shownin Fig. 10. The overall precision gives an average error angleof 19.630°; this is significantly worse than the proposed
method. Smooth estimation errors were obtained even in
the vicinity of boundaries between diffuse and specularreflections but overall an error that drew the surface normal
vectors toward the viewing direction occurred. When color
RGB values are input to the neural network, a significantly
larger amount of training data is required compared to whenmonochrome grayscale values are used; as a result, theestimation precision becomes low. This confirms that byconverting these values to monochrome grayscale values
and then estimating the surface normal vectors, estimationis performed efficiently in terms of both training time andprecision.

Fig. 9. Results of the previous method [8]. (a) Slope;(b) aspect.

Fig. 10. Results of the proposed method using color
RGB values as input. (a) Slope; (b) aspect.

5.1.2. Surface normal vectors

54

5.1.3. Relationship between reflection
component decomposition accuracy andestimation precision
In order to create training data for the proposed
method, we decompose the reflection components in training images of a sphere. We performed experiments toevaluate what effect the accuracy with which this decomposition was performed had on the precision with which thesurface normal vectors are estimated.
The proposed method treats objects on which sharp
specular reflections occur; however, we now evaluated theerror when we had erroneously decomposed the reflection
component setting it up to 50 percent larger than its actual
value and as low as 50 percent less than this. The results areshown in Fig. 11. The horizontal axis shows the proportionof error in the decomposition of the specular reflection
component while the vertical axis shows the average andthe maximum error angle given this error in the decomposition. From these results we can see that while an error ofplus or minus 10 percent away from the true specularreflection component does not produce a significant change
in the estimation results, a larger error in this decompositionof the components will result in a large error in the estimation errors around boundary regions of specular reflection.

Fig. 12. (a) Slope and (b) aspect estimated by the
proposed method.

coefficients and the neural network for computing surface
normal vectors we used approximately 1000 training data
sets, 400 training iterations, and a spread constant of 0.4.
The training took approximately 15 minutes for each ofthese networks and the estimation of both the surface normal vectors and the color reflectance coefficients tookapproximately 10 seconds in each case.
Figures 19(a) to 19(d) show four of the images of theobject used as input. The object used in these experiments
was a glazed ceramic container. The surface of the sphereused in training was coated with the same type of paint inorder to create the same reflectance properties. These experiments were performed with the assumption that interreflections and cast shadows did not occur.
Figures 12 and 19(e) show the results of the neuralnetwork-based photometric stereo. Color diffuse reflectance coefficients and surface normal vectors are estimatedalso for a real object.
However, in the clothing or wrinkles seen in humanforms there are regions that are locally concave and therefore the effects of interreflection may arise; as a result there
may be some slight errors in the neural network-based
photometric stereo. In addition, parts that have shadows
cast on them by the nose or the cheekbones are also theoretically in a similar manner. This is because no secondaryreflections or cast shadows will arise on the sphere used intraining. The problem of cast shadows is another issue andis not dealt with in this paper; however, it would be possible
to obtain some improvements by also applying a method
such as the illumination planning method proposed by
Fukui and colleagues [14].

5.2. Experiments using neural network-based
color photometric stereo with real objects
Figure 17(a) shows one of the images of a sphere used
for training; this image was decomposed in its diffuse
reflection components [Fig. 17(b)] and its specular reflection components [Fig. 17(c)] and color reflectance coefficients were added to it at random [Fig. 17(d)] and then theneural network training was performed. In the training ofboth the neural network for computing color reflectance

5.3. Evaluation of the accuracy of neuralnetwork-based rendering using spheres
Next we evaluated the accuracy of our proposed
method for neural network-based rendering. Since it is notpossible to perform a quantitative evaluation with a realFig. 11. Relation between the precision of decomposingof components and the estimated results.

55

Fig. 13. (a) Real image of sphere. (b) Image estimatedby proposed method, (c) estimated by Torrance–Sparrowmodel, and (d) estimated by Phong model.

Fig. 15. Residual between real and virtual image withan arbitrary light source direction.

representing sharpness of the specular reflections in the
Phong model was n = 289.
The results of rendering with the proposed model, theTorrance–Sparrow model, and the Phong model are shownin Figs. 13(b) to 13(d). Figure 14 shows the results ofdisplaying the brightness distribution for the specular reflection regions of each of the images in terms of height.
The mean error in the specular reflection region for the
proposed model was 0.005 while its variance was 0.075.
The mean error for the Torrance–Sparrow model was 0.093
with a variance of 0.102; the mean error of the Phong model
was 0.102 with a variance of 0.058.
The results of the evaluation were that the proposed
method generated quantitatively better results than the
Phong model and the Torrance–Sparrow model.

object of unknown shape, we used a known shape, a sphere,
to perform these experiments and compared the renderingresults with the Torrance–Sparrow model and the Phongmodel. Figure 13(a) shows real images taken of the sphereused here. The light source direction l here is l = (0.36, 0.31,
0.88); this is not one of the directions used during thetraining of the neural network. During the training of theneural network we use images of the sphere taken withseven different light source directions. The training dataconsist of approximately 1000 examples, the number oftraining iterations was 200, and the spread constant was set
to 0.4. The training took approximately 10 minutes. Given
the results obtained from the neural network-based color
photometric stereo, we estimated the parameters for the
Torrance–Sparrow model and the Phong model by using theleast squared method on the seven images of the sphere used
for training the neural network. It was assumed that theseparameters were constant over the whole sphere. The geometric decay coefficient was estimated as G = 1, the Fresnel
reflection coefficient as F = 1.06, and σ representing theroughness of the object’s surface as 0.0272. The parameter

5.4. Experiments generating virtual images
with an arbitrary light source
We performed experiments generating images of thetarget object with a light source direction that was entirely
different from the directions used in the images of spheres

Fig. 14. Specular reflection region. (a) Real image; (b)proposed method; (c) Torrance–Sparrow model; (d)Phong model.

Fig. 16. Height distribution.

56

Fig. 17. (a) Image of a sphere used for training. (b) Diffuse reflection component. (c) Specular reflection component.
(d) Example with randomly added reflection coefficient.

Fig. 18. (a–d) Four input images; (e) the resulting color reflectance coefficient. [Color figure can be viewed in the online
issue, which is available at www.interscience.wiley.com.]

Fig. 19. (a–d) Four input images; (e) the resulting color reflectance coefficient. [Color figure can be viewed in the online
issue, which is available at www.interscience.wiley.com.]

Fig. 20. (a) Virtual image generated by the proposed method; (c) Torrance–Sparrow model virtual image; (d)–(f) virtual
images from arbitrary viewpoints; (b, g) real images. [Color figure can be viewed in the online issue,
which is available at www.interscience.wiley.com.]57

generalization capacity of the neural network-based reflection characteristic function working effectively.

for training. The light source direction of the image generated was l = (–0.55, 0.35, 0.75) (this was not a light source
direction that occurred in the training data).
Figure 20(a) shows the results of rendering with theproposed method. The time for rendering an image of 340
× 160 pixels was approximately 2 to 3 seconds. Figure 20(b)shows the real image with this light source direction. The
difference between Figs. 20(a) and 20(b) is shown in Fig.
15. The average error between Figs. 20(a) and 20(b) is 0.073
with a variance of 0.067. For comparative purposes weshow the results of rendering with the Torrance–Sparrowmodel in Fig. 20(c). The average error between Figs. 20(c)and 20(b) is 0.140 with a variance of 0.185.
Slight differences between the image generated by
the proposed method, as with the other models, can be seen
when there are local interreflections and cast shadows.
Except for these points, it is confirmed that the trainedneural network is able to generate virtual images that havea different light source direction than that of the trainingdata.

6. Considerations from a Practical
Perspective
The proposed method has extended a previous
method to allow it to be applied to color images, but in its
requirement that a sphere with the same reflectance properties as the target object be prepared for the training of theneural network it remains unchanged. Either of the following methods could be used to prepare a sphere for training
purposes.
• Creating a sphere from the same material as thetarget object• Painting the sphere and the target object with thesame paintIf the composition and the material used in the target object
are known, then it is possible to create a sphere with thesame material. In addition, if the surface of the target object
is coated in paint, then we can create the same surface
reflectance properties by coating the sphere in the same
paint. However, there will be cases where we do not know
the material from which the object is made and the surfaceof the object cannot be coated in paint. For objects for which
we cannot create a sphere to be used as reconstruction
targets, we require a method for creating training data that
does not require the preparation of a sphere.
In previous work, self-calibration neural networkbased photometric stereo has been proposed as an attempt
at finding a method that does not require a sphere [16].
Self-calibration is a method for generating training data forthe neural network based on the application of geometricconstraints and optical constraints on multiple images taken
while rotating the object around one of its axes. However,
the training data generated by this method is designed to be
used with monochrome neural network-based photometricstereo [8] and it would be difficult to apply it directly to themethod presented here. From a practical perspective, basedon the approach presented in Ref. 16, the development of amethod that does not require a sphere for training is an
important issue and one we leave for future work.

5.5. Experiments generating virtual images
with arbitrary viewpoints
Figure 16 shows the height distribution of the object
obtained by integrating the surface normal vectors obtained
from the neural network-based photometric stereo. From
the results of neural network-based color photometric stereo for the target object in various different poses, we
computed the integrals and performed a reciprocal rotationof each height distribution and examined the concordance
rate of these; from the fact that for areas with continuous
height the error was within 3 percent, we judged that theseheight distributions that we had integrated and created werea suitable shape. We then rotated these height distributions
to generate images in different poses.
We set the light source direction as l = (0.40, –0.05,
0.92) and generated virtual images with the viewing direction being varied; these results are shown in Figs. 20(d) to20(f). Figure 20(g) shows the actual object after a rotation(the viewpoint was changed); Fig. 20(f) is a virtual image
for the corresponding viewpoint. The results of evaluating
the error using Eq. (16) for Figs. 20(d) to 20(f) were an
average error of 0.059 with a variance of 0.010. On the other
hand, the Torrance–Sparrow model results in an average
error of 0.084 with a variance of 0.017. Based on a comparison of the precision of our proposed method and that inRef. 8, we have obtained better results for both the surface
normal vectors and the surface reflectance coefficients forour method. In addition, as shown by Fig. 14 and the resultsof the precision comparison with the Torrance–Sparrowmodel, realistic virtual images can be generated due to the

7. Conclusion
In this paper we have extended neural network-based
photometric stereo to allow its application to color images
and presented a method for estimating the surface normal
vectors of a target object as well as its color reflectancecoefficients. This method does not make use of a parametric58

function to characterize the reflection of the object simply
using the general assumption of a dichromatic reflection
model. In addition, by splitting the neural network into twostages, we have been able to avoid an explosion in theamount of combinations that must be used as training data,
and been able to perform training with the same scale oftraining data and the same amount of time as the previous
method while achieving higher estimation accuracy.
In addition, based on the results of neural networkbased color photometric stereo, we have proposed a neuralnetwork-based rendering method to enable the generationof virtual images of an object with an arbitrary light source
direction and from an arbitrary viewpoint. The results ofcomparisons of these virtual images that we have generated
with the actual images, confirmed that we were able torender specular reflections with a higher level of accuracy
than models using a parametric reflectance function.
However, since the proposed method assumed thatcast shadows and interreflections do not occur on the target
object, these problems are left to future work. Besides this,future work will include reconstruction and virtual image
generation for the entire circumference of the object and thedevelopment of a method that does not make use of a spherein training.

relative rotations of light source. Trans IEICE
2000;J83-D-II:927–937. (in Japanese)
5. Woodham RJ. Photometric method for determining
surface orientation from multiple images. Opt Eng, p
139–144, 1980.
6. Woodham RJ. Gradient and curvature from thephotometric-stereo method, including local confidence estimation. J Opt Soc Am A, p 3050–3068,
1994.
7. Iwahori Y, Woodham RJ, Ozaki M, Tanaka H, Ishii
N. Neural network based photometric stereo with anearby rotational moving light source. IEICE TransInf Syst 1997;E80-D:948–957.
8. Iwahori Y, Woodham RJ, Bhuiyan MS, Ishii N. Neural network based photometric stereo for object with
non-uniform reflectance factor. Proc 6th International Conference on Neural Information Processing(ICONIP’99), Vol. III, p 1213–1218.
9. Tomita Y, Kaneko S, Honda T. 3-Dimensional shapereconstruction using color photometric stereo. Technical Report of the SIG Computer Vision of theInformation Processing Society of Japan, 080-016, p
115–120, 1992. (in Japanese)
10. Barsky S, Petrou M. Color photometric stereo: Simultaneous reconstruction of local gradient and colorof rough textured surfaces. Eighth IEEE Int Conf on
Computer Vision, Vol. II, p 600–605, 2001.
11. Ikeuchi K, Sato Y, Nishino W, Sato I. Creating opticalconsistency for mixed reality images. Journal of the
Virtual Reality Society of Japan, Special Issue on
Mixed Reality 1999;4:623–630. (in Japanese)
12. Tominaga M, Ohashi S. A model of object color
reflectivity. J Inf Process Soc Japan 1992;33:37–45.
(in Japanese)
13. Chen S, Cowan CFN, Grant PM. Orthogonal leastsquares learning algorithm for radial basis function
networks. IEEE Trans Neural Networks 1991;2:302–309.
14. Fukui S, Iwahori H, Woodham RJ, Iwada A. Classifying curves from monochrome image using neuralnetworks and illumination planning. Trans IEICE
2000;J83-D-II:610–622. (in Japanese)
15. Iwahori Y, Kawanaka H, Woodham RJ, Funahashi K.
Neural network based modeling and color rendering
for mixed reality. Proc 17th International Conference
on Pattern Recognition (ICPR2004).
16. Iwahori Y, Watanabe Y, Woodham RJ, Iwata A. Selfcalibration and neural network implementation ofphotometric stereo. Proc 16th International Conference on Pattern Recognition, 2002.

Acknowledgments. We thank Professor S. Sato of
Nagoya Institute of Technology for help and advice during
this work. This research was supported by a Scientific
Research Grant [Basic Research (C) (2) 16500108], theKayamori Foundation of Informational Science Advancement (Yuji Iwahori), the Hori Information Science Promotion Foundation (Kenji Funahashi), and the Institute for
Robotics and Intelligent Systems and the Natural Sciencesand Engineering Research Council (Robert J. Woodham).

REFERENCES
1. Sato Y, Ikeuchi K. Temporal-color space analysis ofreflection. J Opt Soc Am A 1994;11:2990–3002.
2. Sato Y, Wheeler MD, Ikeuchi K. Object shape andreflectance modeling from observation. Proc ACM
SIGGRAPH 97, p 379–387.
3. Nishiyama Y, Saito H, Ozawa S. Estimation of 3-dimensional shape and object color from multiple
viewpoint color images. Trans IEICE 1999;J82-DII:990–1000. (in Japanese)
4. Omata K, Saito H, Ozawa S. Estimation of object
shape and surface reflectivity characteristics using

59

AUTHORS (from left to right)

Haruki Kawanaka (full member) graduated from the Department of Engineering, Electrical Engineering and Information Science, Nagoya Institute of Technology, in 1999 and completed the master’s and doctoral programs in electrical engineeringand information science in 2001 and 2005. Currently he is a postdoctoral researcher in the Center for Advanced Measurement
Research at Chubu University. His research interests are in electron microscopy, neural networks, computer vision, and mixed
virtual reality. He holds a D.Eng. deree, and is a member of the Institute of Image Electronics Engineers of Japan.
Yuji Iwahori (full member) graduated from the Department of Information Sciences at Nagoya Institute of Technology
in 1983, completed the master’s and doctoral programs in electronics and electrical engineering at Tokyo Institute of Technology
in 1985 and 1988, and became a research associate in the Center for Information Processing Education at Nagoya Institute ofTechnology. He was appointed a professor at the Center for Information and Media Studies in 2002. He has been a professor
in the Department of Computer Science at Chubu University since 2004, and also a visiting researcher at the University of
British Columbia. His research interests are in computer vision, neural networks, and mixed reality. He holds a D.Eng. degree,
and is a member of the Information Processing Society of Japan and IEEE.
Robert J. Woodham graduated from the Department of Mathematics at the University of Western Ontario in 1971 andcompleted the master’s program in mathematics and electrical engineering at Massachusetts Institute of Technology (MIT) in
1974 and the doctoral program at the same university (AI Lab) in 1977. After serving as a doctoral researcher at MIT, he joined
the University of British Columbia in 1978, and has been a professor of computer science there since 1990. His research interests
are in computer vision and pattern recognition. He holds a Ph.D. degree, and is a member of IEEE.
Kenji Funahashi (full member) graduated from the Department of Electrical Engineering and Information Science atGifu University in 1993, completed the master’s and doctoral programs in information engineering at Nagoya University in
1995 and 1998, and became a research associate at the Center for Information Processing Education at Nagoya Institute ofTechnology. He moved to the Center for Information Media Education in 2000, and has been an associate professor there since2005. His research interests are in virtual reality and computer graphics. He holds a D.Eng. degree, and is a member of theSociety for Virtual Reality of Japan.

60