Linear estimation of 4-D illumination light fieldfrom diffuse reflections
Takahito Aoto∗ , Tomokazu Sato∗ , Yasuhiro Mukaigawa† and Naokazu Yokoya∗∗ Nara Institute of Science and Technology, Japan{takahito-a, tomoka-s, yokoya}@is.naist.jp† ISIR, Osaka University, Japan
mukaigaw@am.sanken.osaka-u.ac.jp(a) Flashlight with lensand reflector
Fig. 1.

(b) Setup for estimation(c) Captured image(d) Relighting result
using 4-D light field(e) Relighting result
using 2-D light field

Experimental setup and comparison of experimental results.

Abstract—This paper proposes a linear estimation method for
4-D illumination light field using an inverse lighting framework.
Although lighting environments have conventionally been modeled by the combination of point light sources or a spherical lightfield map, complex light sources such as an ordinary flashlightwith lenses and reflectors, can not be correctly modeled bythese approaches. In order to correctly model these complex
light sources, we employ 4-D illumination light field. Unlike
conventional work, we decompose observed intensities on adiffuse-reflection board into intensities of 4-D light rays bysolving a linear system. We validate the effectiveness of the
proposed method through the experiments on real and virtualenvironments.

I.

I NTRODUCTION

Estimation of illumination is useful for many applications,e.g. photorealistic image synthesis, photometric stereo andBRDF estimation. For this purpose, conventional methods
often employ the point light source model (Fig. 2 (a)) for
its simplicity [1], [2]. However, as shown in Fig. 1, themethods using these simple illumination models do not workfor complex light sources including an ordinary flashlight withlenses and reflectors, due to its simplicity.
One simple extension is to allow different radiance for adifferent direction from a single point light source or for asingle observation point (see Fig. 2 (b)). In this case, radiancein different direction is represented as a 2-D map on a sphere(here, we call this model as a 2-D light field) [3], [4]. By using
the 2-D light field, we can accurately model a light source withstrong directivity and a certain volume that cannot be ignored
e.g. LED. This model is also useful for modeling complex
lighting environments for a certain 3-D position.
However, in order to completely model the lighting environments, the 4-D light field (see Fig. 2 (c)) is necessary.
Conventional approaches using the 4-D light field model sufferfrom the cost for measuring all rays. One straightforward

Fig. 2. Illumination models. a) parallel light source and isotropic point lightsource, b) 2-D light field, c) 4-D light field.

method for measuring the 4-D light field is to capture images
for all the directions from all 3-D positions. Actually, there
exist many researches that directly capture a huge number ofimages using a camera mounted on a robot arm [5]–[8]. In
order to reduce the cost for measurement, the mirrors [9],lens array [10], and filter [11] are used in the literatures.
One of the problems in these direct methods is the difficultyin estimation of missing rays which have not captured in
the images. On the other hand, inverse lighting methods
estimate the light field by analyzing captured images [2], [12].
In these methods, direct observation of a reference object
is unnecessary; instead, they back-trace the rays from thecamera to the light source using the known shape of thereference object and its reflection parameters. Although manymethods for inverse lighting have been proposed in recentyears, simple illumination models are employed in most ofthese works [2]. To the best of our knowledge, there exists nowork that estimate the 4-D light field using an inverse lighting
framework. In this paper, we show that radiance parameters of
the 4-D light field emitted from one complex light source canbe estimated using an inverse lighting framework by solving

a linear system. However, the naive approach has a drawbackin computational cost and stability. We propose a solution forthese problems using the real spherical harmonics function andvisibility masks.
II.

"ρ ; if ray j hits X.
0 ; otherwise(2)s(j) is intensity of ray j and ρ is reflectance of M. It
should be noted that actual irradiance i(X) is represented as aweighted sum of intensities of rays, since 4-D light field is discretized with parameters (u, v, θ, φ). The relationship between
observations and 4-D light field is given. By concatenating nobservations of i(Xn ), the following equation is derived fromEqs. (1) and (2) as a linear system,(3)

i = As,where,AFig. 3.

Light rays that hit a diffuse-reflection board.

is represented by a weighted sum of the bases of real sphericalharmonics function, which is represented bys(θ, φ) =L )+l)

(4)cl,m yl,m (θ, φ).

l=0 m=−laX (j) =iscamera

BASIC I DEA

In this paper, we estimate a 4-D light field for one complex
light source using multiple images in which an illuminateddiffuse reflection board is captured from various positions,while camera position and light position are fixed. Figure 3
illustrates a scene where light rays pass through virtual plane
L and hit diffuse-reflection board M. Here, 4-D light field ismodeled as the intensities of rays defined by the parametersof position (u, v) and direction (θ, φ). In this representation,
irradiance i(X) in the local region X on M is proportional
to the integral of all intensities of rays that hit X as follows:!
i(X) =aX (j)s(j)dj,(1)j= (u, v, θ, φ)T ,whereLightg sources===(i(X1 ), i(X2 ), . . . , i(Xn ))T(s(j ), s(j2 ), . . . , s(jm ))T⎞⎛ 1aX1 (j1 ), . . . , aX1 (jm )⎜ a (j ), . . . , a (j ) ⎟X2 m ⎟⎜ X2 1
⎟.
⎜
..
..
..
⎟⎜
.
⎠
⎝
.
.
aXn (j1 ), . . . , aXn (jm )In order to achieve efficient estimation using a limited
number of unknown parameters, we introduce visibility mask
for rays. In this system, a large number of rays actually do notaffect observed intensities on the camera due to two reasons:(1) ray does not hit the board, (2) ray hits the board at outsideof the view. These rays can be easily determined and masked inour system by detecting dark regions on the captured images.
By introducing the real spherical harmonics function andvisibility constraint, Eq. (4) can be transformed as follows:∈ Rn ,∈ Rm ,

Because all parameters for A and i can be determined from theobservations, theoretically, we can linearly solve this systemby the least squares method if the number of n observed
intensities is larger than that of m unknown parameters.
III.

Here, there exist k = (L+1)2 different weight parameters cl,mand they are unknown parameters. Representable directivity ofrays depends on the number of k parameters. It should be notedthat although more complex light field can be reconstructed as
k increases, the size of the system also becomes larger.

REDUCTION OF SYSTEM

In practice, the size of A is too large to be solved due to thelarge number of unknown parameters. In order to reduce thesize of the linear system in Eq. (3), in this paper, we employ thereal spherical harmonics function and the visibility constraint.
First, in order to reduce the number of unknown parameters,
radiance intensities emitted from a position (u, v) on virtual
plane L are represented using the real spherical harmonics
function. More specifically, s for the ray from local region X

Yi,j :=*i = Y c,(5)TT Tc = (cT
1 , c2 , . . . , c t ) ,(6)0; if (s(jm ) = 0)yl,m (θ,φ)R(ϑ)
d2; otherwise,(7)

where t is the number of discretized region on L, Yi,j isan element of matrix Y , R(ϑ) is a reflectance distributionfunction, ϑ is determined by the angle between a normal of
the board and the ray, and d is the distance between (u, v) andX on M.
IV.

E XPERIMENT

To validate the effectiveness of the proposed method, weestimate the 4-D light field using the synthesized and real data
set. For all experiments, we used a calibrated camera with aresolution of 480×360 pixels with 16-bit color depth. With afixed camera and light source, we captured multiple images of
the Lambertian board at various distances from the camera.

!"#$%$"&'#(%%$&)#'"*'+$**,#(-.(*/(0%$"&'1"2.+',)**+"##

3$.%,2/'4/2&('$'#!!# !&#,)**+$##!

#()**+!"#$%"!$##!$# !"#$##!%#"##!"##"%#(a)&##!$%# ()**+!$'#(b)Fig. 4. Experimental settings (a) layout of camera, light source, and diffuse board. (b) positions of directional light sources on virtual plane L. The blue andred lines indicate the positions of diffuse-reflection board for captured images. Only images for blue positions are used for input and the others are used forcomparison. The yellow circles indicate the position of the directional light sources.

190 [mm]

210 [mm]

230 [mm]"#!
Fig. 5.

240 [mm]$

270 [mm]

300 [mm]!"

Captured images and given light field map for generating input images. Radiance of light field map is normalized to 0 - 1.

A. Light field estimation in a simulation
In this experiment, we estimate the 4-D illumination lightfield using the simulation environment. Figure 4 (a) shows thesetup of a camera, light sources and a board in this experiment.
As input, four out of seven images are used. From the 3D position of yellow circles in Fig. 4 (b), strong directional
lights, which are five alphabets ’L-G-I-H-T’, are projected tothe board so that they can be read as ’L-I-G-H-T’ with different
disparity on the projected images as shown in Fig. 5. By using
these synthesized images, we have estimated the light field forthe position of grid points shown in Fig. 5. Figure 6 compares
relighted images using estimated light field with the following
three methods.
(i)

4-D light field estimation with visibility constraint,(ii)

4-D light field estimation without visibility constraint,(iii)

2-D light field estimation without visibility constraint.

For the method (iii), coefficient parameters c of the real
spherical harmonics function are estimated for the 3-D position(250, 0, 100). Except for this, common parameters are usedand dimension k of the real spherical harmonics function isset to 441 in this experiment. By comparing generated images
and ground truth shown in Figs. 5 and 6, we can confirm that
relighting images generated by the proposed method (i) are thebest among the compared methods others. As we can see (ii)in this figure, relighted images without visibility constraints
are blurred and the words are unreadable for 190mm and300mm, which are outside of the depth range of the board forinput images. This is caused by the shortage of the availableparameters for the real spherical harmonics to adjust all theobserved intensities. As we can see in (iii), 2-D light fieldcannot produce good relighting images for complex light field.
Each circle in Fig. 6 shows estimated light field map foreach position. From the results for the methods (i) and (ii), wecan confirm that estimated radiance intensities in (i) for eachposition and direction are more accurate than that in (ii).
B. Light field estimation in real scene
We have also compared three methods in real environment.
In this experiment, the flashlight shown in Fig.1 is used as a190 [mm]210 [mm]

230 [mm]

240 [mm]

270 [mm]

300 [mm]

(i)

(ii)

(iii)

Fig. 6. Relighted images by estimated light fields from compared methods. (i): 4-D light field estimation with visibility constraint. (ii) 4-D light field estimation.
(iii) 2-D light field estimation.

(i)

(ii)

"#!
Fig. 7.

(iii)

$!"

Output light field map. (i)-(ii) correspond Fig. 6. Estimated light field maps for each grid point.

light source. Relative pose of the board and the camera isestimated using attached markers.
Figure 8 shows the captured images where the board is set
at different distances from the camera (400, 500, 600, 700mm).
In this experiment, we set a virtual plane L in front of theflashlight and we have estimated light map for grid points of
5×5 regular grid where the size of each grid is 20mm×20mm.
Figure 9 shows relighted images using estimated light field
for the compared methods. Fig. 10 shows estimated light field
map with k = 121. As we can confirm from these figures,the proposed method works even for complex light field in
real environment, and visibility mask is very important toacquire stable result. However, we can see that there exist
many negative values in estimated light field in this figure. It isconsidered to be appeared due to the lack of valid observations
for solving linear system. A solution for removing negativevalues should be investigated in future work for estimating
4-D light field more accurately.
V.

C ONCLUSION

In this paper, we have proposed a method for estimating 4D illumination light field defined for one complex light source.

Unlike conventional approaches, the proposed method does not
require a huge number of images for direct observation of lightsources to estimate 4-D light field as shown in experiments.
Our approach decompose the observed intensities on diffusereflection board into radiance intensities of light rays withlinear solution. We have introduced spherical harmonics andvisibility mask for stable solution of the system and theireffectiveness is validated through the experiments. Estimated
4-D light field will be used to develop a more practical 3-Dreconstruction technique with photometric stereo in next stage.
ACKNOWLEDGMENT
This research is supported in part by MEXT (Japan)
KAKENHI 23240024 and the Japan Society for the Promotionof Science (JSPS) through the Funding Program for Next
Generation World-Leading Researchers.
R EFERENCES[1] T. Aoto, T. Taketomi, T. Sato, Y. Mukaigawa, and N. Yokoya, “Positionestimation of near point light sources using clear hollow sphere,” inProc. IAPR ICPR2012, 2012, pp. 3721–3724.

(a) 400 mm
Fig. 8.

(b) 500 mm(c) 600 mm(d) 700 mm

Captured images in a real situation.

400 [mm]

500 [mm]

600 [mm]

700 [mm]

(i)

(ii)

(iii)

Fig. 9. Relighting images by estimated light fields from compared method in real scene. (i): 4-D light field estimation with visibility constraint. (ii) 4-D lightfield estimation. (iii) 2-D light field estimation.

(i)

(ii)

"#!
Fig. 10.

(iii)

$!"

Estimated light field map for each grid point in real scene.

[2] T. Takai, S. Nimura, A. Maki, and T. Matsuyama, “Self shadows andcast shadows in estimating illumination distribution,” in Proc. European
Conf. on Visual Media Production, 2007, pp. 1–10.

[9][3] C. Verbeck and D. GreenBerg, “A comprehensive light source description for computer graphics,” IEEE Computer Graphics and Applications, vol. 4, no. 7, pp. 66–75, 1984.

[10][4] P. Debevec and J. Malik, “Recoverring high dynamic range radiancemaps from photo graphs,” in Proc. ACM SIGGRAPH, 2008, pp. 369–378.

[11][5] I. Ashdown, “Near-field photometry : A new approach,” Journal of theIlluminating Society, vol. 22, no. 1, pp. 163–180, 1993.
[6] M. W. Seigel and R. D. Stock, “A general near-zone light source
model and its application to computer automated reflector design,” SPIE
Optical Engineering, vol. 35, no. 9, pp. 2661–2679, 1996.
[7] R. F. Rykowski and C. Wooley, “Source modeling for illumination
dsign,” In Lens Design, Illumination, and Optomechanical Modeling,
SPIE, vol. 3130, pp. 204–208, 1997.
[8] D. R. Jenkins and H. Monch, “Source imaging goniometer method of[12]light source characterization for accurate projection system design,” inProc. of SID Conf., no. 862–865, 2000.
J. Unger, A. Wenger, T. Hawkins, A. Gardner, and P. Debevec, “Capturing and rendering with incident light fields,” in Proc. Eurographics
symposium on rendering, 2003, pp. 141–149.
O. Cossairt, S. K. Nayar, and R. Ramamoorthi, “Light field transfer:
Global illumination between real and synthetic objects,” ACM Trans.
on Graphics, vol. 27, no. 3, pp. 57:1–57:6, 2008.
M. Goesele, X. Granier, W. Heidrich, and H.-P. Seidel, “Accurate
lightsource acquisition and rendering,” in Proc. ACM SIGGRAPH, 2003,
pp. 621–630.
I. Sato, Y. Sato, and K. Ikeuchi, “Illumination from shadows,” IEEE
Trans. Pattern Analysis and Machine Intelligence, vol. 25, no. 3, pp.
290–300, 2003.