VOL. E99-D NO. 9
SEPTEMBER 2016
The usage of this PDF file must comply with the IEICE Provisions
on Copyright.
The author(s) can distribute this PDF file for research andeducational (nonprofit) purposes only.
Distribution by anyone other than the author(s) is prohibited.

IEICE TRANS. INF. & SYST., VOL.E99–D, NO.9 SEPTEMBER 2016

2360

PAPER

Acquiring 4D Light Fields of Self-Luminous Extended Light
Sources Using Programmable Filter
Motohiro NAKAMURA† , Nonmember, Shinnosuke OYA† , Student Member, Takahiro OKABE†a) , Member,and Hendrik P. A. LENSCH†† , Nonmember

SUMMARY
Self-luminous light sources in the real world often havenonnegligible sizes and radiate light inhomogeneously. Acquiring themodel of such a light source is highly important for accurate image synthesis and understanding. In this paper, we propose an approach to measuring 4D light fields of self-luminous extended light sources by using aliquid crystal (LC) panel, i.e. a programmable optical filter and a diﬀusereflection board. The proposed approach recovers the 4D light field fromthe images of the board illuminated by the light radiated from a light sourceand passing through the LC panel. We make use of the feature that thetransmittance of the LC panel can be controlled both spatially and temporally. The approach enables multiplexed sensing and adaptive sensing, andtherefore is able to acquire 4D light fields more eﬃciently and densely than
the straightforward method. We implemented the prototype setup, and confirmed through a number of experiments that our approach is eﬀective formodeling self-luminous extended light sources in the real world.
key words: 4D light field, extended light source, programmable filter, multiplexed sensing, adaptive sensing

1.

Introduction

The appearance of an object depends not only on the geometric and photometric properties of the object but also onlight sources illuminating the object. Therefore, acquiring
the models of self-luminous light sources is highly important in the fields of computer graphics and computer vision,in particular for photorealistic image synthesis and accurate
image-based modeling.
Conventionally, in the field of computer vision, mostoften ideal light sources such as directional light sources(point light sources at infinity) and isotropic point lightsources are assumed for photometric image analysis. Unfortunately, however, this is not the case; an object of interest is often illuminated by nearby light sources, and more
importantly, self-luminous light sources in the real world often have nonnegligible sizes, i.e. they are considered to beextended light sources and radiate light inhomogeneously.
This means that the illumination distribution seen from apoint on an object surface varies over the surface. Therefore, in order to analyze the shading observed on an object
surface under extended light sources, we need to acquire theManuscript received February 10, 2016.
Manuscript revised May 18, 2016.
Manuscript publicized June 17, 2016.
†
The authors are with the Department of Artificial Intelligence,
Kyushu Institute of Technology, Iizuka-shi, 820–8502 Japan.
††
The author is with the Computer Graphics Group, Eberhard
Karls Universität Tübingen, Tübingen, Germany.
a) E-mail: okabe@ai.kyutech.ac.jp
DOI: 10.1587/transinf.2016EDP7068

Fig. 1 (a) One of the representations of a 4D light field, and (b) our proposed setup consisting of an LC panel and a diﬀuse-reflection board.

radiant intensity distributions of the light sources.
The radiant intensity distribution of an extended lightsource is described by a 4D light field [3] which is represented by the amount of light passing through the pair ofpoints (x, y) and (s, t) on two planes in general position asshown in Fig. 1 (a). The diﬃculty in acquiring the radiantintensity distribution of an extended light source is that weneed to measure a wide range of the light field. Note that
consumer light field cameras are not suitable for such a purpose because of their limited entrance pupil. To cope withthis problem, Goesele et al. [4] propose a setup consistingof a static optical filter and a diﬀuse-reflection board, anddemonstrate the eﬀectiveness of the setup for modeling extended light sources.
In this paper, we propose an eﬃcient approach to acquiring the radiant intensity distribution of a self-luminous
extended light source by using an LC panel, i.e. a programmable optical filter and a diﬀuse-reflection board asshown in Fig. 1 (b). The key idea of the proposed approachis to make use of the feature that the transmittance of the
LC panel can be controlled both spatially and temporally.
Specifically, the approach changes the transmittance patterns of the LC panel dynamically, and recovers the 4D lightfield of a self -luminous extended light source from the images of the board illuminated by the light radiated from thelight source and passing through the LC panel.
In the preliminary version of this paper [6], we propose a method based on multiplexed sensing [11], [12], [16],which is a well-known technique for increasing signal-tonoise ratio (SNR) without increasing measurement time, forc 2016 The Institute of Electronics, Information and Communication Engineers
Copyright NAKAMURA et al.: ACQUIRING 4D LIGHT FIELDS OF SELF-LUMINOUS EXTENDED LIGHT SOURCES

2361

acquiring 4D light fields more eﬃciently than the straightforward method. The proposed method divides an area ofinterest of the LC panel into small patches with the same
size, i.e. measures the 4D light field of an extended lightsource with uniform spatial resolution. Unfortunately, however, the measurement with uniform spatial resolution is notsuited for eﬃciently acquiring the 4D light field; the highfrequency components of the 4D light field are often lost,or the number of patches and measurement time increase inorder to acquire those high-frequency components.
Accordingly, we further make use of the feature of the
programmable filter, and propose a method for acquiring the
4D light field of a self-luminous extended light source with
non-uniform spatial resolution. Specifically, the proposed
method measures the 4D light field by using a small numberof patches with large size at first, and then iteratively divides
important patches into smaller ones and measures the 4D
light field adaptively by using them. That is, the proposed
method adaptively and non-uniformly increases the spatial
resolution according to the radiant intensity distribution of alight source of interest.
We implemented the prototype setup, and confirmed
through a number of experiments that our multiplexed
method can acquire the 4D light field of a self-luminous extended light source more eﬃciently than the straightforward
method. In addition, we confirmed that our adaptive method
can densely acquire the 4D light field by using a smaller
number of patches, i.e. in a shorter measurement time thanour multiplexed method. The main contribution of this paper is to demonstrate that our proposed approach using a
programmable filter is eﬀective for modeling self-luminous
extended light sources in the real world.
The rest of this paper is organized as follows. We
briefly summarize related work in Sect. 2. An approachto acquiring 4D light fields of self-luminous extended lightsources by using a programmable filter and a diﬀusereflection board is proposed in Sect. 3. We report the experimental results in Sect. 4 and present concluding remarks
in Sect. 5.
2.

Related Work

Existing techniques can be classified into 3 categories: (i)techniques for acquiring 2D radiant intensity distributions ofself-luminous point light sources, (ii) techniques for acquiring 4D light fields of self-luminous extended light sources,and (iii) techniques for acquiring 4D light fields of generalscenes. In this section, we briefly explain the existing techniques in each category, and then describe the relationshipbetween those techniques and our proposed approach.
Acquisition of 2D radiant intensity distributions of selfluminous point light sources
Since the size of a point light source is negligible, theradiant intensity distribution of a self-luminous point lightsource is described by a 2D function, i.e. a function withrespect to the direction seen from the center of the point

light source. Verbeck and Greenberg [15] propose a basic
method for measuring the 2D radiant intensity distributionsof anisotropic point light sources by using a goniophotometer. Their method can directly sample the radiant intensity
distribution of a light source by moving a sensor around thelight source. However, their method requires a large amountof measurement time because it samples the radiant intensity distribution only at a single direction at a time.
To cope with this problem, image-based techniques,
which can sample the radiant intensity distribution at alarge number of directions simultaneously, are proposed.
Rykowski and Kostal [10] propose an eﬃcient method formeasuring 2D radiant intensity distributions of LEDs by using the imaging sphere. They make use of the combination of a hemispherical chamber with diﬀuse coating and ahemispherical mirror, and capture the radiant intensity distribution with 2π steradian field of view at a time. Tan andNg [13] use a diﬀuse translucent sheet and a flatbed scanner,and Moreno and Sun [5] use a diﬀuse translucent screen anda camera for eﬃciently capturing the 2D radiant intensity
distributions of LEDs.
It is demonstrated that the above methods are useful for modeling real-world point light sources, in particular for inspecting LEDs. Unfortunately, however, wecannot use them for acquiring the 4D light fields of selfluminous extended light sources because they assume point
light sources, i.e. light sources with negligible sizes.
Acquisition of 4D light fields of self-luminous extended
light sources
As mentioned in the introduction, the radiant intensity
distributions of self-luminous extended light sources are described by 4D light fields. In a similar manner to Verbeck
and Greenberg [15], Ashdown [2] proposes a basic method
for measuring the 4D light field of a self-luminous extended
light source by using a goniophotometer. However, his
method requires a huge amount of measurement time because it samples the 4D distribution only at a single point in
the 4D space at a time.
To cope with this problem, image-based techniques areproposed also for measuring 4D light fields. Goesele etal. [4] propose a method for measuring the 4D light field of
a self-luminous extended light source by using an optical filter and a diﬀuse-reflection board as illustrated in Fig. 2 (a).
Their method recovers the 4D light field from the imagesof the board illuminated by the light radiated from the lightsource and passing through the optical filter. Although theirFig. 2 The rays radiated at two points on the surface of a self-luminous
extended light source are captured by each setup: (a) an optical filter and adiﬀuse reflection board, (b) a camera array, and (c) a light field camera.

IEICE TRANS. INF. & SYST., VOL.E99–D, NO.9 SEPTEMBER 20162362

method is suitable for measuring the 4D light field in a wide
range and works well with the optimally-designed opticalfilter, it is not easy to acquire the 4D light field eﬃciently
and densely because the optical filter is static and one has
to slide the position of the light source (or the optical filter)
manually during the measurement.
Aoto et al. [1] propose a method for recovering the 4D
light field of a self-luminous extended light source from theimages of a diﬀuse-reflection board moving in front of thelight source. Their method is unique in the sense that it
does not require any static or dynamic filters but uses only
a diﬀuse-reflection board. However, it would be diﬃcult
to stably recover the high-frequency components of the 4D
light field from the images of the diﬀuse-reflection board because diﬀuse reflectance behaves like a low-pass filter [9].
Acquisition of 4D light fields of general scenes
Other than the above techniques specialized for measuring the 4D light fields of self-luminous extended lightsources, there are a number of techniques for acquiring 4D
light fields of general scenes. Since it is impossible to cover
all the existing techniques due to limited space, we briefly
mention the advantages and limitations of some of representative approaches when they are used for measuring the 4D
light fields of self-luminous extended light sources.
One approach to general 4D light field acquisition is
to use a spherical mirror array [14] and a camera array [17].
Those methods have the advantage that they can measure thewide range of the 4D light field of an extended light source
as illustrated in Fig. 2 (b). However, they are not suited for
densely measuring the 4D light field because it is not easy
to place spherical mirrors and cameras densely.
Another approach to general 4D light field acquisition
is to use a single light field camera with a micro-lens array [7] and a coded aperture [8]. Those methods have theadvantage that they can measure the 4D light field of a selfluminous extended light source easily. However, they are
not suited for measuring the wide range of the 4D lightfield because of their limited entrance pupil as illustrated
in Fig. 2 (c). Note that the objective of our study is not to acquire the incoming intensity distribution to a small area in ascene but to acquire the outgoing intensity distribution froman extended light source. In general, light field cameras are
suited for the former purpose but are not suited for the latter
purpose.

Fig. 3 The cross section of our proposed setup. Two positions are usedin order to solve for two unknowns: the distance d and the radiant intensity
k of a light source.

by using the images of the diﬀuse-reflection board illuminated by the transmitted light. In order to solve for two unknowns as described below, we move the diﬀuse-reflection
board and observe the reflection of the transmitted light on
the board twice at the positions 1 and 2. Note that we assume that a light source radiates unpolarized light since thetransmittance of an LC panel depends on polarization state† .
We assume that a self-luminous extended light source
is approximately represented by a set of anisotropic point
light sources, and therefore the light passing through x
toward l comes from an unknown anisotropic point lightsource c. We denote the surface normal and distance ofthe board at the first position by n1 and r1 , and those at
the second position by n2 and r2 . We assume that the geometry of the setup is calibrated in advance, i.e. we assume
that those surface normals and distances are known. On theother hand, there are two unknowns; one is the distance d
between x and the point light source c, and the other is theradiant intensity k of the light source toward the direction
l. Our proposed method estimates those two parameters foreach (x, l) by using two radiances observed on the diﬀusereflection board at the positions 1 and 2.
When the diﬀuse-reflection board is placed at the first
position, the radiance i1 of the reflected light is given by††i1 = k3.

Proposed Approach

3.1

Light Source Model

Figure 3 shows the cross section of our proposed setup
which consists of a pair of an LC panel and a diﬀusereflection board. Actually, we place a light source of interestas close as possible to the LC panel so that we can acquire
the 4D light field of the light source in a wider range. Our
proposed method acquires the description of the light passing through a point x on the LC panel toward a direction l(−l) n1,
(d + r1 )2(1)

assuming the Lambertian model and the attenuation according to the inverse-square law††† . Similarly, when the board†
When a light source radiates polarized light, one could use adepolarizing filter in front of the LC panel.
††
In general, the transmittance of an LC panel depends on the
direction of incident light. Since we used an LC display with awide viewing angle of 165◦ in our experiments, we do not take theangle-dependency into consideration.
†††
This description is more complicated than that of the 4D lightfield because we take the distance from an anisotropic point lightsource into consideration.

NAKAMURA et al.: ACQUIRING 4D LIGHT FIELDS OF SELF-LUMINOUS EXTENDED LIGHT SOURCES

2363

for each image and then we need longer total measurement
time. Hence, the straightforward method has a tradeoﬀ between its accuracy and eﬃciency.
3.3 Multiplexed Method

Fig. 4 The filters for the straightforward method (top) and the multiplexed method (bottom). Here, n = 15 for display purpose.

is placed at the second position, the radiance i2 is given by
i2 = k(−l) n2.
(d + r2 )2(2)

Taking the ratio of Eq. (1) and Eq. (2), we can derive(d + r1 )2 i2 (−l) n1=≡ α.
(d + r2 )2 i1 (−l) n2
Thus, we can estimate one of the unknowns d as√
αr2 − r1
d=
√ .
1− α

(3)

(4)

Substituting Eq. (4) into Eq. (1) and/or Eq. (2), we can estimate the other unknown k.
3.2

Straightforward Method

The straightforward method for measuring the light field of
a self-luminous extended light source is to capture the images of the diﬀuse-reflection board at the first and second
positions by using a set of single filters shown in the top of
Fig. 4. Specifically, we divide an area of interest of the LC
panel into n square patches, and then set the transmittanceof a single patch to 1 and those of the other patches to 0 at
a time in turn. The advantage of using a programmable filter, i.e. an LC panel in our case, is that we can control thetransmittance both spatially and temporally without directmanual manipulation.
Unfortunately, however, such a straightforward method
has limitations. In order to acquire 4D light fields more
densely, we need to make the size of each patch smaller.
Since the transmittance of only a single patch is 1 in thestraightforward method, the smaller the size of each patch is,
the smaller the amount of light passing through the LC panel
and reflected on the diﬀuse-reflection board is. Therefore, if
we make the size of each patch smaller while keeping themeasurement time constant, the SNRs of the captured images decrease and then the accuracy of the recovered lightfield is also degraded. On the other hand, if we make thesize of each patch smaller while keeping the SNRs of thecaptured images constant, we need a longer exposure time

To cope with the limitations of the straightforward method,
we propose a multiplexed method that makes more use of
the feature that the transmittance of the LC panel can be controlled both spatially and temporally. Our method utilizes
multiplexed sensing [11], [12], [16], which is a well-known
technique for increasing SNR without increasing measurement time, and acquires 4D light fields more eﬃciently anddensely than the straightforward method.
Specifically, we use the multiplexed filters in which thetransmittances of about half of the patches are 1 and thoseof the other patches are 0 as shown in the bottom of Fig. 4,
and capture the images of the diﬀuse-reflection board illuminated by the transmitted light. We can obtain those n multiplexed filters by applying the so-called S -matrix, which isconstructed on the basis of the Hadamard matrix of order(n + 1), for n individual filters. In an opposite manner, we
can obtain the single filters by applying the inverse matrix
S −1 to the multiplexed filters. Therefore, by applying S −1 tothe captured images of the diﬀuse-reflection board under themultiplexed filters, we can obtain the decoded images under
the single filters. It is known that S −1 can be computed analytically: S −1 = 2(2S  − 1n )/(n + 1), where 1n is an n × nmatrix whose all elements are 1. See Sloane et al. [12] formore detail.
The ratio between the SNR of multiplexed sensing
SNRmulti and that of single sensing SNRsingle is at most√
SNRmultin,(5)SNRsingle
2
when n, i.e. the number of the patches in our case, is large
enough. Therefore, the proposed method based on multiplexed sensing can acquire 4D light fields more eﬃciently
and densely than the straightforward method while keeping
the SNR constant.
3.4 Adaptive Method
Both the straightforward method and multiplexed method
divide an area of interest of the LC panel into small patcheswith the same size, i.e. measure the 4D light field of a selfluminous extended light source with uniform spatial resolution. Unfortunately, however, extended light sources radiate light inhomogeneously, and therefore both the amountof light and the angular distribution of light passing througheach patch are not uniform across the LC panel. Therefore, the measurement with uniform spatial resolution is notsuited for eﬃciently acquiring the 4D light field; the highfrequency components of the 4D light field are often lost,or the number of patches and measurement time increase inorder to acquire those high-frequency components.

IEICE TRANS. INF. & SYST., VOL.E99–D, NO.9 SEPTEMBER 2016

2364

Accordingly, we propose an adaptive method for acquiring the 4D light field of an extended light source with
non-uniform spatial resolution. Specifically, the proposed
method measures the 4D light field by adaptively dividing
some important patches into smaller ones as follows.
1. Setting the initial number of patches to n0 and conducting the multiplexed measurement.
2. Selecting the most important m patches and dividing
each of them into 4 small patches. Thus, the number ofpatches is increased by 3m(= 4m − m).
3. Conducting the multiplexed measurement for the
newly-divided 4m patches. Here, the number of required images is 3m† .
4. Repeating 2 and 3 while the total number of patches isless than the predetermined number n.
We define the importance of a patch on the basis of theimage of the diﬀuse-reflection board when only the transmittance of the patch is 1. More specifically, we computethe discrete Fourier transform of the image and define theimportance as√|F(u, v)|2 u2 + v2 .
(6)u,vHere, u and v are the spatial frequencies, and |F(u, v)|2 is thepower spectral density of the image. That is, the importanceis the weighted√ average of the power spectral density withthe weight u2 + v2 for emphasizing high-frequency components. Therefore, the brighter the image is, and/or the
more high-frequency components the image has, the more
important the corresponding patch is.
3.5

Comparison between Multiplexed and Adaptive Methods

Then, we compare our multiplexed method and adaptive
method, and discuss their advantages and disadvantages
when the total number of patches is n. First, since the number of required images is equal to the number of patches,
the both methods require the same measurement time if the
exposure time for capturing each image is the same. Second, as to the computational cost, we compare the cost for
demultiplexing, that is the most time consuming computation in the both methods. In our multiplexed method, thecost is O(n2 ) since the product between a n × n matrix andn-dimensional vector is computed. In our adaptive method,
we iteratively divide important patches and conduct multiplexed measurement (n − n0 )/(3m) times, and therefore thedemultiplexing cost is O(n20 + (3m)2 × (n − n0 )/(3m)). Assuming that n  n0 and n0  m, the computational cost isO(nm). Since n  m, the computational cost of the adaptive
method is smaller than that of the multiplexed method. Finally, the gains of the SNRs of the multiplexed method and†

The amount of light passing through a single parent patch isequal to the amount of light passing through the corresponding 4
child patches.

√√
adaptive method are about n/2 and 3m/2 respectively
from Eq. (5). Therefore, the multiplexed method is better
than the adaptive method in terms of SNRs. If the SNR of
the adaptive method is not suﬃcient, we can conduct themultiplexed measurement with the adaptively divided filter
although additional time is required.
4.

Experiments

In our setup, we used an LC panel of LCD-AD172SE fromI-O DATA whose resolution and DPI are 1280 × 1024 and
96 respectively. The images of the diﬀuse-reflection board
were captured by using CMLN-13S2C-CS from Point Grey
with 1296 × 964 resolution.
4.1 Eﬀects of Multiplexed Method
To confirm the eﬀectiveness of multiplexed sensing, we
compared the images of the diﬀuse-reflection board captured and decoded by multiplexed sensing with those captured by single sensing. We used a fluorescent light locatednearby the LC panel and set the number of patches n to 63.
Because a small amount of light passes through the LC panel
even though the transmittance is set to 0, we captured animage when all the transmittances are set to 0 and then subtracted the image from all the images captured under the
single and multiplexed filters.
Figure 5 shows the example images of the diﬀusereflection board under a certain single filter taken with afixed exposure time. Here, the single filter is on the x-yplane and the diﬀuse-reflection board is on the s-t plane in
Fig. 1 (a). Therefore, those images show how the light passing through the single filter on one plane from various directions propagate to various positions on another plane. We
consider the average of 1000 images taken under the same
condition as the ground truth (a). We can see that the image captured by single sensing (b) is grained due to noise.
On the other hand, we can see that the image captured anddecoded by multiplexed sensing (c) is similar to the groundtruth. This result qualitatively shows that multiplexed sensing works better than single sensing.
In addition, we conducted quantitative evaluation. Figure 6 shows the RMS errors of the images of the diﬀusereflection board under all the single filters. We can see that

Fig. 5 The example images of the diﬀuse-reflection board under a single
filter: (a) the ground truth computed by averaging, (b) the captured image
by single sensing, and (c) the captured and decoded image by multiplexed
sensing. Pixel values are scaled for display purpose.

NAKAMURA et al.: ACQUIRING 4D LIGHT FIELDS OF SELF-LUMINOUS EXTENDED LIGHT SOURCES

2365

Fig. 6 The RMS errors of the images of the diﬀuse-reflection board under the single filters: captured by the single sensing (dotted line) and captured and decoded by the multiplexed sensing (solid line).

Fig. 7 The images of the diﬀuse-reflection board placed at (a) the position 1 and (b) the position 2 for measurement under two projectors.

the RMS errors of the captured and decoded images by multiplexed sensing (solid line) are always smaller than thoseof captured images by single sensing (dotted line) although
the gain of multiplexed sensing, i.e. SNRmulti /SNRsingle 
0.40/0.19  2.1 is smaller than the theoretical upper limit√
n/2  4.0. This result quantitatively shows that multiplexed sensing works better than single sensing.
Next, to demonstrate the eﬀectiveness of our multiplexed method, we acquired the 4D light fields of two different light sources and used them for image reconstruction.
As described in Sect. 3, we acquired the light fields from theimages of the diﬀuse-reflection board at the positions 1 and
2 by using the straightforward method and the multiplexed
method. Then, we reconstructed the images of the board at
two positions diﬀerent from those for measurement, say positions 3 and 4, when the transmittances of all the patches
are set to 1 by using the acquired light fields. Specifically,
the intensity of each pixel in the reconstructed image is computed by assuming that the corresponding surface point is illuminated by n anisotropic point light sources whose intensities and distances are estimated as described in Sect. 3.1.
In this experiment, we set the number of patches n to 255.
The first light source is two projectors: NP-L51WJD
from NEC with 1280 × 800 resolution. Figure 7 shows theimages of the diﬀuse-reflection board placed at (a) the position 1 and (b) the position 2 for measurement. The transmittances of all the patches are set to 1 for display purpose. We
can see that the characters of “Light” radiated from one projector cross the characters of “Field” radiated from anotherprojector.
Figure 8 shows the closeup images of the diﬀusereflection board at the positions 3 (top) and 4 (bottom): (a)the ground truth images and the reconstructed images by using (b) the straightforward method and (c) the multiplexed

Fig. 8 The reconstruction results on the two projectors. The closeup images of the diﬀuse-reflection board at the positions 3 (top) and 4 (bottom):(a) the ground truth image and the reconstructed images by using (b) thestraightforward method and (c) the multiplexed method.

Fig. 9 The images of the diﬀuse-reflection board placed at (a) the position 1 and (b) the position 2 for measurement under a single projector.

Fig. 10 The reconstruction results on the single projector. The closeup
images of the diﬀuse-reflection board at the positions 3 (top) and 4 (bottom): (a) the ground truth image and the reconstructed images by using (b)the straightforward method and (c) the multiplexed method.

method. We can see that both the straightforward method
and multiplexed method can capture how the characters radiated from the two projectors cross according to the distance from the projectors. Furthermore, we can see that theimages reconstructed by using the multiplexed method areless noisy than the images reconstructed by the straightforward method. Although some artifacts due to the discretization (the number of patches n = 255 is not necessarily large
enough) and errors in geometric calibration are still visible,
this result demonstrates that the proposed method works better than the straightforward method.
The second light source is a single projector. Figure 9shows the images of the diﬀuse-reflection board placed at(a) the position 1 and (b) the position 2 for measurement.
We can see that the characters of “Light” radiated from the
projector is in focus and out of focus depending on the distance from the projector due to a shallow depth of field.
Figure 10 shows the closeup images of the diﬀusereflection board at the positions 3 (top) and 4 (bottom): (a)the ground truth images and the reconstructed images by using (b) the straightforward method and (c) the multiplexed
method. We can see that both the straightforward method

IEICE TRANS. INF. & SYST., VOL.E99–D, NO.9 SEPTEMBER 2016

2366

Fig. 11 The filters used for acquiring the 4D light field of a projector byusing (a) the multiplexed method and (b) the adaptive method.

Fig. 12 The reconstruction results on the projector. The closeup imagesof the diﬀuse-reflection board at the positions 3 (top) and 4 (bottom): (a)the ground truth image and the reconstructed images by using (b) the multiplexed method and (c) the adaptive method. Note that the projector position
and the character size are diﬀerent from Fig. 10.

and multiplexed method can capture how the characters radiated from the projector blur according to the distance fromthe projector. This result shows that acquiring light fields is
eﬀective even for a single projector, which is often treatedas a light source with negligible size. Similar to the above,
we can see that the images reconstructed by using the multiplexed method are less noisy than the images reconstructedby using the straightforward method.
4.2

Eﬀects of Adaptive Measurement

To demonstrate the eﬀectiveness of our adaptive method, we
acquired the 4D light fields of two diﬀerent light sources andused them for image reconstruction in a similar manner toSect. 4.1. In this experiment, we set n = 255, n0 = 15, andm = 4.
The first light source is a single projector. Figure 11
shows the filters used for acquiring the 4D light field of the
projector by using (a) the multiplexed method and (b) the
adaptive method. We can see that the filter for the adaptive
method is divided according to the characters of “Light” radiated from the projector as we expected. Figure 12 shows
the closeup images of the diﬀuse-reflection board at the positions 3 (top) and 4 (bottom): (a) the ground truth images
and the reconstructed images by using (b) the multiplexed
method and (c) the adaptive method. We can see that ouradaptive method works well, whereas some artifacts due touniform spatial resolution on the LC panel are visible in thereconstructed images by using the multiplexed method, inparticular in the top of (b).
The second light source is a large electric torch. Figure 13 shows (a) the image of the large electric torch and(b) the adaptive filter for acquiring the 4D light field of theFig. 13 (a) The image of the large electric torch and (b) the adaptive filter
for acquiring the 4D light field of the torch.

Fig. 14 The reconstruction results on the large electric torch. The
closeup images of the diﬀuse-reflection board at the positions 3 (top) and 4(bottom): (a) the ground truth image and the reconstructed images by using(b) the multiplexed method and (c) the adaptive method.

torch. We can see that the filter for the adaptive method
is divided almost symmetrically with respect to the centerof the rays radiated from the torch. Figure 14 shows thecloseup images of the diﬀuse-reflection board at the positions 3 (top) and 4 (bottom): (a) the ground truth images
and the reconstructed images by using (b) the multiplexed
method and (c) the adaptive method. Similar to the above,
we can see that artifacts due to uniform spatial resolution
on the LC panel are visible in the reconstructed images byusing the multiplexed method.
5.

Conclusion and Future WorkIn this paper, we proposed an approach to measuring the
4D light field of a self-luminous extended light source byusing an LC panel, i.e. a programmable filter and a diﬀusereflection board. The approach recovers the 4D light fieldfrom the images of the board illuminated by the light radiated from an extended light source and passing throughthe LC panel. Our proposed methods make use of the feature that the transmittance of the LC panel can be controlled
both spatially and temporally, and recover 4D light fields efficiently and densely on the basis of multiplexed sensing andadaptive sensing. We implemented the prototype setup, andconfirmed through a number of experiments that the proposed approach is eﬀective for modeling self-luminous extended light sources in the real world.
The future work of this study includes the applicationsof the acquired light fields to computer graphics problems
such as photorealistic image synthesis and computer vision
problems such as image-based modeling.

NAKAMURA et al.: ACQUIRING 4D LIGHT FIELDS OF SELF-LUMINOUS EXTENDED LIGHT SOURCES

2367

Acknowledgments
A part of this work was supported by JSPS KAKENHI
Grant No. 24650077 and Konica Minolta Science and Technology Foundation.

Motohiro Nakamura
received his B.S. and
M.S. degrees in computer science and systems
engineering from Kyushu Institute of Technology, Japan, in 2014 and 2016 respectively. His
research interests include computer vision andcomputational photography, in particular lightfield.

References[1] T. Aoto, T. Sato, Y. Mukaigawa, and N. Yokoya, “Linear estimation of 4-D illumination light field from diﬀuse reflections,” Proc.
ACPR2013, pp.496–500, 2013.
[2] I. Ashdown, “Near-field photometry: a new approach,” Journal of
Illuminating Engineering Society, vol.22, no.1, pp.163–180, 1993.
[3] E. Adelson and J. Bergen, “The plenoptic function and the elementsof early vision,” Computational Models of Visual Processing, pp.3–
20, MIT Press, 1991.
[4] M. Goesele, X. Granier, W. Heidrich, and H.-P. Seidel, “Accurate light source acquisition and rendering,” Proc. ACM
SIGGRAPH2003, vol.22, no.3, pp.621–630, 2003.
[5] I. Moreno and C.-C. Sun, “Three-dimensional measurement oflight-emitting diode radiation pattern: a rapid estimation,” Measurement Science and Technology, vol.20, no.7, pp.1–6, 2009.
[6] M. Nakamura, T. Okabe, and H.P.A. Lensch, “Acquiring 4D lightfields of self-luminous light sources using programmable filter,”Proc. LF4CV2014 (LNCS vol.8926), pp.588–599, 2015.
[7] R. Ng, M. Levoy, M. Bredif, G. Duval, M. Horowitz, and P.
Hanrahan, “Light field photography with a hand-held plenoptic camera,” Stanford Tech Report CTSR 2005-02, 2005.
[8] C.-K. Liang, T.-H. Lin, B.-Y. Wong, C. Liu, and H.H. Chen, “Programmable aperture photography: multiplexed light field acquisition,” Proc. ACM SIGGRAPH2008, vol.27, no.3, 2008.
[9] R. Ramamoorthi and P. Hanrahan, “A signal-processing framework
for inverse rendering,” Proc. ACM SIGGRAPH2001, pp.117–128,
2001.
[10] R. Rykowski and H. Kostal, “Novel approach for LED luminous
intensity measurement,” Proc. SPIE, vol.6910, 2008.
[11] Y.Y. Schechner, S.K. Nayar, and P.N. Belhumeur, “A theory of multiplexed illumination,” Proc. ICCV2003, vol.2, pp.808–815, 2003.
[12] N.J.A. Sloane, T. Fine, P.G. Phillips, and M. Harwit “Codes for multiplex spectrometry,” Applied Optics, vol.8, no.10, pp.2103–2106,
1969.
[13] H.Y. Tan and T.W. Ng, “Light-emitting-diode inspection using a
flatbed scanner,” Optical Engineering, vol.47, no.10, 2008.
[14] J. Unger, A. Wenger, T. Hawkins, A. Gardner, and P. Debevec, “Capturing and Rendering with Incident Light Fields,” Proc. EGSR2003,
pp.1–10, 2003.
[15] C. Verbeck and D. Greenberg, “A comprehensive light-source
description for computer graphics,” IEEE CG&A, vol.4, no.7,
pp.66–75, 1984.
[16] G. Wetzstein, I. Ihrke, and W. Heidrich, “On plenoptic multiplexing
and reconstruction,” IJCV, vol.101, no.2, pp.384–400, 2013.
[17] B. Wilburn, N. Joshi, V. Vaish, E.-V. Talvala, E. Antunez, A. Barth,
A. Adams, M. Horowitz, and M. Levoy, “High performance imaging
using large camera arrays,” Proc. ACM SIGGRAPH2005, vol.24,
no.3, pp.765–776, 2005.

Shinnosuke Oya
received his B.S. degree incomputer science and systems engineering from
Kyushu Institute of Technology, Japan, in 2016.
He is currently working for the M.S. degree at
Kyushu Institute of Technology. His researchinterests include computer vision and computational photography, in particular light transport.

Takahiro Okabereceived his B.S. and
M.S. degrees in physics, and his Ph.D. degree
in information science and technology from the
University of Tokyo, Japan, in 1997, 1999, and2011 respectively. After working at Institute ofIndustrial Science, the University of Tokyo, hejoined Kyushu Institute of Technology, Japan,
as an associate professor in 2013. He was a visiting scholar at Tübingen University, Germany,from 2011 to 2012. His research interests include computer vision, computational photography, image processing, pattern recognition, and computer graphics, in particular their physical and mathematical aspects. He received the IEICE Best
Paper Awards in 2006, 2008, and 2010.

Hendrik P. A. Lensch
holds the chair forcomputer graphics at Tübingen University. He
received his diploma in computers science fromthe University of Erlangen in 1999. He worked
as a research associate at the computer graphics group at the Max-Planck-Institut für Informatik in Saarbrücken, Germany, and received
his PhD from Saarland University in 2003. Hendrik Lensch spent two years (2004-2006) as avisiting assistant professor at Stanford University, USA, followed by a stay at the MPI Informatik as the head of an independent research group. From 2009 to 2011
he has been a full professor at the Institute for Media Informatics at Ulm
University, Germany. In his career, he received the Eurographics Young
Researcher Award 2005, was awarded an Emmy-Noether-Fellowship by
the German Research Foundation (DFG) in 2007 and received an NVIDIA
Professor Partnership Award in 2010. His research interests include 3D appearance acquisition, computational photography, global illumination andimage-based rendering, and massively parallel programming.