Fast Spectral Reflectance Recovery
using DLP Projector
Shuai Han1 , Imari Sato2 , Takahiro Okabe1 , and Yoichi Sato1
1

Institute of Industrial Science, The University of Tokyo, Japan
2
National Institute of Informatics, Japan{hanshuai,takahiro,ysato}@iis.u-tokyo.ac.jp, imarik@nii.ac.jp

Abstract. Spectral reflectance is an intrinsic characteristic of objects
which is useful for solving a variety of computer vision problems. In thiswork, we present a novel system for spectral reflectance recovery with ahigh temporal resolution by exploiting the unique color-forming mechanism of DLP projectors. DLP projectors use color wheels to produce
desired light. Since the color wheels consist of several color segments androtate fast, a DLP projector can be used as a light source with spectrally
distinct illuminations. And, the appearance of a scene under the projector’s irradiation can be captured by a high-speed camera. Our system
is built on easily available devices and capable of taking spectral measurements at 100Hz. Based on the measurements, spectral reflectance ofthe scene is recovered using a linear model approximation. We carefully
evaluate the accuracy of our system and demonstrate its effectiveness byspectral relighting of dynamic scenes.

1

Introduction

The amount of light reflected on an object’s surface varies for different wavelengths. The ratio of the spectral intensity of reflected light to incident lightis known as the spectral reflectance. It is an intrinsic characteristic of objects
that is independent of illuminations and imaging sensors. Therefore, spectralreflectance offers direct descriptions about objects that are useful to computer
vision tasks, such as color constancy, object discrimination, relighting etc.
Several methods have been proposed for spectral reflectance recovery. Maloney used an RGB camera to recover the spectral reflectance under ambient
illumination [1]. This method is limited by a low recovery accuracy due to its
RGB 3-channel measurements. To get measurements that contain more than 3channels, some works attach filters to a light source to modulate the illumination[2] or sequentially place a set of band-pass filters in front of a monochromatic
camera to produce a multi-channel camera [3]. Since switching among filters istime-consuming, these methods are unsuitable for dynamic scenes. To increase
temporal resolution, specially designed clusters of different types of LEDs werecreated [4]. The LED clusters work synchronously with an RGB camera for conducting spectral measurements at 30 fps. Since such self-made light sources, as2

S. Han et alwell as the controller for synchronization, are not easily available, a level of effort
is required to build a similar system.
What we seek is a practical system for fast spectral reflectance recovery built
on easily available devices. In this work, we exploit the unique color-forming
mechanism of Digital Light Processing (DLP) projectors and apply it for spectral
measurements. DLP projectors use color wheels to produce the desired light.
The color wheels are composed of several color segments, and the light thatgets through these segments has specific spectral distributions. In other words,
DLP projectors provide several spectrally distinct illuminations. When the color
wheels rotate quickly, the light emitted from the DLP projectors rapidly switches
among these illuminations. Making use of this switch, we built an imaging systemthat takes spectral measurements with a high temporal resolution.
In the system, a DLP projector is used as a light source, and a high-speed
camera is used to capture the scenes’ appearance under the projector’s irradiation. A standard diffuse white board is placed in the scene to recover theillumination spectra of the captured frames. In order to reduce the number ofrequired measurements for an accurate spectral reflectance recovery, we represent
the spectral reflectance as a linear combination of a limited number of spectral
bases, which was done in previous studies [5, 6]. Using this linear model, thespectral reflectance of the scene points can be reconstructed by using every fiveconsecutive captured frames.
The contributions of this work are summarized below.
•

Dense temporal spectral measurement: Our system is capable of taking
spectral measurements at 100 Hz. This enables measurement for the fastmoving objects, and the recovered results are degraded little by motion blur.
• Easily built imaging system: Considering that high-speed cameras are becoming readily available in end-user markets and no synchronization between
the projector and the camera is required, our system can be easily replicatedby others. Furthermore, using the DLP projectors as light sources, the irradiation uniformity within the entire projection plane can be guaranteed, so
the calibrations are simple and the working volume is large.
This paper is organized as follows. Section 2 gives a brief review of the related
works. Section 3 presents our imaging system and its use for spectral reflectancerecovery. Section 4 verifies its accuracy. Section 5 shows the relighting results ofa static scene and a moving object. We conclude this work in Section 6.

2

Related work

Spectral reflectance can be recovered under passive illumination. Maloney and
Wandell used color constancy and an RGB camera for spectral reflectance recovery [1], but the accuracy of their method was low due to the RGB 3-channel
measurement. For accurate results, Tominaga put a set of band-pass filters infront of a monochromatic camera, so that more than 3 channels can be measured [3]. However, this method trades off temporal resolution for the spectral
resolution, and thus, is unsuitable for dynamic scenes.

Fast Spectral Reflectance Recovery using DLP Projector

3

Other existing methods for spectral reflectance recovery rely on active illumination. DiCalro and Wandell recovered the spectral reflectance as an intermediate result [7], but the accuracy was limited by the expression of thespectral reflectance as a combination of three spectral bases. To recover spectralreflectance with high accuracy, D’Zmura proposed a method using distinct illuminations[8], but the author only showed the results using synthetic data, andhow well the proposed method works for real scenes was left unknown. Cui etal. proposed an algorithm for selecting an optimized set of wide-band filters andbuilt a multi-illumination system [2]. They attached the selected filters to a lightsource, and used it as an additional light source for spectral reflectance recovery
under ambient illumination. This method works well for static scenes. However,switching among different illuminations is time-consuming, so the system is notapplicable for moving objects.
To measure the dynamic scenes, Park et al. built an imaging system basedon multiplexed illumination [4]. They focused on the combinations of different
LEDs and built LED clusters to capture 30 fps multi-spectral videos. However,their system requires specially built LED clusters and synchronization between
the LED clusters and a camera. Accordingly, their system is not easily available.
Moreover, using these self-made LED clusters, irradiation uniformity can be
guaranteed only in a small area, so the working volume is quite limited.
Our work is also related to DLP-based active vision. Nayar et al. implemented
a programmable imaging system using a modified DLP projector-camera pair[9]. Users can control the radiometric and geometric characteristics of the captured images by using this system. Narasimhan et al. exploited the temporaldithering of DLP projectors for a wide range of applications [10]. Zhang andHuang used the fast illumination variation of a DLP projector for real-time 3D
shape measurements [11]. These three works only utilized the fast alternation
between the “on” and “off” statuses of the digital micromirror device in a DLP
projector; the spectral information was disregarded. In contrast, we use the spectral information from the emitted light for the spectral reflectance recovery. Our
work is the first to recover spectral reflectance using a DLP projector.

3

Spectral Reflectance Recovery

3.1

Three steps for spectral reflectance recovery

There are three factors related to image brightness: the incident light, the scene,
and the camera. Suppose the camera has a linear intensity response, this relationship can be expressed asZIm,n = s(λ)cm (λ)ln (λ)dλ,(1)where λ is the wavelength, Im,n is the intensity of a scene point in a captured
frame, s(λ) is the spectral reflectance of that point, cm (λ) is the spectral response
function of the camera at the mth color channel, and ln (λ) is the spectrum of
the nth illumination.

4

S. Han et al

Fig. 1. Prototype System. Composed of a DLP projector (PLUSTM U2-1130), a highspeed camera (PointGreyTM Lightning) and a white board (labsphereTM SRT-99).

The goal of this work is to recover spectral reflectance s(λ) in a visible range(400−−700[nm]). From Eq. 1, we can see that a large set of spectrally distinct
measurements are required if we want to recover s(λ) with high spectral resolution. To reduce the number of required measurements without sacrificing spectral
resolution, we approximate the spectral reflectance as a combination of a limited
number spectral basis functions. This approximation procedure was also used informer works [4, 7].
Several linear models [5, 6] and a nonlinear model [12] have been built byusing principal component analysis [13] or other tools (see Ref. [14] for a review
about surface reflectance approximation). With regard to how many bases are
required for accurate reconstruction, different works have different conclusions [5,6, 15–17]. We adopt an 8-dimension linear model for spectral reflectance derived
from Ref. [6] on account of its high reconstruction accuracy. On the basis of thislinear model, the spectral reflectance is represented ass(λ) =8X

αj bj (λ),

(2)

j=1

where bj (λ)(j = 1, 2, .., 8) is the jth spectral basis from Ref. [6] (spectral resolution:10nm), αj is the corresponding coefficient. Substituting Eq.2 for Eq.1, we
obtainIm,n =8XZ
αjbj (λ)cm (λ)ln (λ)dλ

(3)

j=1

In this work, we first estimate αj from observed Im,n . Then, spectral reflectance s(λ) is reconstructed by substituting αj into Eq. 2.
As shown in Fig. 1, our imaging system is composed of a one-chip DLP projector, a high-speed RGB camera with a linear intensity response and a standard

Fast Spectral Reflectance Recovery using DLP Projector

5Fig. 2. Color switch caused by rotation of color wheel.

diffuse white board. Using this system, we do spectral reflectance recovery in thefollow three steps.
1. Image acquisition: Scene’s appearance under the projector’s irradiation
Im,n , is acquired by using the high-speed camera. Every five consecutive frames
are used as one measurement for the spectral reflectance recovery. (Section 3.2)
2. Illumination recovery: Illumination spectra ln (λ), changes from frame toframe. We use the diffuse white board as a calibration target to recover theillumination of captured frames. (Section 3.3)
3. Spectral reflectance reconstruction: Based on the 8-dimensional linear
model, spectral reflectance s(λ), can be reconstructed from the acquired images
and recovered illuminations. (Section 3.4)
We explain each of these steps in detail in the following parts.
3.2

Image acquisition by color switch

Different from other kinds of projectors, DLP projectors use color wheels to
produce the desired light. The color wheel consists of several color segments, andthese segments only allow light in a specific wavelength range to get through.
When the color wheel quickly rotates, the light emitted from DLP projectors
changes rapidly. In our work, this temporal variation in light is referred to as“color switch”. A diagrammatic sketch is shown in Fig. 2. In our system, a DLP
projector equipped with a 3-segment color wheel has been used (PLUSTM U21130). since the color wheel rotates at 120 rps (round per second), color switch
occurs at 360 Hz (3 × 120).
The human eyes, and common video cameras work at low rates (24−−30[Hz]), and thus they cannot detect the color switch. In this work, a 500 fps
camera (PointGreyTM Lightning) is adopted to take images of scenes under
the projector’s irradiation. The camera outputs 24bit (8 × 3) color images at a
SXGA resolution (1280 × 1024), and its linear intensity response can be verified
by adjusting the shutter speed. In addition, the spectral response function of
the camera cm (λ) (m = 1, 2, 3), was measured by using a monochromator and aspectrometer. The monochromator is used to generate a sequence of narrow-band
lights. The spectral radiance of these lights is measured by the spectrometer. We

6

S. Han et al

Fig. 3. Camera’s spectral response function for RGB 3 channels.

Fig. 4. One measurement of Macbeth ColorChecker and corresponding illumination
spectra. Top row: 5 frames captured sequentially by the 500 fps camera in 1/100s.
Bottom row: recovered illuminations of corresponding frames.

expose the camera’s sensor to the narrow-band lights and capture images. The
relationship between the RGB values in the captured images and the spectral
radiance of the corresponding lights, i.e., spectral response function, is shownin Fig. 3. During one rotation of the color wheel, the high-speed camera cancapture 4.17 frames. So, we use five consecutive frames as one measurement forthe spectral reflectance recovery. Fig. 4 shows one measurement about Macbeth
ColorChecker. We can see that the scene’s appearance clearly changes under
the color switch of the DLP projector. It should be noted that the color switch
occurs at 360Hz, but the camera operates at 500 fps, so the projector and thecamera work asynchronously.

3.3

Illumination recovery

Our system does not require synchronization between the projector and thecamera. Due to the asynchronism, the illumination changes from frame to frame.
In this section, we describe how to recover the illumination spectrum ln (λ) ofFast Spectral Reflectance Recovery using DLP Projector

7Fig. 5. Spectra of three distinct illuminations of the DLP projector.

every frame using a standard diffuse white board (labsphereTM SRT-99) placed
within the scene as a calibration target.
As mentioned above, light that gets through different segments on color
wheels has distinct spectral distributions. If we use these spectral distributions as
the illumination bases, light emitted from the DLP projectors can be expressedby a linear combination of these bases. In our system, since the three segments ofthe color wheel correspond to the RGB color filters, we can acquire these three
distinct illuminations by inputting the projector (255, 0, 0), (0, 255, 0), and(0, 0, 255) respectively. Their spectra, which are measured by a spectrometer,are shown in Fig. 5. For each frame, its illumination spectrum ,ln (λ), can be
represented asln (λ) =3Xβn,k pk (λ), subject to βn,k > 0,

(4)

k=1

where pk (λ) is the spectrum of the kth illumination basis of the DLP projector,
βn,k is the corresponding coefficient.
By using Eqs.1 and 4, the brightness of a surface point on the white board is
w
Im,n
=

3
X

Z
βn,k

pk (λ)sw (λ)cm (λ)dλ,

(5)

k=1
w
where Im,nis the intensity of that point, and sw (λ) means its spectral reflectance.
Use Pk,m to represent the intensity of the point at the mth channel under the
kth illumination basis
Z
Pk,m = pk (λ)sw (λ)cm (λ)dλ (k = 1, 2, 3),(6)

Eq. 5 can be rewritten as
w
Im,n
=

3
X
k=1

βn,k Pk,m ,

(7)

8

S. Han et al

Pk,m (k = 1, 2, 3) can be measured by using the high-speed camera to capture
images of the white board under three distinct illuminations of the projector.
We only need to measure them once in advance.
From Eq. 7, we see that the intensity of a surface point on the white board
under illumination ln (λ) is a linear combination of its intensities under threeillumination bases
 w w w TIw
= P w βn ,(8)n = I1,n I2,n I3,nwhere I w
n represents the RGB value of a surface point on the white board under
the nth illumination, P w is a 3 × 3 matrix consists of Pk,m (k = 1, 2, 3, m =1, 2, 3), β n is the corresponding 3 × 1 coefficient vector.
In principle, β n can be easily calculated by β n = (P w )−1 I w
n . However, due
to the noise , βn,k (k = 1, 2, 3) may sometimes be negative. This conflicts with
the non-negative constraint of Eq. 4. Thus, we solve βn as a non-negative leastsquares problem:w
2
β n = argmin | I w
n − P βn | ,subject to βn,k ≥ 0 (k = 1, 2, 3)

(9)

βn

Using calculated βn , illumination spectrum ln (λ) can be reconstructed byusing Eq.4.
3.4

Spectral reflectance reconstruction using constrained model

Since ln (λ) is recovered in SectionR 3.3, the integral in Eq. 3 can be represented
as known coefficients: fj,m,n = bj (λ)cm (λ)ln (λ)dλ. One measurement that
contains five consecutive frames can be written in matrix form as
I = F α,(10)where I is a 15 × 1 vector (15 measurements: RGB 3 channels × 5 frames), F
is a 15 × 8 matrix (15 measurements × 8 spectral bases), and α is an 8 × 1
coefficient vector.
If α is estimated from I, spectral reflectance s(λ) can be reconstructed by
Eq. 2. In this way, the problem of spectral reflectance recovery can be solved by
the 8 coefficients estimation. The DLP projector in our system has three spectrally distinct illuminations, and the high-speed camera provides a 3-channel
measurement under each illumination. In total, we can obtain 3 × 3, i.e., 9 effective channels. Thus, the problem of estimating 8 coefficients is over-determined.
However, using the least squares solution in Eq. 10, the reconstructed spectralreflectance does not always satisfy the non-negative constraint and the solutions tend to be unstable. Therefore, we adopted the constrained minimization
method proposed in Ref. [4]. We use the first derivative of the spectral reflectancerespective to λ as the constraint:∂s(λ) 2
α = argmin | I − F α |2 +γ|| , subject to bm α ≥ 0 for all λ,
α
∂λ

(11)

Fast Spectral Reflectance Recovery using DLP Projector

9Fig. 6. Recovered spectral reflectance of some clips on Macbeth ColorChecker by themeasurement shown in Fig. 4. Ground truth: red lines; recovered: black lines.

Fig. 7. RMS error of 24 clips of Macbeth ColorChecker for 200 measurements.

where γ is a weight for the constraint term. bm is a 31 × 8 matrix whose columns
are the 8 spectral bases.

4

Accuracy Evaluation

In this section, we evaluate the accuracy of our system by using Macbeth ColorChecker. In the system, every five consecutive frames captured by the 500 fps
camera are used as one measurement. Thus, the spectral measurements are taken
at 100 Hz. But, the color wheel rotates at 120 rps. Due to the asynchronism between the DLP projector and the camera, frames captured at different times
have different illumination spectra. The accuracy of the recovered results would
be affected by this temporal illumination variation. Thus, we need to evaluate
both the spectral accuracy and temporal accuracy of our system in this section.
We sequentially took 200 measurements (1000 frames) of a static 24-clip
Macbeth ColorChecker to evaluate spectral accuracy. For each clip, we set γ in
Eq. 11 to 50 and reconstructed its spectrum based on the measurements (someresults are shown in Fig. 6); then, the root mean square (RMS) error of all 200

10

S. Han et al

Fig. 8. Average RMS error for 200 measurements. Because color wheel rotates 6 rounds
for every 5 measurements, a pattern of the average RMS error can be seen.

reconstructed results was calculated. We also computed the maximum, mean,
and minimum among the 200 RMS error values for every clip. The results forall 24 clips are shown in Fig. 7. We can see that, for all clips, their maximum
RMS error does not deviate a lot from the minimum one. In addition, the biggest
mean RMS error of all 24 clips is less than 0.11. These results demonstrate that
our system can recover the spectral reflectance at a reasonable accuracy.
Next, we evaluated the temporal accuracy of our system. We reused the 200
measurements taken in previous test. For every measurement, we reconstructed
the spectral reflectance of all 24 clips; then, the RMS error of the 24 reconstructedresults was calculated; after that, we computed the average value of the 24
RMS error values, and used it as the criterion to evaluate each measurement.
The results for all 200 measurements are shown in Fig. 8. The average value
fluctuates in a narrow band (0.047, 0.06) which verifies the temporal accuracyof our system.

5

Image and Video Relighting

We used the spectral reflectance recovered by our method to do spectral relighting of a static scene as well as a moving object. To ensure there was a strong andspatial uniformly distributed light, an LCD projector (EPSONTM ELP-735) wasused as the light source for relighting. The spectral distributions of its white,
red, green, and blue were measured by a spectrometer.
5.1

Image relighting

We set a static scene with fruits, vegetables, and small statues. Five consecutive
frames from the scene were captured by our imaging system. Using them as
one measurement, the spectral reflectance of scene points was recovered pixelby pixel. Then, the scene was spectrally relit by using Eq. 1 with the known

Fast Spectral Reflectance Recovery using DLP Projector
Illumination

Computed Result

11

Ground Truth

Fig. 9. Comparison between relit results and captured images of static scene underilluminations from a LCD projector.

illumination spectra of the LCD projector. A comparison between the relit results
and the real captured images is shown in Fig. 9. We can see from the comparison
that the computed results are very similar to the ground truth, which also reveals
the accuracy of our system.
5.2

Real video relighting

Our system works at 100 Hz, so it is capable of measuring dynamic scenes. This
capability was tested by taking spectral measurements of a manipulated toy
consequently. For every measurement, the spectral reflectance of scene points wasreconstructed. Based on the recovered data, the toy’s movements were spectrally
relit under a variety of illuminations. The results are shown in the top two rows ofFig. 10, and we can that there is a smooth movement, and the computed results
look natural. In the bottom of Fig. 10, a relit result is shown in the middle. It
was computed on the basis of the spectral data recovered by our system. The
left is a image captured by the high-speed camera under the LCD projector’s

12

S. Han et al

Captured image (500 Hz) Relit result(100 Hz) Synthesized image(30 Hz)
Fig. 10. Top two rows: relit results of fast-moving toy. The continuous movements
through very different illuminations are shown. Bottom row: on the left is an image
captured by the 500 fps camera, the middle is relit result, and the right is synthesized
result to simulate captured image by a 30 fps camera. The recovered result by oursystem is only slightly degraded by motion blur.

irradiation. A synthesized result to simulate captured image by a 30 fps camera
is shown on the right side. Through comparisons, we can see that the relit result
resembles the real captured image, and it is degraded little by the motion blurwhich is obvious in the synthesized result. From the comparisons, we can seethat our system is robust to artifacts caused by motion. Therefore, our system
is suitable for spectral reflectance recovery of dynamic scenes.

6

Conclusion

In this work, we exploited the unique color-forming mechanism of DLP projectors. An imaging system for fast spectral reflectance recovery was built bymaking use of this mechanism. This system is capable of taking measurements
as fast as 100 Hz. Every measurement consists of a set of sequentially captured
images. For each set, the spectral reflectance of scene points can be recovered.
Through intensive evaluation, the accuracy and the robustness of our systemhave been verified. Moreover, our system is built on easily available devices, andthe excellent optical design of DLP projectors guarantees simple calibrations andFast Spectral Reflectance Recovery using DLP Projector

13

a large working volume. It can be concluded that our system is practical androbust for the spectral reflectance recovery of fast-moving objects.

Acknowledgement
This research was supported in part by Grant-in-Aide for Scientific Research on
Innovative Areas from the Ministry of Education, Culture, Sports, Science andTechnology.

References
1. L. T. Maloney and B. A. Wandell: Color constancy: a method for recovering surface
spectral reflectance. Journal of the Optical Society of America A 3 (1986) 29-33
2. C. Chi, H. Yoo and M. Ben-Ezra: Multi-spectral imaging by optimized wide band
illumination. International Journal on Computer Vision 86 (2010) 140-151
3. S. Tominaga: Multichannel vision system for estimating surface and illumination
functions. Journal of the Optical Society of America A 13 (1996) 2163-2173
4. J. Park, M. Lee and M. D. Grossberg and S. K. Nayar: Multispectral Imaging Using
Multiplexed Illumination. In: Proc. IEEE International Conference on Computer
Vision (2007)
5. J. Cohen: Dependency of the spectral reflectance curves of the munsell color chips.
Psychon. Science 1 (1964) 369-370
6. J. P. S. Parkkinen, J. Hallikainen and T. Jaaskelainen: Characteristic spectra ofmunsell colors. Journal of the Optical Society of America A 6 (1989) 318-322
7. J. M. DiCarlo and B. A. Wandell: Illuminating illumination. In: Proc. Ninth Color
Imaging Conference (2000) 27-34
8. M. D’Zmura: Color constancy: surface color from changing illumination. Journal of
the Optical Society of America A 9 (1992) 490-493
9. S. K. Nayar, V. Branzoi and T. E. Boult: Programmable imaging: Towards a flexible
camera. International Journal on Computer Vision 70 (2006) 7-22
10. S. G. Narasimhan, S. J. Koppal and S. Yamazaki: Temporal dithering of illumination for fast active vision. In: Proc. European Conference Computer Vision (2008)
11. S. Zhang and P. Huang: High-resolution, real-time 3d shape acquisition. In: Proc.
IEEE Conference on Computer Vision and Pattern Recognition Workshops. Volume
3 (2004) 28-37
12. J. M. DiCarlo and B. A. Wandell: Spectral estimation theory: beyond linear but
before Bayesian. Journal of the Optical Society of America A 20 (2003) 1261-1270
13. D. Tzeng and R. S. Berns: A review of principal component analysis and its applications to color technology. Color Research And Application 30 (2006) 84-98
14. O. Kohonen, J. Parkkinen and T. Jääskeläinen: Databases for spectral color science. Color Research And Application 31 (2006) 381-390
15. J. L. Dannemiller: Spectral reflectance of natural objects: how many basis functions
are necessary? Journal of the Optical Society of America A 9 (1992) 507-515
16. C. Chiao, T. W. Cronin and D. Osorio: Color signals in natural scenes: characteristics of reflectance spectra and effects of natural illuminants. Journal of the OpticalSociety of America A 17 (2000) 218-224
17. L. T. Maloney: Evaluation of linear models of surface spectral reflectance withsmall numbers of parameters. Journal of the Optical Society of America A 3 (1986)
1673-1683