Fast Separation of Specular, Diﬀuse, and Global
Components via Polarized Pattern Projection
Yuto Nisaka1 , Ryo Matsuoka2 , Toshiyuki Amano3 , and Takahiro
Okabe1[0000−0002−2183−7112]
1

Kyushu Institute of Technology
okabe@ai.kyutech.ac.jp
2
The University of Kitakyushu
3
Wakayama University

Abstract. In this paper, we propose a method for fast separation of
specular, diﬀuse, and global components of a dynamic scene by using
a projector-camera system. Both the direct-global separation using spatially high-frequency patterns and the specular-diﬀuse separation basedon polarization have been studied, but a straightforward combinationof those methods has limited temporal resolution. Accordingly, our proposed method rapidly changes not only the spatial patterns but also
the polarization states of illumination by using a self-build polarization projector, and captures their eﬀects on a scene by using a highspeed camera. Our method is easy-to-implement, because it does not
require projector-camera temporal synchronization and it automaticallycalibrates the correspondence between the projection pattern and camera pixel. In addition, our method is robust due to the optimized andquickly-shifted projection pattern and the weights for incorporating spatial correlation. We implemented the prototype setup and achieved fast
separation with 60 fps.
Keywords: direct-global separation · specular-diﬀuse separation · projectorcamera system · polarization

1

Introduction

When a scene is illuminated by a light source, the radiance value observed at eachpoint in the scene consists of two components: direct and global [10]. The directcomponent such as (direct) specular reflection and (direct) diﬀuse reflection iscaused by the direct illumination from the light source itself. On the other hand,
the global component is caused by the illumination from the other points inthe scene due to inter-reflection, subsurface scattering, volumetric scattering,diﬀusion, and so on. Separating those components of a scene is important forvarious CV and CG applications such as shape recovery, image-based material
editing, and improving image quality [10, 9, 4, 17].
In a static scene, we can separate specular and diﬀuse reflection components
on the basis of polarization by using a rotating linear polarizer in front of a

2

Y. Nisaka et al.

light source and a camera with a fixed linear polarizer [18] or a light sourcewith a fixed linear polarizer and a polarization camera. This is because specularreflection components are polarized but diﬀuse reflection components are unpolarized. On the other hand, we can separate direct and global components byprojecting spatially high-frequency patterns from a projector [10], because theglobal components are low-frequency in general. Therefore, those techniques canbe combined for separating specular, diﬀuse, and global components of a static
scene. Unfortunately, however, a straightforward combination of those techniques
is not applicable to dynamic scenes, since the frame rates of oﬀ-the-shelf projectors and polarization cameras are not so high in general.
In this paper, we propose a method for fast separation of specular, diﬀuse,
and global components of a dynamic scene by using a projector-camera system [12, 3, 2, 1]. The key idea of our proposed method is to change not only thespatial patterns but also the polarization states of illumination rapidly by using a self-build polarization projector. Our polarization projector is based on a
DLP (Digital Light Processing) projector; we replace a rotating color wheel ina DLP projector with a rotating linear polarizer. Then, the image sequence ofthe scene is captured by using a high-speed camera so that the eﬀects of therapidly-varying illumination conditions on the scene are observed.
Our proposed method investigates the image sequence of a scene, when the
input to our polarization projector is a single high-frequency pattern. Note thatthe output from the projector rapidly changes in both polarization states andintensities because of the rotating linear polarizer and the temporal dithering [9]due to DMD (Digital Mirror Device) even for a single input pattern. Our method
uses a reference object, and then does not require projector-camera temporalsynchronization and automatically calibrates the correspondence between the
projection pattern and the camera pixel even in a dynamic scene. Furthermore,in order to improve the quality of the separated images, our method optimizes
the projection pattern on the basis of noise propagation analysis, makes use of
the 3D mode in the side-by-side format of a projector for rapidly shifting the
projection pattern, and introduces pixel weights for taking spatial correlation
into consideration.
The main contribution of this study is threefold. First, we achieve fast separation of specular, diﬀuse, and global components with 60 fps by exploiting the
rapidly-varying polarization states and the temporal dithering of our self-build
polarization projector. Second, our proposed method is easy-to-implement, because it does not require projector-camera temporal synchronization and achieves
auto-calibration of the projection pattern-camera pixel correspondence. Third,
our method is robust due to the optimized and shifted projection pattern andthe incorporated spatial correlation.
The rest of this paper is organized as follows. In Section 2, we briefly summarize related work. In Section 3, our polarization projector is introduced, anda method for fast separation of specular, diﬀuse, and global components of a
dynamic scene is proposed. We report the experimental results in Section 4 andpresent concluding remarks in Section 5.

Fast Separation of Specular, Diﬀuse, and Global Components

2

Related Work

2.1

Specular-diﬀuse separation

3

In general, the reflected light observed on an object surface consists of a diﬀuse
reflection component and a specular reflection component. Shafer [14] proposesa method for specular-diﬀuse separation based on the diﬀerence of their colors.
Specifically, the color of a specular reflection component is the same as the colorof a light source, but the color of a diﬀuse reflection component depends on thereflectance of a surface according to the dichromatic reflection model.
We can separate specular and diﬀuse reflection components on the basis of
the low-rank structure of diﬀuse reflection components under varying light source
directions. Specifically, the image of a Lambertian object under an arbitrary
directional light source is represented by the linear combination of three basis
images of the object [15]. The low-rank structure is used for separating specularand diﬀuse reflection components of the images taken under a multi-spectral
light stage [6].
We can separate specular and diﬀuse reflection components on the basis of
the diﬀerence of the polarization states of a specular reflection component anda diﬀuse reflection component. Specifically, when we observe the reflected lightfrom an object surface illuminated by polarized light, the former is polarized
whereas the latter is unpolarized [18]. The polarization-based approach requires
a set of images taken by placing linear polarizing filters in front of a light source
and a camera, and rotating one of them.
In contrast to the above existing methods, we propose a method for separating specular, diﬀuse, and global components of a scene by using a projectorcamera system. We separate not only specular and diﬀuse reflection components
within direct components but also global components by projecting spatiallyhigh-frequency patterns.
2.2

Direct-global separation

The direct-global separation reveals how the light radiated from a light source
interacts with a scene. Nayar et al. [10] propose a method for separating direct
and global components in a scene by projecting high-frequency patterns from a
projector on the basis of the insight that global components are spatially lowfrequency in general.
Gu et al. [4] extend the above direct-global separation for a single lightsource to that for multiple light sources. Specifically, they optimize a set ofhigh-frequency projection patterns in terms of SNR (signal-to-noise ratio) on
the basis of illumination multiplexing [13], and then show that it is eﬀective forscene recovery. Mukaigawa et al. [8] show that projecting high-frequency patterns
is useful also for studying the light transport in scattering media, in particularfor decomposing multiple scattering into each bounce component.
In contrast to the above existing methods, we propose a method for separating specular, diﬀuse, and global components of a scene. In other words, we4

Y. Nisaka et al.

further separate the direct components into specular and diﬀuse reflection components on the basis of polarization.
2.3

Active illumination using DLP projector

Narasimhan et al. [9] propose a method for separating direct and global components of a dynamic scene by using a DLP projector and a high-speed cameraon the basis of the temporal dithering, i.e. the rapid changes in intensities dueto the DMD in a DLP projector. They show that the temporal dithering is useful also for fast active vision such as structured light-based range finding andphotometric stereo.
On the other hand, Han et al. [5] propose a method for recovering the spectralreflectance of a moving object by using a DLP projector and a high-speed cameraon the basis of the color switch, i.e. the rapid changes in colors due to the rotatingcolor wheel in a DLP projector. Maeda and Okabe [7] exploit the rapidly-varying
illumination colors due to the color switch of a multi-primary DLP projector,
and then propose a method for separating direct and global components of astatic scene per primary color of the projector by using an usual camera witha short exposure time. In addition, they propose a method for estimating theSPDs (Spectral Power Distributions) of the primary colors in a non-destructive
manner.
Torii et al. [17] exploit both the color switch and the temporal dithering ofa DLP projector, and achieve the direct-global separation of dynamic scenes
per illumination color. Their method is not a straightforward combination of
the above existing methods based on the temporal dithering [9] and the color
switch [7]. In particular, they realize auto-calibration of the projection patterncamera pixel correspondence on the basis of the consistency in pixel intensities.
Furthermore, they optimize the high-frequency projection pattern on the basisof noise propagation analysis so that it is robust to noises.
In contrast to the above existing methods, in particular to Torii et al. [17], we
exploit polarization. Specifically, by replacing a rotating color wheel with a rotating linear polarizer, we exploit the rapid changes in not colors but polarization
states for fast separation of specular, diﬀuse, and global components.

3

Proposed Method

3.1

Principle of direct-global separation

Nayar et al. [10] propose a method for direct-global separation of a static scene
by using a projector-camera system. It exploits the properties of global components, i.e. they are spatially low-frequency in general, and studies the images ofthe scene under high-frequency projection patterns. Specifically, their simplest
method uses two images; they are captured when a black-and-white checkered
pattern or its negative-positive reversed pattern are projected from a projector
to the scene.

Fast Separation of Specular, Diﬀuse, and Global Components

5

Let us denote the two output intensities from a projector corresponding towhite and black input intensities of a checkered pattern by a and b, e.g. a = 1
and b = 0. The pixel values observed at a certain point in a scene are given by1
I + = aLD + (a + b)LG ,2
1−
I = bLD + (a + b)LG ,
2

(1)
(2)

where LD and LG are the direct and global components at the point, and I +and I − are the larger and smaller pixel values observed there, when a > b [10,9]. Therefore, we can obtain the direct component LD and the global component
LG by solving the simultaneous linear equations in eq.(1) and eq.(2), when the
pixel values I + and I − and the output intensities from the projector a and b are
given.
3.2

Principle of specular-diﬀuse separation

Wolﬀ and Boult [18] show that specular and diﬀuse reflection components can beseparated on the basis of polarization, since specular reflection components arepolarized but diﬀuse reflection components are unpolarized. Specifically, they
propose a method for specular-diﬀuse separation of a static scene by using arotating linear polarizer in front of a light source and a camera with a fixedlinear polarizer or a light source with a fixed linear polarizer and a rotating
linear polarizer in front of a camera. Recently, we can replace the pair of thecamera and the rotating linear polarizer with a single polarization camera in thelatter setup.
Let us consider the latter case, i.e. a light source with a fixed linear polarizer
and a rotating linear polarizer in front of a camera. The pixel values observed
at a certain point in a scene are given by
1
Imax = Ls + Ld ,2
1
Imin = Ld ,
2

(3)
(4)

where Ls and Ld are the specular and diﬀuse reflection components at the point,
and Imax and Imin are the maximal and minimal pixel values observed there,
when the linear polarizer is rotated in front of the camera. Therefore, we canobtain the specular reflection component Ls and the diﬀuse reflection component
Ld by solving the simultaneous linear equations in eq.(3) and eq.(4), when the
pixel values Imax and Imin are given.
3.3

Self-build polarization projector

Both the methods for direct-global separation and specular-diﬀuse separation
require multiple images taken under diﬀerent spatial patterns and with diﬀerent polarization states. Note that we still require multiple images taken under6

Y. Nisaka et al.

Fig. 1. Our self-build polarization projector based on a DLP projector. The rotatingcolor wheel in the DLP projector is replaced with a rotating linear polarizer.

diﬀerent spatial patterns, even if we use a polarization camera. Therefore, astraightforward combination of those techniques is not applicable to dynamic
scenes, since the frame rates of oﬀ-the-shelf projectors and polarization cameras
are not so high in general4 .
Accordingly, our proposed method rapidly changes not only the spatial patterns but also the polarization states of illumination by using a self-build polarization projector. Our polarization projector is based on a DLP projector. Ina single-chip DLP projector, a color wheel consisting of three (or more) color
filters with diﬀerent spectral transmittances is placed between a white lamp anda DMD. The color wheel rotates at high speed in front of the lamp, and generates various colors by mixing the transmitted lights with diﬀerent SPDs via
time-division multiplexing. Then, the DMD controls the intensity of the lighttransmitted from the color wheel by temporal dithering; it switches micro mirrors “on” and “oﬀ” directions at high speed. We replace the rotating color wheel
in a DLP projector with a rotating linear polarizer as shown in Fig. 1; the linear polarizer is installed outside the projector housing in order to prevent heatdamage due to the projector lamp.
3.4

Specular-diﬀuse-global separation

Fig. 2(a) shows our setup for separating specular, diﬀuse, and global components
by using a projector-camera system. Specifically, a scene of interest is illuminatedby our polarization projector, and its images are captured by using a high-speed
camera through a fixed linear polarizer. The input to the projector is a single
checkered pattern with two intensities, but the output from the projector rapidlychanges in both the polarization states and intensities due to the rotating linear
polarizer and DMD respectively.
The propagation of the polarized light is computed by using the Stokes vectors and Mueller matrices. We can derive that the pixel value In observed at a
certain point in a scene at the n-th frame (n = 1, 2, 3, ..., N ) is given by1
In = an Ls [cos 2(χn + ϕ) + 1] + an Ld + (an + bn )LG2
4

See the experimental results and discussion in Section 4.3 for more detail.

(5)

Fast Separation of Specular, Diﬀuse, and Global Components
polarization
projector

rotating
linear polarizer

high-speed
camerafixedlinear polarizer

7

reference
objectscene ofinterest(a)(b)

Fig. 2. (a) Our setup for separating specular, diﬀuse, and global components. A sceneof interest is illuminated by our polarization projector, and its images are captured byusing a high-speed camera through a linear polarizer. (b) Our reference object; a partof a white diﬀuse target is covered by a linear polarizer.

or

1
(6)
In = bn Ls [cos 2(χn + ϕ) + 1] + bn Ld + (an + bn )LG ,2
when the point is directly illuminated by the checkered pattern with the output
intensity of an or bn . Here, Ls , Ld , and LG are the specular, diﬀuse, and global
components to be estimated, an and bn are the output intensities from the
projector, and χn and ϕ are the polarization angle of the projected light and the
pixel-wise phase of the polarization angle respectively.
Our proposed method assumes that the scene is approximately static while
N images are captured by using a high-speed camera, e.g. 16 msec in our experiments, and that the output intensities an and bn and the polarization angle
χn of the projected light are known (see Section 3.5 for more detail). Then, ourmethod estimates the specular, diﬀuse, and global components and the phase of
the polarization angle from N images by solving the set of equations in eq.(5)or eq.(6). Specifically, we fix the phase ϕ of the polarization angle to a certain
value, and then solve the simultaneous linear equations in eq.(5) or eq.(6) withrespect to Ls , Ld , and LG by least squares. We change the value of the phasein a coarse-to-fine manner, and find the optimal Ls , Ld , LG , and ϕ with theminimal least-square errors.
3.5

Auto spatio-temporal calibration

Our proposed method calibrates the output intensities an and bn and the polarization angle χn by using a reference object placed in a scene of interest andcaptured at the same time. It consists of a white diﬀuse target and a linear polarizer; a part of the white diﬀuse target is covered by the linear polarizer as shownin Fig. 2(b). Specifically, we calibrate the output intensities an and bn from the8

Y. Nisaka et al.

pixel values of the white target object and compute the polarization angle χnfrom the pixel values when the white target object is illuminated and observed
through the linear polarizer. Thus, our method calibrates an , bn , and χn fromthe captured images in situ, and therefore it does not require projector-camera
temporal synchronization.
In order to solve the set of equations in eq.(5) or eq.(6), we need to calibrate
the correspondence between the projection pattern and the camera pixel. In
other words, we need to known which block, whose output intensity is an or
bn , directly illuminates each point in a scene. To this end, we select two images
whose polarization angles are almost the same χn ≃ χm and output intensities
satisfy an > am , bn < bm , and (an + bn ) ≃ (am + bm ), and study the diﬀerence
between In and Im . Because (In − Im ) is proportional to (an − am ) or (bn − bm )according to the correspondence between the projection pattern and the camera
pixel, we can calibrate the correspondence from the sign of (In − Im ).
3.6

Optimizing projection pattern

The output intensities from our DLP-based polarization projector, i.e. how theoutput intensities vary due to the temporal dithering depends on the input intensities to the projector. Therefore, we can optimize the two input intensitiesof the checkered pattern in order to separate specular, diﬀuse, and global components robustly. We take account of the noises in the observed pixel values,
and then optimize those input intensities on the basis of the noise propagation
analysis.
Specifically, we study the sum of the condition numbers [11] of the coeﬃcientmatrices of the set of equations in eq.(5):
a1 [cos 2(χ1 + ϕ) + 1] a1 (a1 + b1 )/2
 a2 [cos 2(χ2 + ϕ) + 1] a2 (a2 + b2 )/2 


(7)

,
..
..
..


.
.
.
aN [cos 2(χN + ϕ) + 1] aN (aN + bN )/2
and in eq.(6):
b1 [cos 2(χ1 + ϕ) + 1] b1 (a1 + b1 )/2
 b2 [cos 2(χ2 + ϕ) + 1] b2 (a2 + b2 )/2 



,
..
..
..


.
.
.
bN [cos 2(χN + ϕ) + 1] bN (aN + bN )/2(8)

where N is the number of the captured images from which we compute thespecular, diﬀuse, and global components of a single frame. The condition number
tells how much noises in the observation In propagate to the unknowns to be
estimated; Ls , Ld , and LG 5 . We compute the condition numbers for all the pairs
5

We assume ϕ = 0 when computing the condition numbers. We tested other values,
but the condition numbers were almost constant with respect to ϕ.

Fast Separation of Specular, Diﬀuse, and Global Components

(a)

(b)

9

(c)

Fig. 3. The 3D mode in the side-by-side format: (a) a single input image, (b) an output
image for left eye, and (c) an output image for right eye.

of the input intensities in 8 bit (from 0 to 255), and then select the optimal pairof the input intensities from 32,640 (= 256 C2 ) pairs. Note that smaller condition
number is better.
3.7

Use of 3D mode in side-by-side format

The simplest direct-global separation described in Section 3.1 uses two projection
patterns; one is a black-and-white checkered pattern and the other is its negativepositive reversed pattern. It is known that the simplest method often causes
visible artifacts near the boundaries between the blocks of the checkered pattens,
because the spatial resolutions of a projector and a camera are limited. Nayar
et al. [10] show that those artifacts can be reduced by using more than two
projection patterns, e.g. spatially shifted patterns from the original checkered
patten.
Accordingly, our proposed method makes use of the 3D mode in the side-byside format, which is relatively common function even for consumer projectors,
and then projects two spatially-shifted checkered patterns. When a single image
in Fig. 3 (a) is input to the projector, the projector outputs (b) a left-eye imageand (c) a right-eye image in turn. Since we can distinguish the two spatiallyshifted checkered patterns from the pixel values on the reference object, we
can realize the specular, diﬀuse, and global separation by using spatially-shiftedprojection patterns in a similar manner to the case without the 3D mode.
3.8

Incorporating spatial correlation

In addition to the optimized and quickly-shifted projection pattern in Section
3.6 and Section 3.7, we further improve the quality of the separated imagesby taking the spatial correlation into consideration. Specifically, our proposed
method solves the set of equations in eq.(5) or eq.(6) from not only the pixel
values observed at a certain point pixel-wisely but also those at surrounding
pixels by weighted least squares.
We introduce a single weight per pixel, i.e. per set of equations in eq.(5) oreq.(6). The weight is given by the product of three Gaussian weights. The first
one depends on the distance between the pixels, and the second one depends onthe diﬀerence between the pixel values when the output intensities are almost

10

Y. Nisaka et al.

groundtruth

1st

5,000thspeculardiffuseglobalFig. 4. The ground truth images (top row), our result images using the best (middlerow) and the 5,000th best (bottom row) combinations of the input intensities of thecheckered pattern.

Table 1. The quantitative comparison of Fig. 4. The numerical values in each cell are
the PSNR (upper) and SSIM (lower).

ours:
1st
ours:
5,000thspecular
30.29
0.733
30.18
0.749

diﬀuse
26.38
0.797
27.41
0.776

global
32.85
0.774
26.82
0.608

average
29.84
0.768
28.14
0.711

the same, i.e. an ≃ bm . Therefore, the product of those weights is similar to theweight for the bilateral filter [16]. The third weight has small/large values if the
pixel is close to/far from the boundaries between the blocks of the checkered
patten. The boundaries are detected from the zero crossing of the diﬀerence(In − Im ) (see Section 3.5 for more detail).

4

Experiments

4.1

Experimental setup

In our experiments, we used a self-build polarization projector based on a DLP
projector MH534 from BenQ and a high-speed camera Fastcam Mini UX50 from
Photron. The 3D mode of the projector quickly switches the left-eye and right-eye
images in about 8 msec (about 120 fps), and one revolution of the rotating linearFast Separation of Specular, Diﬀuse, and Global Components

200

200

200

200

200

200

150

150

150

150

150

150

100

100

100

100

100

100

50

50

50

50

50

0

0
0

100

R

200

300

0
0

100

G
1st

200

300

0
0

100

B

200

300

50

0
0

100

200

R

300

11

0
0

100

200

G
5,000th

300

0

100

200

300

B

Fig. 5. The output intensities an and bn of each color channel for the 1st and the
5,000th best combinations.

polarizer takes about 14 msec6 . The frame rate and the exposure time of thehigh-speed camera were set to 10,000 fps and 0.1 ms respectively. Therefore, ourproposed method achieves specular, diﬀuse, and global separation from about
160 images in about 60 fps. As described in Section 3.5, we optimized the pixel
intensities of the input checkered pattern, and then set them to 255 and 203 forthe left-eye image and 255 and 210 for the right-eye image.
4.2

Experimental results for static scenes

We tested our proposed method on static scenes for quantitative evaluation.
In a static scene, we can acquire the ground truth images of specular, diﬀuse,
and global components by using a number of spatially high-frequency patterns.
Specifically, we used 50 (= 2 × 5 × 5) projection patterns; a black-and-whitecheckered pattern and its reversed pattern were sifted along the horizontal andvertical directions 5 times respectively.
Optimization of projection pattern: First, we confirm the eﬀectiveness ofthe optimized projection pattern. Fig. 4 shows the ground truth images (toprow), our result images using the best (middle row) and the 5,000th best (bottomrow) combinations of the input intensities of the checkered pattern. This sceneconsists of a plastic pumpkin with specular and diﬀuse components and cottonwith global components. The pixel intensities of the 5,000th best combinations
are 169 and 45 for the left-eye image and 215 and 156 for the right-eye image.
We can see that our method using the best combination works better thanthat using the 5,000th best combination; block artifacts are more visible in the
result images using the 5,000th best combination. The PSNR and SSIM of theresult images in Table 1 quantitatively show the eﬀectiveness of the optimized
projection pattern.
Note that the separation using the 5,000th best combination was not robust;
we sometimes could not solve the set of equations in eq.(5) or eq.(6) due to ill6

The revolving speed is slower than the original color wheel, because our linear polarizer has larger moment.

12

Y. Nisaka et al.

ground
truth

ours

w/o
shift

w/o
weightspeculardiffuseglobalFig. 6. The ground truth images (top row), our result images (second row) , those
without shift (third row) and weight (fourth row).

Table 2. The quantitative comparison of Fig. 6. The numerical values in each cell are
the PSNR (upper) and SSIM (lower).

ours
w/o
shift
w/o
weightspecular
22.44
0.720
22.98
0.718
22.81
0.740

diﬀuse
24.60
0.703
22.61
0.610
23.02
0.688

global
25.01
0.498
21.78
0.358
22.81
0.405

average
24.02
0.640
22.46
0.562
22.88
0.611

conditioned problems, and then some of the result images are collapsed. Fig. 5
shows the output intensities an and bn of each color channel for the 1st andthe 5,000th best combinations. We can see that the output intensities of the 1st
combinations significantly vary and would yield better condition numbers forsolving eq.(5) or eq.(6).
3D mode for shifting projection pattern: Second, we confirm the eﬀectiveness of the 3D mode for quickly shifting projection pattern. Fig. 6 shows theground truth images (top row), our result images with (second row) and without(third row) shift. This scene includes a mirror hemisphere with specular components, a paper cup with specular and diﬀuse components, a ceramic shoe with
specular and diﬀuse components, and a soap with global components.

Fast Separation of Specular, Diﬀuse, and Global Components

13

60 fps

30 fpsspeculardiffuseglobalFig. 7. Our result images with 60 fps (upper) and 30 fps (lower).

We can see that line artifacts corresponding to the boundaries between theblocks of the checkered pattern are more visible in the result images without shift.
Comparing the PSNR and SSIM of those results in Table 2, the eﬀectiveness ofthe 3D mode for shifting projection pattern is clear quantitatively.
Weights for incorporating spatial correlation: Third, we confirm the effectiveness of the weights for incorporating spatial correlation. Fig. 6 shows theground truth images (top row), our result images with (second row) and without(fourth row) the weights for incorporating spatial correlation. We empirically set
the standard deviations of the Gaussian weights in Section 3.8.
We can see that the result images taking spatial correlation into consideration
are smoother than those without spatial correlation, i.e. pixel-wise separation.
Table 2 quantitatively shows the eﬀectiveness of incorporating spatial correlation
into specular, diﬀuse, and global separation.
4.3

Experimental results for dynamic scenes

Our proposed method achieves fast separation of specular, diﬀuse, and global
components with about 60 fps. Since we capture the eﬀects of the temporaldithering by using a high-speed camera, our method requires only a single projection pattern, i.e. two projection patterns in fact: a checkered pattern withgray intensities for the left-eye image and a shifted checkered pattern with grayintensities for the right-eye image.
On the other hand, consumer polarization cameras cannot capture the effects of the temporal dithering, because their frame rates are not so high. Then,
separating specular, diﬀuse, and global components by using a pair of a polarized light source and a polarization camera requires projector-camera temporalsynchronization and four projection patterns; a black-and-white checkered pattern, its negative-positive reversed pattern, and their shifted patterns. Therefore,
the frame rate of the polarization camera-based method is half in theory, if aprojector has the same frame rate as ours.

14

Y. Nisaka et al.

(0,1,1)(0.5,1,1)(2,1,1)(1,1,0)(1,1,0.5)(1,1,2)

original(s,d,g)=(1,1,1)

Fig. 8. Application to image-based material editing. The glossiness (upper) andtranslucency (lower) are edited.

Accordingly, we compared the performance of our proposed method for varying temporal resolution. Fig. 7 shows the result images with 60 fps and 30 fps. The
latter corresponds to the frame rate of the polarization camera-based method.
The object is a paper cup in motion. We can see that the texture on the paper cup is blurred due to motion in the result images with 30 fps. This result
demonstrates the advantage of our method over the polarization camera-based
method.
4.4

Application to image-based material editing

As a direct application of our specular, diﬀuse, and global separation, we conducted image-based material editing. Specifically, we synthesized various image sequences by linearly combining the separated components with diﬀerent
weights; s, d, and g for the specular, diﬀuse, and global components. Fig. 8
shows the original and synthesized images of a single frame; the glossiness andtranslucency are edited in the images at the upper and lower rows respectively.
We can see that our proposed method is eﬀective for photorealistic image-basedmaterial editing.

5

Conclusion and Future Work

We proposed a fast, robust, and easy-to-implement method for separating specular, diﬀuse, and global components of a dynamic scene by using a projectorcamera system. Specifically, our proposed method exploits the rapidly-varying
illumination conditions due to both the rotating linear polarizer and the temporal dithering. We conducted a number of experiments using our prototype setup
and showed the eﬀectiveness of our method and its components: the optimization of projection pattern, the use of the 3D mode for quickly shifting projection
pattern, and the weights for incorporating spatial correlation. Our future study
includes the extension to colored and polarized pattern projection; we can integrate a color wheel and a linear polarizer into a single colored and polarized
filter for multispectral and polarimetric scene analysis.

Fast Separation of Specular, Diﬀuse, and Global Components

15

Acknowledgments
This work was supported by JSPS KAKENHI Grant Numbers JP20H00612 andJP17H01766.

References
1. Amano, T.: Shading illusion: a novel way for 3-D representation on the paper
media. In: Proc. PROCAMS2012 (2012)
2. Bimber, O., Iwai, D., Wetzstein, G., Grundhöfer, A.: The visual computing of
projector-camera systems. Computer Graphics Forum 27(8), 2219–2245 (2008)
3. Grossberg, M., Peri, H., Nayar, S., Belhumeur, P.: Making one object look like
another: controlling appearance using a projector-camera system. In: Proc. IEEE
CVPR2004. pp. I–452–459 (2004)
4. Gu, J., Kobayashi, T., Gupta, M., Nayar, S.: Multiplexed illumination for scene
recovery in the presence of global illumination. In: Proc. IEEE ICCV2011. pp.
691–698 (2011)
5. Han, S., Sato, I., Okabe, T., Sato, Y.: Fast spectral reflectance recovery using DLP
projector. IJCV 110(2), 172–184 (2014)
6. Kobayashi, N., Okabe, T.: Separating reflection components in images under multispectral and multidirectional light sources. In: In Proc. ICPR2016. pp. 3199–3204(2016)
7. Maeda, K., Okabe, T.: Acquiring multispectral light transport using multi-primary
DLP projector. In: Proc. IPTA2016. pp. 1–6 (2016)
8. Mukaigawa, Y., Yagi, Y., Raskar, R.: Analysis of light transport in scattering
media. In: Proc. IEEE CVPR2010. pp. 153–160 (2010)
9. Narasimhan, S., Koppal, S., Yamazaki, S.: Temporal dithering of illumination forfast active vision. In: Proc. ECCV2008. pp. 830–844 (2008)
10. Nayar, S., Krishnan, G., Grossberg, M., Raskar, R.: Fast separation of direct andglobal components of a scene using high frequency illumination. In: Proc. ACM
SIGGRAPH 2006. pp. 935–944 (2006)
11. Press, W., Teukolsky, S., Vetterling, W., Flannery, B.: Numerical Recipes in C 3rd
Edition: The Art of Scientific Computing. Cambridge University Press (2007)
12. Raskar, R., Welch, G., Low, K.L., Bandyopadhyay, D.: Shader lamps: animating real objects with image-based illumination. In: Proc. EWRT2001. pp. 89–102(2001)
13. Schechner, Y., Nayar, S., Belhumeur, P.: A theory of multiplexed illumination. In:
Proc. IEEE ICCV2003. pp. 808–815 (2003)
14. Shafer, S.: Using color to separate reflection components. Color Research & Application 10(4), 210–218 (1985)
15. Shashua, A.: On photometric issues in 3D visual recognition from a single 2D
image. IJCV 21(1/2), 99–122 (1997)
16. Tomasi, C., Manduchi, R.: Bilateral filtering for gray and color images pp. 839–846(1998)
17. Torii, M., Okabe, T., Amano, T.: Multispectral direct-global separation of dynamic
scenes. In: Proc. IEEE WACV2019. pp. 1923–1931 (2019)
18. Wolﬀ, L., Boult, T.: Constraining object features using a polarization reflectancemodel. IEEE Trans. PAMI 13(6), 167–189 (1991)