Interreflection Removal Using Fluorescence
Ying Fu1 , Antony Lam2 , Yasuyuki Matsushita3 ,Imari Sato4 , and Yoichi Sato1
1

3

The University of Tokyo, 2 Saitama University,
Microsoft Research Asia, 4 National Institute of Informatics

Abstract. Interreflections exhibit a number of challenges for existing
shape-from-intensity methods that only assume a direct lighting model.
Removing the interreflections from scene observations is of broad interest since it enhances the accuracy of those methods. In this paper, we
propose a method for removing interreflections from a single image using
fluorescence. From a bispectral observation of reflective and fluorescentcomponents recorded in distinct color channels, our method separates direct lighting from interreflections. Experimental results demonstrate theeffectiveness of the proposed method on complex and dynamic scenes.
In addition, we show how our method improves an existing photometricstereo method in shape recovery.
Keywords: Fluorescence, bispectral model, bispectral interreflection model,
and interreflection removal

1

Introduction

Interreflection is a global light transport process whereby light reflected froma surface point illuminates other points. In the presence of interreflections, anintensity observation of a scene consists of the directly reflected light rays after
a single-bounce on a surface (direct component) and light rays that bounce offof the scene surface multiple times before they reach the camera (indirect component). Modeling and removing interreflections is of broad interest for making
shape-from-intensity methods to work properly, because most of them are designed to take only the direct component as input.
Recent studies on this problem provide deeper understandings about theinverse light transport and show that the direct and indirect components canbe separated for static scenes image [23, 18, 15]. In addition, some papers havetackled the problem for dynamic scenes but the capture process may require
specialized masks [18, 20] or motion compensation [1].
In this paper, we show that separation can be achieved for dynamic scenes
by capturing only a single image using fluorescence (see Fig. 1). Fluorescentmaterials1 not only reflect incident light but also absorb and emit light at longer
1

In practice, fluorescent materials show both ordinary reflection and fluorescent emission.

2

Ying Fu, Antony Lam, Yasuyuki Matsushita, Imari Sato, and Yoichi Sato
Blue channel

Direct component

Reflective-only

Capture underblue light
Fluorescent-Reflective

Light and albedo
calibration

Indirect component

Red channel

Fig. 1. Overview of the proposed method. A fluorescent object is captured under blue
light, whose blue channel is a reflective-only image and red channel is a wavelengthshifted fluorescent-reflective image. The lighting and albedo are calibrated by imagesof the flat white target and fluorescent sheet captured under blue and red light. Aftercalibration, direct and indirect components can be recovered.

wavelengths [13] (see Fig. 2) . This physical property allows us to obtain singleshot images that contain reflective-only images at the same wavelength as theilluminant and wavelength-shifted images that are a mixture of fluorescent emission and interreflections of those emissions. We call this wavelength-shifted image
a fluorescent-reflective image. We illustrate these concepts with an example. If ascene is illuminated by a blue light source, the reflective-only image is recordedin the blue channel, and the fluorescent-reflective image could be captured in the
red channel (see Fig. 1). Interreflections still exist in both channels but using
this bispectral measurement, we develop a direct-indirect decomposition method
by deriving a new interreflection model for fluorescence and extending Liao et
al. [15] where they used varying light colors and multiple images. Unlike Liao etal.’s method, our method only requires a single image with an assumption ofcommonly available fluorescent materials, which enables interreflection removalfrom a dynamic scene.
In summary, our main contributions are that we– derive a general interreflection model for fluorescent materials,– develop a method to separate direct-indirect components from reflective-only
and fluorescent-reflective measurements, and– show that a single-shot measurement is sufficient for the decomposition using
fluorescence.

2

Related Work

Fluorescence Fluorescent analysis has received attention in recent years incomputer vision. Examples of such work can be found in color rendering [11,25], reflectance and re-radiation modeling [10], camera spectral sensitivity estimation [8], 3D reconstruction [21, 24], immersion range scanning [9], and color
relighting of real scenes [4, 14]. In many of these methods, a phenomenon of fluorescence known as Stokes shift is exploited to achieve results. Specifically, Stokes

Interreflection Removal Using Fluorescence
1

Absorption
Emission

3

Fluorescent
Reflective

0.5

ࣅ࢏ ൏ ࣅ࢕
0
400

500
600
Wavelength (nm)700

ࣅ࢏ ൌ ࣅ࢕

(a)(b)
Fig. 2. (a) An example of absorption and emission spectra in the McNamara andBoswell fluorescence spectral dataset [16]. (b)When the fluorescent material is illuminated, it will reflect at the same wavelength and emit light at longer wavelengths.

shift can be described simply as the absorption of light at shorter wavelengths
and emission of light at longer wavelengths [13]. The way this works is that whenincident light hits a fluorescent surface, the surface’s absorption spectrum will
determine how much of the light is absorbed. Some of the absorbed energy isthen released in the form of an emission spectrum at longer wavelengths than theincident light. The remainder of the absorbed energy is released as heat. In thispaper, we take advantage of Stokes shift to assist with interreflection removal.
Interreflection There have been a number of methods that analyze interreflection and demonstrate applications such as shape recovery. Koenderink andDoorn [12] presented a general model for diffuse interreflections. Forsyth et al. [3,2] studied how interreflections affect shape recovery. Later, Nayar et al. [19] addressed the interplay between interreflections and shape by iteratively refining
the shape and reflectance of surfaces. They then extended their algorithm tocolored and multi-colored surfaces [17].
Despite the effectiveness of past approaches such as [19] and [17], the modeling and separation of interreflections is of broader interest because most shapefrom-intensity methods assume only direct lighting. Thus solving the problem
of separating out interreflections would allow for improving an entire body ofmethods in the literature. An early example of such work was presented by
Funt et al. [5, 6] where the color different effect in interreflections was used toseparate interreflections from direct lighting. Later, Seitz et al. [23] proved theexistence of an inverse light transport operator capable of separating m-bounced
light from scenes with uniform albedo and Lambertian surfaces. However, their
method requires a laser and scene capture is very time consuming. In addition,
their method is not robust to complex shapes. Nayar et al. [18] were able to separate direct and global lighting components for complex scenes using spatiallyhigh frequency illumination. In principle, their method would be very fast but
in practice, they require additional images or reduced resolution.
More recently, Liao et al. [15] removed interreflections by using spectrumdependent albedo in scenes but their method still needs two images captured
under different illuminations. In this paper, we show that by exploiting properties of fluorescence, interreflections can be removed using only using one color
image under one illumination. Our method also does not require a highly specialized setup and images can be captured using a standard RGB camera in astraightforward manner. This provides an advantage over other methods for in-4Ying Fu, Antony Lam, Yasuyuki Matsushita, Imari Sato, and Yoichi Satoterreflection removal on dynamic scenes that need more specialized setups suchas masks [18, 20] or motion compensation [1]. Like Liao et al. [15], our maindrawback is that the object needs to be homogeneously colored with the reflective and fluorescent component. However, even when an object does not satisfy
this condition, it is possible to color it using commonly available paint.

3

Bispectral Model for Interreflection Removal

In this section, we first introduce the bispectral model for fluorescent materials.
We then briefly review the interreflection model for ordinary reflective materials
with a uniform albedo and Lambertian surface and derive the interreflection
model for fluorescence. After that, we present our model to separate direct andindirect lighting based on fluorescence. Finally, we describe the practical issuesof lighting and albedo calibration.
3.1

Bispectral Model

In general, the appearance of fluorescent materials consists of both reflective
and fluorescent components [22]. If we assume a Lambertian surface, the local
reflection of the reflective component can be described as
Er (x, λ) = ρ(x, λ)L(λ) cos θ,(1)where ρ(x, λ) is the albedo of the surface point x at wavelength λ, L(λ) is theincoming radiance from the light source, and θ is the angle between the surface
normal and light source directions. We refer to ρ(x, λ) as the reflective albedo.
Unlike the reflective component that reflects light rays at the same wavelength as the incident light, the fluorescent component absorbs and emits lightat different wavelengths from the incident one [25, 10, 22]. Another interesting
property is that, as shown by Glassner [7], fluorescence emissions have no directional characteristics and are uniformly radiated in all directions. In other words,
fluorescence emissions behave like light reflected from a Lambertian diffuse surface. Therefore, the outgoing radiance Ef from a fluorescent material under thelight L(λi ) can be written as
Ef (x, λi , λo ) = η(x, λi , λo )L(λi ) cos θ,(2)where λi and λo represent the incident and outgoing wavelengths, respectively,and η(x, λi , λo ) is a direction-invariant function that describes the re-radiation
property of the fluorescence. According to the characteristics of fluorescence [22],η(x, λi , λo ) can be factored asη(x, λi , λo ) = µ(x, λo )a(x, λi ),(3)where a(x, λi ) and µ(x, λo ) define the absorption and emission factors of the fluorescent material, respectively. In the following, we call µ(x, λRo ) the fluorescent
albedo. The incoming radiance energy is absorbed by a(x, λi ) as a(x, λi )L(λi )dλi ;therefore, Eq. (2) can be rewritten asZEf (x, λo ) = µ(x, λo )a(x, λi )L(λi )dλi cos θ.
(4)Interreflection Removal Using Fluorescence5

Finally, the observation E(= Er + Ef ) becomesZ
E(x, λo ) = ρ(x, λo )L(λo )δ(λi − λo ) + µ(x, λo ) a(x, λi )L(λi )dλi cos θ, (5)in which Er and Ef are independently observed at two distinct wavelengths λi
and λo , respectively. The delta function δ(λi −λo ) is associated with the reflective
component because an ordinary reflective component only reflects light at thesame wavelength as its incident light (λi = λo ).
3.2 Bispectral Interreflection ModelWe briefly review the interreflection model for the ordinary reflective component on a Lambertian surface [12] and extend the interreflection model for the
fluorescent component.
Interreflection model for reflective component. The interreflection geometry [12] between points x ∈ R3 and x0 ∈ R3 is described by a kernel K
as
K(x, x0 ) =P os[n(x)T (x0 − x)]P os[n(x0 )T (x − x0 )],kx0 − xk2

P os[a] =a + |a|,2(6)where n(x) ∈ R3 is the surface normal at point x. The outgoing radiance of thereflective component Ir (x, λ) can be expressed as the sum of direct and indirectcomponents asZ
ρ(x, λ)ρ(x, λ)Ir (x, λ) =P (x, λ) +K(x, x0 )Ir (x0 , λ)dx0 ,(7)ππwhere P (x, λ) is the irradiance from the light source towards the surface pointx at wavelength λ. By defining iterated kernels Km asZ
K(x, x0 )K(x, y)K1 (x, x0 ) =, Km (x, x0 ) =Km−1 (y, x0 )dy (m > 1), (8)ππEq. (7) can be rewritten as the polynomial function of ρ asZ∞
P (x, λ) X m
P (x0 , λ) 0
Ir (x, λ) = ρ(x, λ)+ρ (x, λ) Km−1 (x, x0 )
dx .
π
π
m=2

(9)

By defining
R1 (x, λ) =Rm (x, λ) =P (x, λ),Z πP (x0 , λ) 0
dx
Km−1 (x, x )
π
0

(10)
(m > 1),

Eq. (9) becomes
Ir (x, λ) =∞Xρm (x, λ)Rm (x, λ).

(11)

m=1

as described in [15]. Interested readers can refer to [12], [19] and [15] for moredetails.

6

Ying Fu, Antony Lam, Yasuyuki Matsushita, Imari Sato, and Yoichi SatoInterreflection model for fluorescent component. Analogous to interreflections of the reflective component, the direct component of the fluorescentcomponent isRa(x, λi )P (x, λi )dλiµ(x, λo ).
(12)πFrom Eq. (6), we can see that the interreflection geometry K(x, x0 ) is independent of the albedo of the objects and the energy absorbed from the light source.
Also, the fluorescence emissions are typically not re-absorbed by the same fluorescent material again2 ; therefore, the interreflection of fluorescence emissions
behaves like that of the reflective components except for the initial emission.
Therefore, the second-bounce component can be written asRZa(x0 , λi )P (x, λi )dλi 0
0
dx ,
(13)
µ(x, λo )ρ(x, λo ) K1 (x, x )πwhere ρ(x, λo ) is the reflective albedo at the outgoing wavelength λo . The radiance of the fluorescent component at surface point x is, therefore represented as
the sum of the direct component and interreflections asRa(x, λi )P (x, λi )dλi
If (x, λo ) = µ(x, λo )
+
π
R
Z
∞
(14)
X
a(x0 , λi )P (x, λi )dλi 0µ(x, λo )ρm (x, λo ) Km (x, x0 )
dx .
π
m=1
By definingRa(x, λi )P (x, λi )dλi,
πRZa(x0 , λi )P (x, λi )dλi 0
Fm (x, λi ) = Km−1 (x, x0 )dx , (m > 1),π

F1 (x, λi ) =(15)

the interreflection model for the fluorescent component can be written as
If (x, λo ) = µ(x, λo )F1 (x, λo ) +∞Xµ(x, λo )ρm (x, λo )Fm+1 (x, λo ).

(16)

m=1

Unlike the conventional interreflection model for the reflective component (Eq. (11)),the derived model includes both fluorescent and reflective albedos at the outgoing wavelength λo .
3.3

Separation of Direct and Indirect Components

In theory, one would need an infinite-bounce model to fully describe interreflections. Fortunately, in practice, a 2-bounce model is sufficient for accurately modeling interreflections [6]. We thus restrict our attention to the 2-bounce case. If
2

The overlap between absorption and emission spectra is small. As a result, only anegligible amount of emitted light is re-absorbed.

Interreflection Removal Using Fluorescence

7(a) Blue light(b) Red light
Fig. 3. Lighting and albedo calibration.

we observe the reflective component at the same wavelength (λi = λo ) as theillumination, we obtain
Ir (x, λi ) = ρ(x, λi )R1 (x, λi ) + ρ2 (x, λi )R2 (x, λi ).

(17)

The fluorescent component can be observed at a longer wavelength as
If (x, λi , λo ) = µ(x, λo )F1 (x, λi ) + µ(x, λo )ρ(x, λo )F2 (x, λi ).

(18)

For convenience, we remove λi and λo in all functions in Eqs. (17) and (18) anduse ρ1 and ρ2 to represent the reflective albedo at incident wavelength λi andoutgoing wavelength λo , respectively. We also assume that the scene consistsof a uniform material, so the albedos ρ(x) and µ(x) will be the same for all
points x and can be represented as ρ and µ. Therefore, Eqs. (17) and (18) canbe rewritten as
Ir (x) = ρ1 R1 (x) + ρ21 R2 (x),If (x) = µF1 (x) + µρ2 F2 (x).
(19)
R
As will be detailed in Sec. 3.4, we condition P (x, λi ) = a(x, λi )P (x, λi )dλi so
Rm (x) = Fm (x)(m=1,2). Thus Eq. (19) can then be written as

 


Ir (x)
ρ1 ρ21R1 (x)=.
(20)
If (x)µ µρ2 R2 (x)We later detail how ρ1 , ρ2 and µ can be determined but provided they are
known, we can solve for R1 (x) and R2 (x) in Eq. (20) by a matrix inverse. From
this, ρ1 R1 (x) and µR1 (x) would give us the direct component in the reflective
and fluorescent parts respectively.
Compared with [15], our method requires one less image to capture. Since
the 2-bounce model can accurately approximate interreflections in practice, ourmethod can remove interreflections only with a single image. As demonstrated
in later in experiments, this allows the interreflection removal of a dynamic scene
by the straightforward recording of a video using a standard RGB camera.
3.4

Lighting and Albedo CalibrationR
The equality Rm (x) = Fm (x) holds only if P (x, λi ) = a(x, λi )P (x, λi )dλi ,which is not the general case. Furthermore, in order to solve Eq. (20), the valueof the albedos are required. We will address these two issues in the following.

8

Ying Fu, Antony Lam, Yasuyuki Matsushita, Imari Sato, and Yoichi Sato

Light Intensity Calibration As inR Eq. (19), Rm (x) = Fm (x) only whenirradiance P (x, λi ) is equal to A(x) = a(x, λi )P (x, λi )dλi . To make these twovalues equal, we would need to calibrate the fluorescent absorption spectrum andcontrol the light source carefully. This would be technically very challenging so
we instead use a simpler calibration procedure. Let us assume P (x, λi ) is α times
A(x), then for every surface point x, the radiance L(λi ) from theR light source
for the reflective component is also α times the absorbed energy a(λ)L(λi )dλi
for the fluorescent component with uniform albedo. That is,
α= R

L(λi )
P (x, λi )=R.
a(x, λi )P (x, λi )dλia(λi )L(λi )dλi

(21)

Because of this relation, we can calculate a single α for all points x. As defined in
Eqs. (10) andR (15), Rm (x) and Fm (x) are linearly dependent on the irradianceP (x, λi ) and a(x, λi )P (x, λi )dλi , respectively:
R1 (x) = αF1 (x), and R2 (x) = αF2 (x).
Therefore, Eq. (20) can be written asIr (x) = ρ1 R1 (x) + ρ21 R2 (x)µ(R1 (x) + ρ2 R2 (x)) .
If (x) = α

(22)

(23)

Albedo Ratio We discussed earlier that with known reflective albedos ρ1 andρ2 and fluorescent albedo µ, we can solve for the direct and indirect lighting
components. However, directly measuring these values is actually quite difficult.
From from Eq. (1) and Eq. (4), reflective albedo ρ and fluorescent albedo µ canbe described asρ(x, λ) =Er (x, λ)Ef (x, λo ), and µ(x, λo ) = R.
L(λ) cos θa(x)L(λi )dλi cos θ(24)

Determining the albedos from Eq. (24) is difficult because we would need toknow θ, the angle between the incident light and the surface normal.
Fortunately, a simple calibration procedure can be used to obtain albedo
ratios which would also be sufficient for our purposes. If we take ρ1 R1 (x) andρ21 R2 (x) as the unknown variables, Eq. (20) can be reformulated3 as

 


1
1
Ir (x)
ρ1 R1 (x)= µ µ ρ2.
(25)
ρ21 R2 (x)If (x)
ρ1 α ρ1 α ρ1Then measuring the relative surface albedo for a point x as opposed to theabsolute value of the albedo can be simply done with the following ratios derived
from Eq. (24)µEf (x, λ)=,ρ1 αEr1 (x, λ)
3

and

ρ2
Er2 (x, λ) L1 (λ)=,ρ1Er1 (x, λ) L2 (λ)(26)

The reformulation solves for scaled versions of R1 (x) and R2 (x) but this is still
sufficient for analyzing direct and indirect lighting.

Interreflection Removal Using Fluorescence

9

where L1 and L2 are blue and red lights respectively4 .
In the calibration process, the objects used to compute the albedo ratios
should be made of the same material as the object under consideration, as well
as being flat or convex to avoid any interreflections. As shown in Fig. 3, we usea
flat fluorescent sheet and a spectrally flat white reflectance target to calibrateEfµρ2ρ1 α and ρ1 . The scene is captured under blue and red light. Er1 is the ratio of
the intensity of the fluorescent sheet in the blue and red channels under the blue
Er2light. E
is the ratio of the intensity of the fluorescent sheet in the red channel
r1
L1
under the red light and the blue channel under the blue light. L
is the ratio of2
the intensity of the white target in the blue channel under the blue light and thered channel under the red light.
So far, the possibility to make Eq. (25) work is based on the assumptionthat the surface reflective albedo cannot be constant across the entire spectrum(ρ1 6= ρ2 ).

4

Experimental Results

We tested our method on real fluorescent objects. To obtain both reflective andfluorescent components effectively, we used pink fluorescent objects where thereflective component is strong in the blue channel and the fluorescent emission is strong in red channel. Such color characteristics make reflective-only andfluorescent-reflective capture ideal for use with a standard RGB camera. In practice, capture with other colors can also be done but different light sources orcamera filters would be needed to remove some wavelengths so that reflectiveonly and fluorescent-reflective images can be captured. Our method requires all
surface points to exhibit the same color. In a real application, objects could
be easily spray painted. We could then get the benefits of our method’s ability
to separate direct and indirect components for moving objects. For our experiments, we used objects that naturally exhibit fluorescence and a leaf dish thatwas painted. These objects were all illuminated by blue light and captured in
the blue and red channels of a CCD camera (SONY DXC-9000).
4.1

Separation Results

In this section, we show the results for the separation of direct and indirect components. We first give an overview of the entire process. As shown in Fig. 1, an
RGB image is first captured under blue light. Since we used our pink fluorescent color, the blue channel is a reflective-only image while the red channel isa fluorescent-reflective image. As described in Sec. 3.4, the lighting and albedo
are then calibrated by images captured under blue and red light, where the calibration targets are both flat. After calibration, direct and indirect components
are recovered by Eq. (25).
Our first scene is a pink fluorescent v-shape object. The scene was first taken
under blue light (Fig. 4 (a)). Its blue channel is then a reflective-only image
4

In our method, L1 should be in the short wavelength range, and L2 should be thesame appearance with the fluorescent emission spectrum so that it shows the spectralreflectance in the fluorescent emission area.

10

Ying Fu, Antony Lam, Yasuyuki Matsushita, Imari Sato, and Yoichi Sato(a) Input color image(b) Reflective-only(c) Fluorescent-reflective(d) Direct component(e) Indirect component
Fig. 4. Separation results for the v-shape. (a) The input image captured under blue
light. (b) The reflective-only image from (a)’s blue channel. (c) The fluorescentreflective image from (a)’s red channel. (d) The separated direct component. (e) The
separated indirect component.

(Fig. 4 (b)) and the red channel is a fluorescent-reflective image (Fig. 4 (c)).
We calibrated the lighting and albedo by using the method in the previous
section and found ρ1µα and ρ1µα ρρ12 to be 1.10 and 2.07 for the pink fluorescentsheet, respectively. After calibration, the direct and indirect components wereseparated by Eq. (25) as shown in Fig. 4(d) and (e). In the figures, we have scaled
the indirect component to 0 ∼ 255, for visualization purposes. We can see thatwhen we light one side of the v-shape, the direct component is strong on one side
while the indirect component is strong in other side. This observation fits our
expectations of how interreflections would physically behave and demonstrates
that our method removes interreflection effectively.
As mentioned, our method only requires a one-shot measurement to separate
the direct and indirect component and so is applicable to dynamic scenes. In
Fig. 5, seven successive video frames for a dynamic pink fluorescent cloth areInterreflection Removal Using Fluorescence

11(a) Input color image(b) Reflective-only(c) Fluorescent-reflective(d) Direct component(e) Indirect component
Fig. 5. Separation results for the moving fluorescent cloth. (a) The input image captured under blue light. (b) The reflective-only image from (a)’s blue channel. (c) The
fluorescent-reflective image from (a)’s red channel. (d) The separated direct component.
(e) The separated indirect component.

shown. We can see the changing of direct and indirect components in Fig. 5 (d)and (e) as a result of the object’s motion. The direct component is strong in theflat area (Fig. 5 (d)) and the indirect component is strong in the wrinkled area. As
with the v-shape, these observations fit our expectations of how interreflections
would physically behave and demonstrates our method’s effectiveness.
4.2

Photometric Stereo

To demonstrate a sample application and to further validate our separation results, we use the recovered direct component to perform photometric stereo [26].
We also tested using reflective-only and fluorescent-reflective images as inputs tophotometric stereo and show our recovered direct component provides the best
results. In our experiments, for each object, 12 images were captured with theobject illuminated from different light source directions. The light source wasabout 1 m away from the object, which was 4 ∼ 10 cm in diameter.
We tested our method on the pink fluorescent v-shape sheet and a “leaf dish”.
Fig. 6 shows one of the input images and the corresponding separated direct
and indirect components. The surface normals were recovered by photometricstereo and the shape was integrated from the normal map. Fig. 6(d)(h) show therecovered shapes by using our recovered direct components.

12

Ying Fu, Antony Lam, Yasuyuki Matsushita, Imari Sato, and Yoichi Sato(a) Input(b) Direct(c) Indirect(d) Reconstruction(e) Input(f) Direct(g) Indirect(h) Reconstruction
Fig. 6. Reconstructed results for the fluorescent v-shape and leaf dish. (a)(e) One of the
input images captured under the directional blue light. The corresponding separated
direct and indirect components are shown (b)(f) and (c)(g). (d)(h) The recovered shapeform the direct components.

In [19], Nayar et al. showed that interreflections cause recovered shapes tobe shallow. We quantitatively show the improvement in using our interreflection
removal method by comparing against recovered shape depths from using thereflective-only images and fluorescent-reflective images as inputs to photometricstereo.
We can see that the shape recovered from the fluorescent-reflective images(Fig. 7(b) green line) is more shallow than that from the reflective-only images(Fig. 7(b) blue line). This is because the reflectance spectrum is stronger inthe red channel than in the blue channel (Fig. 7(a)). Recall that the fluorescentindirect component shown in Eq. (25), ρ1µα ρρ12 = 2.07 is greater than the reflective
indirect component which is 1. Thus the indirect component 2.07ρ21 R2 (x) in the
fluorescent-reflective images is stronger than the ρ21 R2 (x) in the reflective-only
images. From Fig. 7(a), we also see that the spectral reflectance has a strong
overlap with the fluorescence emission spectrum. In the fluorescent photometricstereo work of [21] and [24], they both assume that spectral reflectance does notoverlap with the fluorescent emission spectrum. This is not always the case andour method addresses situations where spectral reflectance does overlap with thefluorescent emission spectrum.

Interreflection Removal Using Fluorescence1

13

Reflectance
Emission

0.5

0
400

450

500
550
600
Wavelength (nm)

650

700

(a)
(b)
(c)
Fig. 7. (a) The reflectance and emission spectra for the fluorescent pink sheet. (b)and (c) The cross-sectional view of recovered shapes from fluorescent-reflective images (green line), reflective-only images (blue line) and direct components (dark line)
superimposed into the side view of the v-shape and leaf dish, respectively.

Finally, the recovered shape from the direct component (Fig. 7(b) dark line)
is deepest one, which indicates that our method removes the indirect component effectively. The ground truth angle between the two sides for the v-shape
is about 90 degrees. The recovered angles for the shape from the direct component, reflective-only and fluorescent-reflective cases are 91, 101 and 114 degrees,
respectively. Thus our recovered direct component provided a result very close
to the ground truth. Similar results can be seen for the leaf dish in Fig. 7(c).

5Conclusion

We presented a novel method for separating direct and indirect components using the phenomenon of fluorescence. By exploiting Stokes shift, we were able toobserve separated reflective and fluorescent components simultaneously in two
channels of an RGB camera. While both channels still contained interreflections,
we devised a simple but effective decomposition method for separating out interreflections. From this, we derived a general interreflection model for fluorescentmaterials and showed that a single shot measurement is sufficient for the decomposition. In contrast to existing methods, this single shot ability allowed for
effective operation on complex dynamic scenes as demonstrated in the experiments. In addition, we showed that our method’s effective recovery of the directcomponent greatly improved an existing photometric stereo method. Although
we only demonstrated this on the photometric stereo method, our method canbe easily applied to any methods that assume direct lighting and a uniform Lambertian surface. In the future, we would like to explore the benefits of separating
for dynamic scenes through applications such as dynamic color relighting.

References
1. Achar, S., Nuske, S., Narasimhan, S.G.: Compensating for motion during directglobal separation. In: Proc. of International Conference on Computer Vision(ICCV). pp. 1481–1488 (Dec 2013)
2. Forsyth, D., Zisserman, A.: Mutual illumination. In: Proc. of IEEE Conference on
Computer Vision and Pattern Recognition (CVPR). pp. 466–473 (Jun 1989)
3. Forsyth, D., Zisserman, A.: Shape from shading in the light of mutual illumination.
Image Vision Computing 8(1), 42–49 (1990)
4. Fu, Y., A., L., Sato, I., Okabe, T., Sato, Y.: Separating reflective and fluorescentcomponents using high frequency illumination in the spectral domain. In: Proc. ofInternational Conference on Computer Vision (ICCV) (2013)

14

Ying Fu, Antony Lam, Yasuyuki Matsushita, Imari Sato, and Yoichi Sato

5. Funt, B.V., Drew, M.S., Ho, J.: Color constancy from mutual reflection. International Journal of Computer Vision (IJCV) 6(1), 5–24 (1991)
6. Funt, B., Drew, M.: Color space analysis of mutual illumination. IEEE Trans.
Pattern Analysis and Machine Intelligence (PAMI) 15(12), 1319–1326 (1993)
7. Glassner, A.S.: A model for fluorescence and phosphorescence. In: Photorealistic
Rendering Techniques, pp. 60–70. Springer Berlin Heidelberg (1995)
8. Han, S., Matsushita, Y., Sato, I., Okabe, T., Sato, Y.: Camera spectral sensitivity estimation from a single image under unknown illumination by using fluorescence. In: Proc. of IEEE Conference on Computer Vision and Pattern Recognition(CVPR) (2012)
9. Hullin, M.B., Fuchs, M., Ihrke, I., Seidel, H.P., Lensch, H.P.A.: Fluorescent immersion range scanning. ACM Trans. on Graph. (ToG) 27, 87:1–87:10 (2008)
10. Hullin, M.B., Hanika, J., Ajdin, B., Seidel, H.P., Kautz, J., Lensch, H.P.A.: Acquisition and analysis of bispectral bidirectional reflectance and reradiation distributionfunctions. ACM Trans. on Graph. (ToG) 29, 97:1–97:7 (2010)
11. Johnson, G., Fairchild, M.: Full-spectral color calculations in realistic image synthesis. IEEE Computer Graphics and Applications 19, 47–53 (1999)
12. Koenderink, J.J., vanDoorn, A.J.: Geometrical modes as a general method to treat
diffuse interreflections in radiometry. Journal of the Optical Soceity of America(JOSA) 73(6), 843–850 (1983)
13. Lakowicz, J.R.: Principles of Fluorescence Spectroscopy. Springer (2006)
14. Lam, A., Sato, I.: Spectral modeling and relighting of reflective-fluorescent scenes.
In: Proc. of IEEE Conference on Computer Vision and Pattern Recognition(CVPR) (2013)
15. Liao, M., Huang, X., Yang, R.: Interreflection removal for photometric stereo byusing spectrum-dependent albedo. In: Proc. of IEEE Conference on Computer
Vision and Pattern Recognition (CVPR). pp. 689–696 (2011)
16. McNamara, G., Gupta, A., Reynaert, J., Coates, T.D., Boswell, C.: Spectral imaging microscopy web sites and data. Cytometry. Part A: the journal of the International Society for Analytical Cytology 69(8), 863–871 (2006)
17. Nayar, S.K., Gao, Y.: Colored interreflections and shape recovery. In: Proceedingsof the Image Understanding Workshop (1992)
18. Nayar, S.K., Krishnan, G., Grossberg, M.D., Raskar, R.: Fast separation of direct
and global components of a scene using high frequency illumination. In: ACM
SIGGRAPH. pp. 935–944 (2006)
19. Nayar, S., Ikeuchi, K., Kanade, T.: Shape from interreflections. International Journal of Computer Vision (IJCV) 6(3), 173–195 (1991)
20. O’Toole, M., Mather, J., Kutulakos, K.N.: 3d shape and indirect appearance bystructured light transport. In: Proc. of IEEE Conference on Computer Vision and
Pattern Recognition (CVPR) (Jun 2014)
21. Sato, I., Okabe, T., Sato, Y.: Bispectral photometric stereo based on fluorescence. In: Proc. of IEEE Conference on Computer Vision and Pattern Recognition(CVPR) (2012)
22. Sato, I., Zhang, C.: Image-based separation of reflective and fluorescent components
using illumination variant and invariant color. IEEE Trans. Pattern Analysis andMachine Intelligence (PAMI) 35(12), 2866–2877 (2013)
23. Seitz, S.M., Matsushita, Y., Kutulakos, K.N.: A theory of inverse light transport.
In: Proc. of International Conference on Computer Vision (ICCV). pp. 1440–1447(2005)
24. Treibitz, T., Murez, Z., Mitchell, B.G., Kriegman, D.: Shape from fluorescence. In:
Proc. of European Conference on Computer Vision (ECCV). pp. 292–306 (2012)Interreflection Removal Using Fluorescence

15

25. Wilkie, A., Weidlich, A., Larboulette, C., Purgathofer, W.: A reflectance modelfor diffuse fluorescent surfaces. In: International conference on Computer graphics
and interactive techniques. pp. 321–331 (2006)
26. Woodham, R.J.: Photometric method for determining surface orientation frommultiple images. Optical Engineering 19(1) (1980)