Illumination Brush: Interactive Design of All-frequency Lighting
Makoto Okabe∗

Yasuyuki Matsushita†∗ The University of Tokyo

Takeo Igarashi∗,‡

Li Shen†† Microsoft Research Asia

‡ SORST, JST

Abstract
We present an appearance-based user interface forartists to efficiently design customized image-based lighting environments. 1 Our approach avoids typical iterationsof parameter editing, rendering, and confirmation by providing a set of intuitive user interfaces for directly specifying the desired appearance of the model in the scene. Thenthe system automatically creates the lighting environment
by solving the inverse shading problem. To obtain a realistic image, all-frequency lighting is used with a spherical radial basis function (SRBF) representation. Rendering
is performed using precomputed radiance transfer (PRT)
to achieve a responsive speed. User experiments demonstrated the effectiveness of the proposed system compared
to a previous approach.

1. Introduction
Image-based lighting, or the environment map, is amethod of representing a large-scale lighting environment
around a target scene as a texture map [5]. It compactly represents complicated incoming light from distant sources andenables real-time rendering of the scene with realistic lighting effects [28]. However, most systems rely on captured
environments for image-based lighting [6, 10], and few attempts have been made to manually design such complex
lighting environments. The artist might paint the environment map directly, but this is labor-intensive and makes itvery difficult to obtain the desired rendering result due tothe non-intuitive relation between lighting conditions andthe appearance of objects. Because a captured environment
is not always available, a practical method for manually designing complicated lighting environments is in great demand.
We propose an appearance-based user interface for designing image-based lighting environments. Instead of
placing lights in the surrounding environment, users candirectly specify the appearance of the resulting image by
painting and dragging the color of outgoing radiance on thetarget model. Then the system constructs an appropriate
image-based lighting model by solving the inverse lighting
1 This work was done while the first author was visiting Microsoft Research Asia.

(a)

(b)

(c)

(d)

Figure 1. Designing image-based lightingwith Illumination Brushes. The bunny model
has a measured white glossy BRDF.(a) The
user paints the desired diffuse appearance
directly on the 3D model. Pink and orange
diffuse brushes are shown. (b) The scene
is rendered using the estimated environment
map. The user paints a blue highlight on
the bunny with specular brush. (c) All of thepainted lighting effects are satisfied by rendering the bunny using the designed imagebased lighting environment shown in (d).
problem. Other appearance-based interfaces for lighting design have been proposed [19, 24, 20, 12, 14, 1, 26, 2, 18],
but all of them are limited to simple lighting models, such as
point or directional lights, and are not designed for imagebased lighting, which can produce more appealing results
than simple lighting. It is particularly useful for adding realistic lighting effects to synthetic objects when compositing
them into a photographed or video-recorded background.
Our system also supports the design of high-dynamic range(HDR) lighting on a standard low-dynamic range (LDR)
monitor by introducing interactive tone-mapping, in whichusers specify a region of interest (ROI) and the system automatically adjusts the tone-mapping parameter.
Given the surface appearance specified by the user, thesystem must estimate the corresponding image-based lighting model. This process is formalized as an inverse lighting
problem, that is, recovering unknown lighting from known
geometry, outgoing radiance, and the surface bidirectionalreflectance distribution function (BRDF) [21]. To solve thisproblem at a responsive speed, we represent an image-based
lighting model using the spherical radial basis functions(SRBF) [29], and estimate coefficients in a precomputed

radiance transfer (PRT) framework [27, 28]. The SRBF
representation allows our system to represent all-frequency
lighting effects and arbitrary BRDFs, which can be either
analytic or measured. The system allows interactive viewchanges and quick update of the lighting environment according to user input.
Our current implementation supports dynamic editing of
HDR image-based lighting for a fixed geometry with arbitrary BRDF. It can handle both specular and diffuse lighting
effects with appropriate treatment of self-occlusions (softshadows). More advanced effects, such as texture mapping,interreflections, and deformation, will be addressed in future work.

2

Prior Work

Several methods have been proposed to manipulate lighting effects in a rendered 3D scene, as an alternative to manually setting raw lighting parameters. Pellacini et al. proposed using cast shadows for lighting design [19]. Someother methods use sketching or painting interfaces that allow users to directly draw lighting effects [24, 20, 12, 2,
26, 18]. While these methods are effective, they are limited to simple local lighting models, or simple geometry.
Other related works have incorporated illumination control
using psychophysical effects [14], and interactive controlof highlighted shapes in cartoon animations [1]. Automatedlighting design systems are also proposed [25, 9].
Our system utilizes inverse rendering, estimating the distribution of light sources given the scene geometry, surface
properties, and the final image. Previous methods have estimated light source distributions from a variety of different
input types, such as cast shadows [23] and shading [30].
Li et al. used integrated cues of shading, shadows, andspecular reflections [15]. Nishino and Nayar proposed amethod for capturing an environment map reflected by ahuman eye in an image, and using it for relighting [17]. Ramamoorthi and Hanrahan proposed methods for estimating
either a material BRDF, an illumination condition, or bothfrom a single image or multiple images of a scene [21]. Our
work builds on these previous results and provides a practical method suitable for interactive editing without excessive
precomputation.

3

User Interface

This section describes our prototype system for an
appearance-based lighting design from the artist’s point ofview. The system allows users to directly specify the appearance of the final image by painting the outgoing radiance on the target model. The system also provides dragging tools to modify the lighting environment. The lighting environment and screen image are updated in real time.
Users can freely change the viewpoint during the lighting

Figure 2. Our prototype system.
design process. All light sources are assumed to be distant,and the painting process changes the positions, colors, intensities, and shapes of light sources. The system assumes
a known geometry and a predefined BRDF for each pointon the surface of the target model. Figure 2 shows a snapshot of the prototype system. Given a new 3D scene, theuser must first run several minutes of precomputation before starting an interactive lighting design session.

3.1

Illumination Brushes

Illumination brushes are painting tools for specifying thedesired appearance of a surface. Users draw a stroke on thescreen using a dragging operation. The stroke is projected
onto the object surface and serves as a constraint that specifies the color of nearby mesh vertices to estimate the lighting environment. Color and position can also be changed
later in the process. The system provides two types of illumination brushes: the diffuse brush, for designing diffuse
lighting effects, and the specular brush, for editing specular
lighting effects. An example is shown in Figure 3. Note that
specular highlighting is view-dependent and moves along
the surface as a user changes the viewpoint, while the diffuse component is independent of viewing direction.
For both brushes, users simply select a color and begin
painting on the 3D model. The system immediately andcontinuously estimates the corresponding illumination andupdates the image on the screen during the painting process.
However, user input can be inconsistent. Figure 4 shows
two such examples. In (a), the user paints white and black
lighting effects at different positions with the same surface
normal. Because no lighting environment can exactly accommodate this input, the system tries to satisfy the contradictory constraints as much as possible in a least squares
sense. In this example, Figure 4 (b) shows that the painted
positions will have the same gray color, which is the average of the painted lighting effects. In (c), the user paints
white and black lighting effects at different positions withthe same surface normal. Such input seems to be inconsistent, however, because the area painted in black can be

(a)
(b)
Figure 3. Painting with illumination brushes.
The bunny model has a glossy aluminumbronze BRDF. (a) A user paints green andpurple lighting effects on the 3D model using diffuse brushes. The resulting lighting environment (shown as a cross) is diffused andblurry, which is reflected in the rendered image. (b) The same colors are painted at thesame positions using specular brushes, which
results in a lighting environment with sharperfeatures and specular lighting effects.
in the statue’s shadow, the input is consistent and thus thesystem can compute the lighting environment accordingly,
which results in a soft shadow under the statue (d).

3.2

Manipulation of Painted Strokes

The result of a painting operation is stored as a strokeon the object surface, whose color and position can then be
modified. To change the color of a stroke, the user simply clicks it and adjusts its color using the color panel. Tochange the location of the stroke, the user clicks and drags
the stroke on the object surface. As the dragged stroke
moves, it adapts its shape to match the surface (Figure 5).
These operations are useful for locally adjusting the appearance of the scene. Similar operations are used in [11, 3].
If the viewpoint is changed, the position of a stroke nolonger corresponds to the created highlight. To help users
modify strokes in a consistent manner, the system automatically recovers the original viewpoint each time a stroke isclicked.

3.3

Interactive Tone-Mapping

A standard monitor can display only LDR images, wherecolor is represented as 8 bits per channel. When designinga lighting environment, however, editing in such a narrow
range is a severe limitation. To enable users design in a
HDR domain, we provide an interactive tone-mapping interface. Users specify the ROI using a rubber band on thescreen, as shown in Figure 6, and the system automaticallyadjusts the exposure so that the contrast in the ROI is maximized. The manipulated color is always converted into the
HDR domain using the exposure settings. Users can alsochange the exposure directly by controlling a slider.

(a)
(b)
(c)
(d)
Figure 4. Painting with inconsistent multiple colors. The Buddha model has a diffuse BRDF. (a) The user paints white andblack lighting effects at different positions
with the same normal, resulting in a uniform
gray color (b). In (c), the user repeats thisinconsistent input, but at a different locationon the 3D model. Because the region painted
black is identified as the statue’s shadow, theinput becomes consistent, resulting in a soft
shadow under the statue (d).

3.4

Rotation of Lighting Environment

During the light-editing process, users may want torotate the entire lighting environment. We provide anappearance-based tool to achieve this with an intuitive dragging operation on the scene surface. Users can grab any
feature, such as highlights and shadows on the surface, andmove it to another location (Figure 7). Internally, the system rotates the entire lighting environment so that the color
under the mouse cursor remains constant while dragging.
Users can directly rotate the light sphere representing theentire lighting environment, but dragging inside the scene(e.g., dragging cast shadows around the scene) can be moreconvenient in some cases. Figure 8 illustrates an example,
where identical dragging operations result in opposite rotations depending on the scene geometry and lighting conditions.

4

Algorithm

In this section, we describe the algorithms for the interactive rendering and user interfaces. To develop the interactive rendering algorithm, we used the SRBF-based PRTframework proposed by Tsai et al. [29], for several reasons.
First, the rendering speed is fast, allowing users to preview
rendering results in real time. Second, unlike spherical harmonics or wavelet-based systems, the basis function is in
the spatial domain rather than the frequency domain. This
makes the estimation algorithm simple and stable. Third,
SRBFs can be added progressively to locally control the illumination resolution.

(a)
(b)
(c)
Figure 5. Manipulation of painted strokes. (a)The user selects the orange painted strokeat the top of the fish model. (b) The user
drags the stroke to a different location, andthe environment lighting is updated accordingly. (c) The color of the stroke is modified
from orange to blue.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 7. The user clicks and drags thewhite-lit part to rotate the lighting environment (a-c). The lighting environment rotatessmoothly so that the surface colors underthe mouse cursor remain constant. The user
can also click and drag the shadow under theteapot (d-f).
(a)

(b)

(c)

Figure 6. The system allows scene editing
with tone mapping. (a) The ROI is selectedusing a red rubber band. Because the intensity distribution in the ROI is relatively bright,
the system adjusts the exposure to make theentire scene darker (b). The bottom-left image of (b) shows the original exposure. The
dark area is selected with the red rubber
band, and the system makes the entire scene
brighter (c).

The rendering algorithm is based on the previous PRT
framework [29]: the system renders the scene in real timeby fixing the geometry and BRDF while the user interactively changes the view and lighting. The major problem
with the previous method is that the precomputation takes aprohibitively long time before start painting a new 3D scene(e.g., up to 24 h for a 3D scene with 50,000 vertices). Therefore, our system approximates the result using a simplified
method to reduce precomputation time. First, instead of accurately modeling the shading of a triangle in the spirit ofPhong’s model, we simply interpolate the colors of the meshvertices to render a triangle. Second, we use a simplified
method to approximate a function with SRBF. Instead ofsolving a global least-squares system to compute the SRBF
coefficients, we determine the coefficients of each SRBF
independently assuming that each SRBF component is relatively independent. These changes shorten the precompu-

Figure 8. The same dragging operations canresult in opposite rotations, depending on
the geometry and lighting conditions.

tation time from 1 day to several minutes. It also slightly
reduces rendering accuracy and speed (it takes 1 second
to render a scene with 10,000 vertices when the view ischanged), but remains sufficiently fast for a preview during
interactive design.

4.1

Interactive Rendering

Representation For distant illumination, the outgoing radiance I from a surface point x is described by the rendering
Equation [13] as follows:Z

I(x, ω o ) =L(ωi )R(x, ωi , ωo )V (x, ωi ) max(0, ωi · n)dωi , (1)Ωiwhere ωi and ωo are the light and view directions, respectively; L is the incoming ray from the environment lighting;
R is the BRDF; V is the visibility map (which takes [0|1]);
and n is the surface normal at x. We rewrite the Equation (1)as follows:Z

I(x, ωo )=L(ωi )T (x, ωi , ωo )dωi ,Ωi(2)T (x, ωi , ωo )def=

R(x, ωi , ωo )V (x, ωi ) max(0, ωi · n),(3)

If the viewpoint, ωo , is fixed, then T (x, ωi , ωo ) = Tx (ωi ) andI(x, ωo ) = Ix . The subscript x indicates that the quantity is
associated with the vertex x.
We represent the light transfer function, Tx (ωi ),and the distant illumination, L(ωi ), with SRBF expansions. The SRBF centers of light transfer function, ΞT ={ ξT,1 , ..., ξT,nt }, and the SRBF centers of lightsources, ΞL ={ ξL,1 , ..., ξL,nl }, are the sets of distinct points
on the unit sphere, S2 . The associated bandwidth parameters ΛT ={λT,1 , ..., λT,nt } and ΛL ={λL,1 , ..., λL,nl } are defined by sets of real numbers. Tx (ωi ) and L(ωi ) can be represented by the SRBF function G and coefficients as

Rendering Our system renders the scene by computing
the right-hand side of Equation (6). We have implemented
our system on a platform that includes a GeForce 8800 GPU
and an Intel Core2 2.4-GHz CPU. Computation of Tx andmultiplication of AL is performed using the CPU, and otheroperations take place on the GPU. When the user changes
viewpoints, the system computes Tx for all visible vertices.
Computation of Tx requires access to the visibility map ofx and BRDF data. On average, this process takes about 1
second for 10, 000 vertices.
Our system allows the user to change the lighting in realtime (20-60 FPS), but view changing runs at an interactive
rate (several seconds) that is proportional to the number ofvertices. For this reason, our system uses a simplified model
during camera motion.

nt

Tx (ωi )≈∑ tx,i G(ωi , ξT,i ; λT,i ),

(4)

i=1
nl

L(ωi )≈∑ l j G(ωi , ξL, j ; λL, j ),

(5)

j=1

where G(·, ξ ; λ ) is an SRBF centered at ξ with the bandwidth parameter λ . In the following sentences, T-SRBF isdescribed by G(ωi , ξT,i ; λT,i ), and L-SRBF is described byG(ωi , ξL, j ; λL, j ). By combining Equations (2–5), we obtainthe following matrix notation:

4.2

Estimation of Lighting Environment

We initialize the lighting environment with uniformly
distributed L-SRBFs obtained by subdividing the icosahedron several times, usually twice to provide 162 L-SRBFs.
We describe a method to estimate the light source coefficients from diffuse and specular brushes. We also describe
a method of adaptively controlling the number of L-SRBFs.
Basic Formulas Let T0 x = (TTx A)T . Then Equation (6)can be written as
0T
Ix = T x L.

(8)

(6)

Given a set of the user-painted colors, I 0 = {I10 , ..., In0 p }, on

where Tx = [tx,1 tx,2 . . . tx,nt ] is a light transfer coefficient
vector, L = [l1 l2 . . . lnl ] is a light source coefficient vector,
and the (m, n) component of the matrix A is given by

the painted vertices, P = {p1 , ..., pn p }, the system estimatesa new lighting environment such that rendered vertex colors
are as close as possible to the user-specified colors. The
simple form of the estimation problem can be written in aleast squares form as

Ix = TTx AL,

Z

Amn =

G(ωi , ξT,m ; λT,m )G(ωi , ξL,n ; λL,n )dωi .

(7)

Ωi

Equation (7) is called spherical singular integral, and it canbe efficiently evaluated for Gaussian SRBF as described in[29].
Precomputation To make the initial set of T-SRBF centers, ΞT , we use an icosahedron subdivided three times toprovide 642 vertices, and project them on the unit sphere
S2 . ΛT is assigned the same value based on the relationshipbetween the variance and the bandwidth parameters for theGaussian SRBFs: σ 2 = 1 − (coth(2λ ) − (2λ )−1 ) [16]. We
fixed the variance at π/40 radians to derive ΛT . It is possible to increase the number of T-SRBF centers to increase
the accuracy.
Several minutes of precomputation of the visibility map
V (x, ωi ) are required for each new 3D scene. The visibility
map of each vertex is obtained by setting the camera at thevertex and rendering the scene into a cube map with flat
shading. To avoid aliasing, we first render the cube map at
a resolution of 6 × 128 × 128, and then downsample it to
6 × 32 × 32.

np

TL̂ = argmin ∑ (Ii0 − T0 pi L)2 ,
L

(9)

i=1

where L̂ = [lˆ1 lˆ2 . . . lˆnl ]T is the estimated light source coefficient vector. To derive a unique solution in the underconstrained case, we add a regularization term. Equation (9)can then be written asnp

TL̂ = argmin ∑ (Ii0 − T0 pi L)2 + κL2 ,
L

(10)

i=1

where κ, the regularization factor that penalizes models toprevent overfitting, is empirically set to 1.0e − 4.
Diffuse and Specular Brushes As we have described in
the user interface section, using the diffuse brush changes
the lighting of the overall environment while the specularbrush changes the lighting only in the direction of specular reflection. Figure 9 shows our design philosophy foreach brush. We obtain these effects by assigning a different weight to each light source. That is, when the specularbrush is used, the system assigns smaller weights to the lightsources located in the direction of specular reflection so thatthese light sources are mainly modified. On the other hand,

(a)

(b)

ScreenFigure 9. The user is painting an orange color
on a glossy surface point. (a) The specularbrush produces effects for light sources located in the direction of specular reflection.
This results in a specular lighting effect. (b)The diffuse brush produces effects for lightsources located off the axis of specular reflection. This results in a diffuse lighting effect.
when the diffuse brush is used, the system assigns smaller
weights to the light sources off the direction of specular reflection. To incoporate these features in the process, wechange Equation (10) tonpTL̂ = argmin ∑ (Ii0 − T0 pi L)2 + κ(diag(W)L)2 .
L

(11)

i=1

Solving the Linear System To avoid negative lighting intensity, the estimated lighting L̂ must satisfy the following
condition:lˆi ≥ 0Figure 10. Dragging a stroke on the object
surface. The system moves the grabbededge and then determines the locations of
the surrounding edges.
L-SRBFs. As mentioned earlier, there are 162 L-SRBFs,
as determined by the number of vertices in an icosahedronsubdivided twice. While the original number of L-SRBFs
is limited, the resolution can be locally enhanced by adding
more. When the user provides new input to the system using the specular brush, an L-SRBF is added in the directionof the specular reflection. This mitigates the low-sampling
problem of the illumination and ensures that the light sourcein the highlight direction always exists when edited.

4.3

where W = [w1 w2 . . . wnl ]T are the weighting factors controlling the variability of the light sources. Smaller valuesof wi means larger changes in li , and vice versa.
To determine components of W, we use the light trans0 t 0 . . . t 0 ], that reprefer coefficient vector, T0x = [tx,1x,nlx,2sents which light source coefficient is important in determining Ix (see Equation (8)). The weighting factors
Wx = [wx,1 wx,2 . . . wx,nl ]T at painted vertex x are determined as follows: if the vertex x is painted with the
diffuse brush, the weighting factors are given by wx,i ∝
0 /max {t 0 }, and if the vertex x is painted with the spectx,ij x, jular brush, the weighting factors are given by wx,i ∝ 1 −
0 /max {t 0 }. We assign weighting factors of zero to lighttx,ij x, jsources that are invisible at the surface point x. Finally, W
is computed as W = ∑ W pi for all painted vertices, pi . To
avoid a zero weight factor, we add ε = 2.0e − 4 to each element of W.

∀i ∈ [1, nl ].

(12)This gives us a least square problem with inequality constraints. We use the Active Set method [7] on UMFPACK
Version 4.1 [4] to solve the problem iteratively.
Adaptive Control of Number of L-SRBFs The originalenvironment map is represented by uniformly distributed

Object surface

Dragging Painted Strokes

When the user grabs and drags a stroke, the systemmoves the grabbed edge on the surface and then determines
the location of the neighboring vertices one by one so thatthe local positional relationships among neighboring vertices are maintained (Figure 10). This process is viewed
as a walking operation on the surface: given the original
location and the target location of an edge (vi , vi+1 ), the target location of a neighboring vertex vi+2 is determined sothat both the angle between edges (vi , vi+1 ) and (vi+1 , vi+2 )projected onto the tangent plane at vi+1 and the geodesic
distance between vi+1 and vi+2 remain constant. The location of the dragged edge is determined so that it follows
the mouse cursor and its orientation on the screen remains
constant.

4.4

Rotation of Environment Lighting

The lighting environment can be revolved by rotating thecenters of the L-SRBFs (ξ ). When users perform a dragging
operation using the rotation tool, the system automatically
updates the rotation angles so that the radiance under themouse cursor remains the same. To be more precise, thecomputation proceeds as follows (Figure 11). Suppose thatthe mouse cursor moves from the screen coordinate p0 to
p1 . The corresponding surface vertices are v0 and v1 . The
system estimates the new orientation minimizing the difference between the original radiance Iv0 at v0 and the newLighting rotated by R

Lighting rotated by R’

Iv 1

Vertices
Precomputation (sec)Viewpoint (msec)
162 L-SRBFs (FPS)
642 L-SRBFs (FPS)

Iv’1

Iv 0

surface radiance Iv0 1 at v1 after rotation.
We solve this problem using a standard minimization
method. The rotation matrix R is represented by three rotation angles: α, β , and γ. The system uses the smallest
changes to the rotation angles (dα, dβ , and dγ) that satisfy Iv0 1 = Iv0 . To achieve this, the system solves the following minimization problem using the Lagrange multipliermethod:min{dα 2 + dβ 2 + dγ 2 } s.t. dIv1 =

∂ Iv1
dJ,
∂J

To achieve the operation described in Section 3.3, we use
Reinhard et al. ’s global tone-mapping operator [22], which
can be efficiently implemented using programmable graphics hardware [8]. The operator calculates the log averageluminance L̄w as
L̄w = exp
1log(δ + Lw (x, y)) ,
∑
N x,y

(14)

where N is the number of pixels in the image, (x, y) represents the pixel coordinate, and δ is a small constant used to
avoid numerical underflow when Lw (x, y) = 0. The source
luminance values (Lw (x, y)) are mapped to the relative luminance (Lr (x, y)) bya
Lw (x, y),L̄w

Lr (x, y) =(15)

where a is a key value that controls whether the tonemapping image appears relatively bright or relatively dark.
Then Lr (x, y) is mapped to the display luminance Ld (x, y)byL (x, y)Ld (x, y) =r

1 + Lr (x, y)

.

(16)

Finally, the RGB display values Rd , Gd , and Bd are determined by
Rw
Lw

Fish
7,634
127
696
62
30

Table 1. Time and rendering performance ofthe proposed approach. The run-time FPS in
the last two rows lists the rendering performance for lighting environment changes witheither 162 or 642 L-SRBFs.

b, Gd = L d

Gw
Lw

b


, Bd = Ld

Bw
Lw

In our implementation, a and L̄w are initially set to 2.0 and
1.0, respectively. When the user selects the ROI, the system recalculates the log average luminance L̄w to update the
RGB values. The user can also change a by directly manipulating the slider bar to control the brightness distribution in
the image. We use a fixed value of 0.6 for b. The LDR color
values used can easily be converted to HDR values Rw , Gw ,and Bw by solving Equation (17), and the system stores the
HDR representation internally.

5

High-dynamic Range Support

Dragon23,960
400
2,166
56
26

(13)

where J = [α β γ]T , and dIv1 is the target differential scalarof the radiance at v1 (Iv0 − Iv1 ). The system continuously
solves this equation in the background process, and updates
J using a small gradient step εJ = 0.05.

Rd = Ld

Bunny
17,483
292
1,575
61
27

Iv’0

Figure 11. Rotation of light source using a
dragging operation. The system computes R
so that the resulting radiance at v1 is identicalto the previous radiance at v0 .

4.5

Buddha23,884
398
2,136
56
26

b
.

(17)

ResultFor all of the glossy objects illustrated, we used bothmeasured and synthetic BRDFs. The reflectance properties
are unrelated to the numerical performance or stability inthe computation. However, more user input is needed whenan object’s reflectance contains high frequency component.
Table 1 shows the time and rendering performance of the
proposed approach. It takes a few minutes for precomputation, a few seconds to change the viewpoint, and less than
40 msec for the lighting change. The system estimates theillumination within a few seconds (e.g., 1-5 seconds for 162
L-SRBFs). Optimization speed depends not on the size ofthe 3D model but on the numbers of L-SRBFs and iterationsof the Active Set method.
3D Object Insertion The system is also useful for adjusting photometric consistency when inserting a synthetic 3D
object into a photograph. For example, users can load anddisplay a background image behind the 3D object, and thenpaint and adjust the lighting effects of the 3D object so thatit seamlessly matches the lighting conditions of the background photograph. Figure 12 and the accompanying video
show the results of inserting a synthetic 3D object into a
photograph using the system. Each 3D object has a measured glossy BRDF, and shadows under the 3D object are
inserted using Adobe Photoshop.
Editing imported environment map The more reflective
an object surface is, the greater is the number strokes thatmust be drawn on the object to design the lighting environment map. This is a fundamental limitation of any inverselighting algorithm. One reasonable solution is to edit an im-

(a) Diffuse:14 / Specular:0(b) Diffuse:12 / Specular:2

Figure 12. Adjusting the photometric effects of a superimposed 3D model with respect to the background image. The two images shown in (a) are the result of inserting a 3D bunny with pink plastic
BRDF. The left image has an arbitrary illumination and the image on the right is rendered using animage-based lighting environment designed by our system. (b) The result of inserting a 3D buddawith white glossy material.
in a segment change together preserving the original image
structure as shown in Figure 13(c) and (d). We used 2562
L-SRBFs. The mask image had four colors that created fourlabeled groups of L-SRBFs.
(a) Original(b) Mask Image(c) Modified #1(d) Modified #2

Figure 13. Editing an existing environment
map. (a) Original lighting environment andrendered scene. (b) Mask Image. (c-d) Modified versions. Painted colors are also shown.
ported environment map. This allows users to input a sparsecollection of strokes and still retain the spatial structure of
the original environment.
The proposed system can also be extended for this purpose as shown in Figure 13. This extension takes two images as input: an original environment map and a mask image that labels pixels as groups. During the design process,the system updates the lighting environment while retaining the visual structure of the original environment maps.
The design process is as follows. First, the original environment map is segmented using image manipulation tools(e.g., Photoshop) to create a mask image as shown in Figure 13(b). Second, the environment map and the mask image are loaded into the system. The system fits L-SRBFs
to the environment map and creates labeled groups of LSRBFs based on the mask image. Finally, the user designs
L-SRBF lights using the system tools. During lighting estimation, the system allows only one scaling factor to be
updated for each group. In this way, the pixel intensities

User Test To validate the effectiveness of the proposedsystem, we performed a user test to compare its performance to a combination of Adobe Photoshop and a HDRShop plug-in (the most popular combination for editing anenvironment map today). Eight computer science students,
all novice users of our system and 3D graphics interfaces ingeneral, participated in the study. Each subject was givena target image (Figure 15 a) and a 3D bunny model undera default lighting condition (Figure 15 b). Each scene has
the same measured BRDF, aluminum-bronze. After a brief
tutorial, the subjects were asked to adjust the lighting environment around the bunny model to that of the target image,using either our system or the alternative software (Photoshop+HDRShop). For both design tools, subjects are allowed to pick a color from the target image and use it for illumination design. The testing order of the software was alternated between subjects; four used Photoshop+HDRShop
first, and the other four used our prototype system first. The
subjects were allowed to work on the task until satisfied, forup to 20 min. Most of the subjects spent approximately 70
minutes for the study including tutorial. Figure 15 showssome of the resulting images.
Figure 14 (a) illustrates the time of design process acrosseight subjects and two design tools. Subjects using Illumination Brush overall took less than 75% of the time that they
did when using Photoshop+HDRShop. Figure 14 (b) shows
the subjective evaluation of the produced images. Sixteen
people voted on the rendered images, regarding their similarity to the image rendered with the ground truth environment map (Figure 15 c). The numbers represent the votes
for each system. A binomial test of the scores shows that
our system performed significantly better than the Photo-Photoshop+HDRShop

Illumination Brush

15
10
5
0

Photoshop+HDRShop

Number of Votes

Time (minutes)

20#1 #2 #3 #4 #5 #6 #7 #8 average

Illumination Brush

20
4

15

9
13

10

12

14

13

14

14

2

3

2

2

11.6

12

5
0

7
3

4

4.4

#1 #2 #3 #4 #5 #6 #7 #8 average

Subject

Subject(a)(b)

Figure 14. Subjective evaluation of the designed scenes. Each of eight users designed
two environment lightings, one using Illumination Brush and the other using Photoshop+HDRShop, so that the rendered images
were similar to the target image (shown in
Figure 15 a). (a) illustrates the time of design process across eight subjects and two
design tools. (b) Then, 16 people voted on
the results, regarding how similar each rendered image matched the bunny image rendered using the ground truth lighting condition (shown in Figure 15 c). The numbers represent the votes for each system.
shop+HDRShop system in editing a lighting environment(p < .001). These results show that our system achieves
better quality in less time than the combination of Adobe
Photoshop and a HDRShop plug-in.

6

Limitations and Future work

The system approximates each transfer function using TSRBFs. The ability to render strong specularity depends on
the number of T-SRBFs. We tested the system using 642
T-SRBFs that can represent not specular but glossy BRDFs.
If the number of T-SRBFs is increased, more specular material can be rendered, but it will result in a slower rendering
speed and more memory usage.
Our future work also includes the evaluation/analysis of
the relationship between reflectance property and illumination estimation algorithm. For the weighting factors described in 4.2, since it is difficult to obtain the best parameter analytically, we have used parameters and formulations
that are empirically obtained. In the future, we want to findthe best weighting factors with respect to reflectance property.
It would be interesting to extend our system to separate
diffuse lighting and specular lighting. We could achieve thisby incorporating two environment maps, one for each typeof lighting. (A diffuse environment map can be represented
more compactly by SH than L-SRBFs.) This could be useful for computer graphic designers, who often perform similar separations by rendering a scene into several components, which can be modified one by one and subsequently
composed. The extension of our system may help them withthe lighting design.
We are interested in appearance-based design tool not
only for illumination but also for material, i.e., BRDF, BSSRDF, and spatially variant BRDF, etc. Our future research
includes supporting BRDF editing in the unified framework.
Because recent works have designed methods for converting image-based lighting environments into a manageable number of light points [10], our system is useful for designing illumination for scenes that must be rendered withlimited computation power, such as 3D games. Our future
research includes such an application.

7

Acknowledgements

This work is supported in part by JSPS Research Fellowship and Microsoft Institute for Japanese Academic Research Collaboration.

References[1] K. Anjyo and K. Hiramitsu. Stylized highlights for cartoon rendering and animation. IEEE Comput. Graph. Appl.,
23(4):54–61, 2003.
[2] F. Anrys and P. Dutré. Image based lighting design. In
The 4th IASTED International Conference on Visualization,
Imaging, and Image Processing, 2004.
[3] H. Biermann, I. Martin, F. Bernardini, and D. Zorin. Cutand-paste editing of multiresolution surfaces. In SIGGRAPH’02, pages 312–321. ACM Press., 2002.
[4] T. A. Davis. A column pre-ordering strategy for theunsymmetric-pattern multifrontal method. ACM Trans.
Math. Softw., 30(2):165–195, 2004.
[5] P. Debevec. Rendering synthetic objects into real scenes:
bridging traditional and image-based graphics with global
illumination and high dynamic range photography. In SIGGRAPH, pages 189–198, 1998.
[6] P. E. Debevec and J. Malik. Recovering high dynamic range
radiance maps from photographs. In SIGGRAPH, pages
369–378, 1997.
[7] R. Fletcher. Practical Methods of Optimization. John Wiley& Sons, 2nd edition, 1987.
[8] N. Goodnight, R. Wang, C. Woolley, and G. Humphreys. Interactive time-dependent tone mapping using programmablegraphics hardware. In Proc. of the 14th Eurographics workshop on Rendering, pages 26–37, 2003.
[9] S. Gumhold. Maximum entropy light source placement. In
VIS ’02: Proceedings of the conference on Visualization ’02,
pages 275–282, Washington, DC, USA, 2002. IEEE Computer Society.
[10] V. Havran, M. Smyk, G. Krawczyk, K. Myszkowski, and
H.-P. Seidel. Interactive system for dynamic scene lighting
using captured video environment maps. In Eurographics
Symposium on Rendering 2005, pages 31–42,311, 2005.
[11] T. Igarashi and J. F. Hughes. Clothing manipulation. In UIST
’02, pages 91–100. ACM Press., 2002.
[12] T. Jung, M. D. Gross, and E. Y.-L. Do. Light pen – sketchinglight in 3d. In Proc. of CAAD Futures, pages 327–338, 2003.

(a) Target(d) Our system #2(b) Default(e) Our system #5(c) Answer(f) Our system #7(g) Our system #8(h) Photoshop+HDRShop #1 (i) Photoshop+HDRShop #2 (j) Photoshop+HDRShop #3 (k) Photoshop+HDRShop #4
Figure 15. Results of the user test comparing the proposed system to Photoshop+HDRShop. The
users were presented a target image (a) and asked to modify the bunny scene (b) to exhibit a similar
illumination environment. The middle row (d-g) presents the resulting environment maps obtainedby different users of the proposed system. The bottom row (h-k) shows the results using Photoshop+HDRShop. Note that users were not shown the ground-truth environment map. The groundtruth environment map and the rendered scene are given in (c).
[13] J. T. Kajiya. The rendering equation. In SIGGRAPH, pages
143–150, New York, NY, USA, 1986. ACM Press.
[14] J. K. Kawai, J. S. Painter, and M. F. Cohen. Radioptimization: goal based rendering. In SIGGRAPH, pages 147–154,
1993.
[15] Y. Li, S. Lin, H. Lu, and H.-Y. Shum. Multiple-cue illumination estimation in textured scenes. In Int’l Conf. on
Computer Vision, pages 1366–1373, Nice, France, 2003.
[16] F. J. Narcowich and J. D. Ward. Nonstationary wavelets on
the m-sphere for scattered data. Applied and Computational
Harmonic Analysis, 3(4):324–336, 1996.
[17] K. Nishino and S. K. Nayar. Eyes for relighting. ACM Trans.
Graph. (SIGGRAPH), 23(3):704–711, 2004.
[18] F. Pellacini, F. Battaglia, R. K. Morley, and A. Finkelstein.
Lighting with paint. ACM Trans. Graph., 26(2):9, 2007.
[19] F. Pellacini, P. Tole, and D. P. Greenberg. A user interface forinteractive cinematic shadow design. In SIGGRAPH, pages
563–566, 2002.
[20] P. Poulin, K. Ratib, and M. Jacques. Sketching shadowsand highlights to position lights. In Proc. of Conference on
Computer Graphics International, pages 56–63, 1997.
[21] R. Ramamoorthi and P. Hanrahan. A signal-processing
framework for inverse rendering. In SIGGRAPH, pages
117–128, 2001.
[22] E. Reinhard, M. Stark, P. Shirley, and J. Ferwerda. Photographic tone reproduction for digital images. ACM Trans.
Graph. (SIGGRAPH), 21(3):267–276, 2002.

[23] I. Sato, Y. Sato, and K. Ikeuchi. Illumination from shadows. IEEE Trans. Pattern Anal. Mach. Intell., 25(3):290–
300, 2003.
[24] C. Schoeneman, J. Dorsey, B. Smits, J. Arvo, and D. Greenburg. Painting with light. In SIGGRAPH, pages 143–146,
1993.
[25] R. Shacked and D. Lischinski. Automatic lighting design using a perceptual quality metric. Computer Graphics Forum,20(3):1067–7055, 2001.
[26] A. Shesh and B. Chen. Crayon lighting: Sketch-based illumination of models. Proceedings of Pacific Graphics 2006,
pages 113–116, 2006.
[27] P.-P. Sloan, J. Hall, J. Hart, and J. Snyder. Clustered principal components for precomputed radiance transfer. SIGGRAPH, 22(3):382–391, 2003.
[28] P.-P. Sloan, J. Kautz, and J. Snyder. Precomputed radiancetransfer for real-time rendering in dynamic, low-frequency
lighting environments. In SIGGRAPH, pages 527–536,
2002.
[29] Y.-T. Tsai and Z.-C. Shih.
All-frequency precomputed radiance transfer using spherical radial basis functions and clustered tensor approximation. ACM Trans.
Graph.(SIGGRAPH), 25(3):967–976, 2006.
[30] Y. Wang and D. Samaras. Estimation of multiple directional light sources for synthesis of augmented reality images. Graph. Models, 65(4):185–205, 2003.