Eclipse: Disambiguating Illumination and Materials using Unintended Shadows
Supplementary Material
S1. Additional results
Figure S1 contains additional synthetic results, similar to
Figures 1 and 4 of the main paper, and Figure S2 contains
additional real results from the Stanford-ORB dataset [18],similar to Figure 8 from the main paper. As explainedin the paper, the recovered illumination estimated by ourmodel corresponds to the incident light at the object, which
is typically extremely warped relative to the collection ofenvironment maps provided by the dataset as ground truth,
since those are obtained using a light probe placed at different points in the scene. We therefore compute our quality metrics relative to the mean environment map obtained
from these differently-placed probes, blurred using a (normalized) spherical Gaussian kernel p(θ, ϕ) ∝ exp(κ cos θ)(we set κ = 100) to fuse the misaligned light sources. See
Figure S10 for an example of this process.
Figure S3 provides additional evidence that the signal
provided by unintended shadows improves the recovery ofenvironment illumination, expanding on Figure 9 in themain paper.
Figure S4 contains an additional result for recovered illumination and material properties when the object geometry
is not given, as in Figure 7 of our main paper.

truth ones, the masks appear to be less plausible. This canbe explained by the difficulty of recovering mask values inregions occluding dark regions of the environment. This istrue almost everywhere in the case of highly-directional illumination, like the environment map shown in Figure 8.
See Figure 5 in the main paper and the text in Section 5 fora demonstration of this effect in synthetic data.

S2. BRDF model
The experiments in the paper and supplement were performed using either a standard Lambertian BRDF (where
specified) or a BRDF based on Unreal Engine’s version of
GGX [15, 43]:
  &\!\!\!\!\!\!\! f(\bx , \lightdir , \viewdir ) = \nonumber \\ & \frac {1}{\pi }\rho (\bx ) (1-F(\uvn \cdot \lightdir ; \fzero ))(1-F(\halfvec \cdot \lightdir , \fzero ))(\uvn \cdot \lightdir )_+ \nonumber \\ +& \frac {D(\uvn \cdot \halfvec ; \alpha )F(\halfvec \cdot \lightdir ; \fzero )G(\uvn \cdot \viewdir , \uvn \cdot \lightdir ; \alpha )}{4(\uvn \cdot \lightdir )_+(\uvn \cdot \viewdir )_+}\,,(S1)
where:
  F(\cos \theta ; \fzero ) &= \fzero + (1 - \fzero ) (1-\cos \theta )^5\,, \\ D(\cos \theta ; \alpha ) &= \frac {\alpha ^2}{\pi (1 + (\alpha ^2-1)\cos ^2\theta )^2}\,, \label {eq:trowbridge} \\ G(\cos \theta _i, \cos \theta _o; \alpha ) &= g(\cos \theta _i; \alpha )g(\cos \theta _o; \alpha )\,, \\ g(\cos \theta ; \alpha ) &= \frac {\cos \theta }{k(\alpha )+(1-k(\alpha ))\cos \theta }\,, \\ k(\alpha ) &= \frac {(\alpha +1)^2}{8}\,, \\ \halfvec &= \frac {\lightdir + \viewdir }{\|\lightdir + \viewdir \|}\, \label {eq:halfvec}.

Relighting. Figure S5 shows that our recovered albedo
enables convincing relighting results. See our supplemental
video for rendered video results.
Non-diffuse results. Figure S6 shows the results obtainedby our method when applied to the same microfacet BRDF
model from our main paper, but with a roughness value of
0.2 (see Section S2).
Recovered masks. The main goal of our approach is therecovery of illumination and materials. However, it may
also be informative to visualize the recovered per-image occluder masks. Figure 5 in the main paper shows the recovered masks in the case of letter-shaped occluders, andwe show additional recovered masks from the same experiment in Figure S7. Similarly, Figure S8 shows the recovered masks from another experiment done on the potatoes scene, where the masks are the spherical caps withsolid angle 0.1 · 4π steradians used in all other experiments(see description in Section 5 of the main paper). Finally,
Figure S9 shows masks recovered from the Stanford-ORB
dataset [18], corresponding to the recovered illumination
shown in Figure 8 of the main paper. Note that although therecovered lights and albedos in Figure 8 match the ground(S7)
We omit the positional dependence of the surface normal
vector n̂ in the point x on the surface.
The 5 parameters describing the BRDF at a given location x are therefore the RGB albedo ρ, the microfacet
roughness α, and the specular reflectance at normal incidence κ which determines the strength of the Fresnel factor
F.

S3. Full Description of Renderer
We describe our renderer’s approximation of the integral in
Equation 2 of the main paper. Our renderer follows wellestablished practices that are readily documented in public
textbooks such as PBRT [32].
In order to render a specific pixel, we begin by choosing a random ray from the camera center through the squarePSNR = 32.3 dB

RMSE = 0.051

PSNR = 32.2 dB(a) Sample input image (b) Recovered albedo

RMSE = 0.118(c) True albedo(d) Recovered illumination(e) True illumination

Figure S1. Additional results on diffuse objects, similar to Figures 1 and 4 of the main paper. We report the RMSE of each environment
map in linear color space but plot the images after tonemapping for better evaluation of the full dynamic range. The albedo PSNR values
are reported on object pixels only.

(a) Sample input image

RMSE = 0.039(e) Recovered illumination

PSNR = 18.9 dB(b) Recovered albedo

PSNR = 13.5 dB(c) Recovered albedo (no occluder)

RMSE = 0.112(f) Recovered illumination (no occluder)(d) True albedo(g) Warped true illumination

Figure S2. Results on the captured salt007 scene from Stanford-ORB [18], extending Figure 8 of the main paper. Note that the “warped
true illumination” environment map (g) was captured by a light probe that was not co-located with the object, any may therefore be
significantly warped. The recovered RMSE reported is computed with respect to the average of all environment maps provided with thedataset rotated the same coordinate frame, see text and Figure S10.

footprint of the pixel. The intersection point of the ray withthe object’s mesh is then computed, as well as the normal
vector at that point (interpolated from per-vertex normals).
Then, we use multiply importance sampling, and randomly
sample sℓ + sm = 1024 rays from this intersection point,with sℓ = 512 sampled from the lighting distribution, andsm = 512 sampled from the material distribution. The
lighting distribution is simply a piecewise-constant distri-bution proportional to the average environment map values
taken across the three channels, weighted by the Jacobianof the parameterization, sin θ:  \label {eq:pl} p^{(\ell )}(\bomega ) \propto \sum _{c\in \{R, G, B\}}\tilde {L}_{c}(\bomega )\sin \theta , (S8)

where L̃c is the c-th channel of the environment map obtained using nearest-neighbor interpolation, and θ is the po-

With Occluders
Without Occluders
Ground Truth
Figure S3. The signal provided by unintended shadows cast by unseen occluders improves the quality of recovered environment illumination. This is an expansion of Figure 8 in the main paper, but for an object with purely Lambertian material (we use the potatoes scene
for these experiments). Here, we plot the relative MSE for the recovered environment maps under two scenarios: (blue) using images
rendered with unobserved occluders and jointly estimating materials, illumination and occluder shape; and (orange) using images rendered
without occluders and only estimating materials and illumination (this is similar to the problem setting considered by Swedish et al. [37],but optimized using our method). The cue of unintended shadows consistently improves the quality of estimated illumination across varying (first column) number of observed images, (second column) illumination frequency content, (third column) number of secondary rays
traced, (fewer secondary rays causes increased Monte Carlo noise), and (fourth column) additive Gaussian noise. We display recovered
environment maps corresponding to two points on each plot.

plar angle corresponding to ω̂, i.e. sin θ = 1 − (ω̂ · ẑ)2 ,with ẑ denoting the unit vector in the z direction.
The material distribution is simply the Trowbridge-Reitz
normal distribution function reparameterized for samplingincoming light directions:  \label {eq:pm} p^{(m)}(\bomega ) = D(\uvn \cdot \halfvec ; \alpha )\frac {|\halfvec \cdot \hat {\mathbf {z}}|}{4(\halfvec \cdot \viewdir )_+}\,, (S9)

where ω̂ h is the half-vector defined in Equation S7, and D is
the Trowbridge-Reitz distribution with roughness parameterα, as defined in Equation S3.
The two sets of samples are combined using the power

heuristic [40]:  \label {eq:importancesampling} C(\bx , \bomega ^{(j)}, \viewdir ) = & L_t\!\left (\bomega ^{(j)}\right )f_{\bx }\!\left (\bomega ^{(j)}, \viewdir \right )\!\left (\uvn (\bx )\cdot \bomega ^{(j)}\right )_+ \nonumber \\ \tilde {I}_t(\bu ) = \frac {1}{s_m}\sum _{j=1}^{s_m} & \frac {{\beta _m}\left (\bomega ^{(j)}\right ) C(\bx , \bomega ^{(j)}, \viewdir ) }{p^{(m)}\left (\bomega ^{(j)}\right )} \nonumber \\ +\frac {1}{s_\ell }\sum _{j=s_m\!+\!1}^{s_m\!+\!s_\ell } & \frac { {\beta _\ell }\left (\bomega ^{(j)}\right ) C(\bx , \bomega ^{(j)}, \viewdir )}{p^{(\ell )}\left (\bomega ^{(j)}\right )}\,, \\ & \text {with } \bomega ^{(1)}, \ldots , \bomega ^{(s_m)} \sim p^{(m)}(\bomega )\,, \nonumber \\ & \text {and } \bomega ^{(s_m + 1)}, \ldots , \bomega ^{(s_m + s_\ell )} \sim p^{(\ell )}(\bomega )\nonumber \,,(a) Sample input image(b) Recovered albedo(c) True albedo(e) Recovered illumination(g) True illumination

Figure S4. Expansion of Figure 7 in the main paper. We are able to recover illumination even when geometry is unknown by first optimizing
a volumetric representation of geometry using a NeRF-based method. Despite this inaccurate proxy geometry, our method still recovers
plausible (albeit blurry) illumination (e) and albedo (b).

Original illumination, α = 0.8

Relit, α = 0.6

Relit, α = 0.4

Relit, α = 0.2

Figure S5. Our method enables modifying the illumination and material properties of the object. On the left, we show the originalillumination and rough material (α = 0.8 in our BRDF model). The three columns to the right feature our recovered albedo, but rendered
under different environment maps and progressively lower roughness values.

where βm and βℓ are the power heuristic weights for multiple importance sampling, with exponent 2, as in [40]:  \beta _m(\bomega ) &= \frac {\left (s_m p^{(m)}(\bomega )\right )^2}{\left (s_m p^{(m)}(\bomega )\right )^2 + \left (s_\ell p^{(\ell )}(\bomega )\right )^2}, \\ \beta _\ell (\bomega ) &= \frac {\left (s_\ell p^{(\ell )}(\bomega )\right )^2}{\left (s_m p^{(m)}(\bomega )\right )^2 + \left (s_\ell p^{(\ell )}(\bomega )\right )^2}.

rial distributions, we use inverse transform sampling, where
the n input pairs of samples (u0 , v0 ), . . . , (un−1 , vn−1 ) ∈[0, 1]2 are computed from 2n i.i.d. uniform random variables r0 , . . . , rn−1 , t0 , . . . tn−1 ∼ Uniform[0, 1] using:  u_i = \frac {\text {mod}(i, s) + r_i}{s}\\ v_i = 2\cdot \frac {\lfloor i / s\rfloor + t_i}{s}\,,(S12)
Similar to the observation of Zeltner et al. [45], our experiments show that it is beneficial to “detach” gradients fromthe sampling procedure for ω̂ (j) as well as from the PDFs
p(m) and p(ℓ) in Equations S8 and S9.
In order to generate samples from the lighting and mate-(S14)
where:
  s = 2^{\left \lfloor \frac {\log _2(n)+1}{2} \right \rfloor }. 

(S15)This procedure divides the unit square into an 2s × s grid,Illumination

Albedo ρ

Roughness αFresnel κ

Ground truth

Recovered

Sample image

Figure S6. Our results on a shinier object. Our method is designed for diffuse objects, but it still manages to extract accurate material
properties and illumination when materials are more specular (α = 0.2 for this figure). For visualization purposes the roughness values α
are linearly mapped from [0.15, 0.25] to [0, 1], and the specular reflectance at normal incidence κ are mapped from [0.03, 0.05] to [0, 1](with the true value being 0.04), in order to show small errors.

Figure S7. Expansion of Figure 5 in the main paper, showing a random selection of recovered masks (top image in each pair), and thecorresponding ground truth (bottom image).

and uniformly samples a single pair (ui , vi ) in each one of
the grid cells, resulting in a stratified sampling pattern.
For generating the data, we repeat the entire rendering
process described above 16 times using different primaryand secondary rays, for every pixel and average the resultsin order to antialias the results. Note that this corresponds
to using a box reconstruction filter, which may have visible
artifacts, yet we decided to use it in our experiments for
simplicity.

Figure S8. Masks recovered by our method on the potatoes scene. The top image in each pair shows the recovered masks, and the bottom
one shows its corresponding true mask.

(a) Sample environment maps from the same captured scene

Figure S9. Masks recovered by our method on the salt004
scene from Stanford-ORB [18]. Note that the masks are not guaranteed to be plausible, since regions which occlude small amountsof light cannot be recovered.

S4. Theoretical Considerations
S4.1. Figure 3 from Main Paper
Figure 3 from the main paper is based on a simplified “flatland” version of our problem, where the occluders and materials are known. A circular object with radius 1 and Lambertian BRDF with albedo 1 is placed at the origin, and illuminated by a (monochromatic) environment map L(ϕ). An
occluder of some angular width ∆θ and central angle θ is(b) Mean environment map(c) Blurred mean environment map

Figure S10. Our blurring process for environment maps provided
with the Stanford-ORB dataset [18]. The environment maps provided with the dataset are not aligned since they were captured bya light probe not co-located with the object, as shown by the two
misaligned images in panel (a), which makes them unusable forcomputing error metrics for illumination recovery. We therefore
first: (b) average all environment maps provided along with eachscene, and then (c) blur them using a normalized spherical Gaussian.

then placed at some radius r from the origin, blocking all
shadow rays from the object to the illumination.
We parameterize the surface of the object using a single

angle parameter λ ∈ [0, 2π), i.e. the “surface” of the disk
is the set of points S = {(cos λ, sin λ) : λ ∈ [0, 2π)}. The“image” at the point λ at time t is then:  \label {eq:flatrender} I_t(\lambda ) = \int _{0}^{2\pi } L(\phi ) M_t(\lambda , \phi ; \theta _t, \Delta \theta _t)(\cos (\lambda -\phi ))_+ \, d\phi \,, (S16)
with the “flatland” (unnormalized) BRDF set to 1, and theoccluder value at time t is:
  &M_t(\lambda , \phi ; \theta _t, \Delta \theta _t) = \mathbbm {1}\!\left [\left |\psi (\lambda , \phi ) - \theta _t \right | < \frac {\Delta \theta _t}{2}\right ],

(S17)

with:
  &\psi (\lambda , \phi ) = \arctan \!\left (\frac {\sin \lambda + t(\lambda - \phi )\sin \phi }{\cos \lambda + t(\lambda - \phi )\cos \phi }\right ), \\ &t(\delta ) = \sqrt {\cos ^2(\delta )+r^2-1} - \cos (\delta ). \nonumber

We then discretize the integral in Equation S16 for everyt = 1, . . . , T , and stack the resulting equations:  \label {eq:matrixell} \underbrace {\begin {bmatrix}\mathbf {I}_1\\ \vdots \\\mathbf {I}_T\end {bmatrix}}_{\mathbf {I}} = \underbrace {\begin {bmatrix}A_1\\ \vdots \\A_T\end {bmatrix}}_{A}\boldsymbol {\ell }, (S19)

where It ∈ RN is a vector of rendered values {It (λ)} attime t, ℓ ∈ RM is a vector of illumination values {L(ϕ)},
and At ∈ RN ×M contains the corresponding values of theoccluders M (λ, ϕ; θt , ∆θt ) at time t and the cosine lobe(cos(λ − ϕ))+ .
Figure 3 then shows the singular values of A (which are
the square roots of the eigenvalues of A⊤ A), normalized tohave maximum value 1, for a few scenarios:• “No occlusions”: The scenario described above with asingle observation (T = 1) but without an occluder, i.e.
where the mask is set to 1 everywhere (or, alternatively,∆θ = 2π).
• “1 observation”: The scenario described above, with asingle occluder (T = 1) placed at θ = 0.
• “T observations”, for T ∈ {2, 8, 16, 32}: A contains
T stacked matrices corresponding to a discretization of
Equation S16, each one for a different occluder location
θt uniformly spread over [0, 2π), and the same ∆θ.
The exact values used for Figure 3 are occluder width∆θ = 0.7 rads, placed at distance r = 10 from the origin,with 512 samples per observation.

S4.2. Spherical Harmonics Intuition
In the main paper we show that expressing the occluder
masks in the basis of spherical harmonics, instead of thestandard basis, leads to significantly improved results. The
reasoning behind that can be illustrated in 1D using theFourier basis (sines and cosines), since spherical harmonics are the spherical equivalent of the Fourier basis.
Similar to Equation S19, the integral in Equation S16 canalso be discretized and combined into a linear equation in
the occluder values. However, the resulting matrix is blockdiagonal, with each Mt only depending on It at the sametime t:  \label {eq:matrixem} \mathbf {I}_t = B\mathbf {m}_t, (S20)
where It is again a vector of image values at time t, mt
is a vector of occluder values at the same time, and B describes their linear relation according to a discretization of
Equation S16. B is composed of a circulant matrix (corresponding to the BRDF’s convolution), multiplied by a diagonal matrix with the illumination values L along its diagonal. This means that when the illumination is uniform, B is
also a circulant matrix, and is therefore diagonalized by the
Discrete Fourier Transform (DFT) matrix. This makes the
problem diagonal in the Fourier basis, which makes adaptive optimizers such as Adam [16] especially effective.
When the illumination is non-uniform, B is not generally diagonalized by the DFT matrix. However, for naturallighting, when the DFT matrix F is applied to B, most of
the matrix’s energy is along the diagonal elements, i.e. theelements of the ith row of F BF ⊤ are maximized by the ithelement. See Figure S11 for examples showing rows of the
F BF ⊤ matrix for different illumination spectra. The factthat the Fourier basis (or in the original problem, the basisof spherical harmonics) “nearly-diagonalizes” the problem
is the reason for its effectiveness.
Furthermore, since we are not interested in recovering
the occluders in regions with low illumination (see right of
Figure 5 of the main paper), it is informative to consider the
problem of estimating the masked illuminant:  \label {eq:matrixellem} \mathbf {I}_t = C(\boldsymbol {\ell }\circ \mathbf {m}_t), 

(S21)

where ◦ denotes elementwise multiplication, and C is a circulant matrix corresponding to the (discretized) convolution
in Equation S16. The problem of estimating ℓ ◦ mt given
It is in fact diagonalized by the Fourier basis, because C iscirculant.

S4.3. Image Pyramid Intuition
The set of linear equations satisfied by the illumination ℓdescribed in Equation S19 cannot be separately solved foreach t, and therefore we must consider the entire systemof equations described by the matrix A ∈ RN T ×M . Figure S12 shows the entries of A⊤ A, for different choices ofT . The entries of the matrix exhibit a block structure in
the standard basis, i.e. without applying the DFT matrix toit, with the sizes of the blocks becoming smaller for larger
values of T . In fact, for very large values of T the matrix
approaches a circulant matrix, which is diagonalized by the

a=1

a = 0.1

Light spectrum

1.0

1.0

0.5

0.5

0.5

0.0

0 32 64 96127

0.0

0 32 64 96127

0.0

1.0

1.0

1.0

0.5

0.5

0.5

0.0
a = 10Row 321.0

0 32 64 96127

0.0

0 32 64 96127

0.0

1.0

1.0

1.0

0.5

0.5

0.5

0.0

0 32 64 96127

0.0

0 32 64 96127

0.0

Row 64S5. Geometry Estimation Method

0 32 64 96127

We estimate unknown object geometry from input images
by recovering a Neural Radiance Field (NeRF) [25] of theobject, based on the Instant Neural Graphics Primitives [26]representation. To improve the recovered normal vectors(computed as the negative normalized gradient of the volume density field), we use the normals orientation regularization and MLP-predicted normals technique from RefNeRF [41].

0 32 64 96127

0 32 64 96127

Figure S11. Most of the energy of the matrix B is concentrated around the diagonal elements in the Fourier basis. Eachrow shows, from left to right: the incident illumination spectrum(which modifies B), and the magnitude of the elements of the 32nd
and 64th rows of F BT ⊤ , where F is the DFT matrix. Each row
corresponds to a 1/f a spectrum for a = 0.1, 1, 10. See text for
more details.

T =1

T =2

T =4

T =8

T = 16

T = 32

S6. Data Specification
All BRDF parameters (RGB albedo, roughness, and thespecular reflectance at normal incidence), are output by acoordinate-based MLP taking in a positionally-encoded location:
  \gamma (\bx ) = \big (&\sin (\bx ),\cos (\bx ), \sin (2\bx ), \cos (2\bx ), \nonumber \\ &\ldots , \sin (64\bx ), \cos (64\bx )\big ).
(S22)
The MLP has 4 layers with 128 hidden units each, with
ReLU nonlinearities. The weights are initialized aroundzero, and the output BRDF parameters are obtained by mapping the MLP’s output through a sigmoid function, meaningthat they are all initialized around a value of 0.5.
The spherical harmonic coefficients {atℓm } from Equation 6 of the main paper are all set to zero, but a positive
bias of 100 is added to the pre-sigmoid value (which canequivalently be done by initializing at00 to a constant). We
find that initializing the masks to be close to 1 everywhere
improves our method’s performance and prevents it fromgetting stuck in a local minimum.
The environment map pyramid levels are also all initialized to zero, such that the illuminant is set to all-ones, due
to the exponential nonlinearity (see Equation 7 in the main
paper).

S7. Additional Data Details

Figure S12. The structure of A⊤ A for the light transport matrix
A, plotted for different values of the number of observations T .
Large values in red, small values in blue.

DFT matrix. However, we wish our method to not make useof that fact and work even for a sparse set of observations,and therefore a natural choice for parameterizing the illumination ℓ is by using an image pyramid, which matches theblock structure of A⊤ A for general T values.

The geometry of the objects in the paper and their textures
originated in the following BlendSwap models:
1. Potatoes: created by mik1190, CC0 license (model#15725)
2. Chair: created by 1DInc, CC0 license (model #8261).
3. Mannequin: created by salimrached, CC0 license(model #27747).
4. Toad: created by arenyart, CC0 license (model #13078).
5. Plant: created by New Enemy, CC0 license (model#30071).
6. Giraffe: created by amx360, CC-BY license (model#29651).
The environment maps are from the following Poly
Haven assets:

1. “Canary Wharf”: created by Andreas Mischok, CC0 license.
2. “Abandoned Factory Canteen 01”: created by Sergej
Majboroda, CC0 license.
3. “Outdoor Umbrellas”: created by Sergej Majboroda,
CC0 license.
4. “Thatch Chapel”: created by Dimitrios Savva, Jarod
Guest, CC0 license.
5. “Evening Road 01 (Pure Sky)”: created by Jarod Guest,
Sergej Majboroda, CC0 license.
6. “Marry hall”: created by Sergej Majboroda, CC0 license.