Eurographics Symposium on Rendering 2015
J. Lehtinen and D. Nowrouzezahrai(Guest Editors)

Volume 34 (2015), Number 4

Path-space Motion Estimation and Decomposition
for Robust Animation Filtering
Henning Zimmer1

Fabrice Rousselle1
1

Wojciech Jarosz

Wenzel Jakob2
2

Olga Sorkine-Hornung

1 Disney Research Zurich

2 ETH Zurich

Oliver Wang1

David Adler3

Alexander Sorkine-Hornung1
3 Walt Disney Animation Studios

Figure 1: Starting from noisy, low-resolution frames generated with a path tracer (red borders), our method improves quality
and reduces computational cost by computing spatially and temporally upsampled and denoised frames (green and blue borders)
while properly preserving view-dependent shading effects like the reflections in the picture frame and on the robot.
Abstract
Renderings of animation sequences with physics-based Monte Carlo light transport simulations are exceedingly
costly to generate frame-by-frame, yet much of this computation is highly redundant due to the strong coherence inspace, time and among samples. A promising approach pursued in prior work entails subsampling the sequence inspace, time, and number of samples, followed by image-based spatio-temporal upsampling and denoising.
These methods can provide significant performance gains, though major issues remain: firstly, in a multiple
scattering simulation, the final pixel color is the composite of many different light transport phenomena, and thisconflicting information causes artifacts in image-based methods. Secondly, motion vectors are needed to establish
correspondence between the pixels in different frames, but it is unclear how to obtain them for most kinds of lightpaths (e.g. an object seen through a curved glass panel).
To reduce these ambiguities, we propose a general decomposition framework, where the final pixel color is separated
into components corresponding to disjoint subsets of the space of light paths. Each component is accompanied bymotion vectors and other auxiliary features such as reflectance and surface normals. The motion vectors of specularpaths are computed using a temporal extension of manifold exploration and the remaining components use a
specialized variant of optical flow. Our experiments show that this decomposition leads to significant improvementsin three image-based applications: denoising, spatial upsampling, and temporal interpolation.
Categories and Subject Descriptors (according to ACM CCS): I.3.7 [Computer Graphics]: Three-Dimensional
Graphics and Realism—I.3.7 [Computer Graphics]: Color, shading, shadowing, and texture—

1

Introduction

The recent widespread adoption of physically based lighttransport techniques in the animation and visual effects industries [KKG∗ 14] has led to tremendous computational
challenges. Given that hundreds of thousands of frames areneeded for a feature-length film with per-frame computation
times in the order of several hours, rendering costs have bec 2015 The Author(s)Computer Graphics Forum c 2015 The Eurographics Association and John
Wiley & Sons Ltd. Published by John Wiley & Sons Ltd.

come a critical bottleneck, which will continue to increaseas home and cinema displays continue to move to steadily
higher resolutions and frame rates.
At the same time, the underlying computation is characterized by a high amount of redundancy due to the coherencein space, time and among samples. Many acceleration techniques build on this property by using a reduced number of

H. Zimmer et al. / Path-space Motion Estimation and Decomposition for Robust Animation Filtering

samples, followed by image-based post-processing to fill in
missing information. Examples include using fewer samples
per pixel and denoising, leaving out some of the frames andinterpolating them temporally, or rendering a subset of pixels and performing spatial upsampling. A key advantage ofimage-based methods is that their computational complexity
scales with the number of pixels and not with the scene andlighting complexity. Consequently, such methods have become a vital part of production rendering pipelines [God14].
Unfortunately, filtering in image space is inherently illposed. Prior works have investigated ways of disambiguating
the problem by exporting auxiliary data from the rendering
process, e.g. motion vectors or depth values. This has been
leveraged for real-time frame interpolation [BMS∗ 12], spatiotemporal upsampling [HEMS10], as well as for denoising
path-traced renderings [SD12]. However, when processing
images produced by physically based light transport simulations, filtering remains ill-posed even with such auxiliary data:
the problem is that the color of a pixel becomes a composite
of various light transport effects, including the reflectanceof objects, shadows, and specular reflections and refractions.
These effects generally follow different trajectories over time,making it impossible to assign a single motion vector per
pixel. Thus, averaging neighboring pixels for denoising or
upsampling inevitably leads to blurring, ghosting and other
undesirable interference.
Contributions. To address these issues, we propose a general decomposition framework, where the final pixel color
is separated into components corresponding to disjoint subsets of path space [Vea98]. Such decompositions have beenused for different applications (Section 2); the key innovationof our approach is that each component is accompanied bymatching motion vectors, which makes it suitable for a range
of useful filtering applications. We also show how to extract
component-specific auxiliary features and propose a novel
way of decomposing reflectance and incident illuminationthat significantly improves on prior work.
Computing motion vectors for higher order light paths
requires a new set of tools: for specular paths, we propose aspace-time extension of Manifold Exploration (ME) [JM12].
For the remaining components of the decomposition, we use
an image-based optical flow method, bootstrapped with the
motion vectors of the visible surface points.
We demonstrate the impact of our decomposition and motion estimation framework in three applications and combinations thereof: denoising, spatial upsampling, and temporalinterpolation. As our framework successfully resolves theambiguity problems discussed above, we obtain significant
improvements in image quality compared to baseline methods using non-decomposed image data. Our method thus
effectively reduces computational costs for producing highquality renderings of animation sequences.

2

Related Work

Decompositions. Several prior works compute sets of feature buffers which capture certain aspects of the underlying
light scattering process, for instance to emphasize specular reflections in the context of non-photorealistic rendering [ST90]. The NVIDIA Iray rendering system [NVI12]
uses a regular expression-based decomposition of path space
to emphasize certain aspects of the light transport for artistic control. Dabała
˛
et al. [DKR∗ 14] decompose images into
many components using Whitted-style ray trees to control
stereoscopic disparity. Our construction is designed for a fundamentally different purpose: to capture and decompose animated multiple scattering effects so that they do not interfere
when frames are further processed using image-based methods. Related to this idea, Lochmann et al. [LRR∗ 14] separate
diffuse reflections from specular reflections and refractions
for novel view generation. However, their image space approach is restricted to a single specular bounce, whereas we
propose a more fine-grained decomposition that considers
arbitrary chains of specular effects.
Specialized decompositions are also used in a variety ofrendering algorithms: a seminal example is the irradiancecaching algorithm of Ward et al. [WRC88], which exploits
the smooth nature of indirect illumination to reconstruct irradiance values from a sparse set of samples. An important observation of the irradiance caching algorithm is that the reconstruction becomes considerably easier once the effects of texture variation are separated from the incident irradiance. Irradiance factorization is commonly used in global illumination
methods, e.g. for interactive rendering [WKB∗ 02, SIMP06]
or factorized axis-aligned filtering [MYRD14]. A fundamental problem of this approach is that it builds upon a linear
shading model, which is invalidated by most realistic shading models [HMD∗ 14], spatial anti-aliasing, and distribution
effects such as depth-of-field or motion blur. We propose ageneralized lighting–texture factorization that does not sufferfrom this problem; we demonstrate its effectiveness on depthof-field and state-of-the-art microfacet reflectance models.
Motion estimation. Rendering engines can typically provide motion vectors for the scene geometry, which for example has been exploited for rendering novel viewpoints[MMB97]. However, these motion vectors do not capture
apparent motion caused by secondary effects like shadowsor reflections. In the context of filtering for anti-aliasing,
Shinya [Shi95] and Igehy [Ige99] proposed techniques that
can handle specular interactions as well as moving shadows.
For general motion estimation, one could also resort to imagebased techniques like optical flow [BBPW04] to estimate theapparent motion in the rendered images, but even state-of-theart methods do not achieve the accuracy needed for highly
accurate interpolation results. In particular, most methods
only generate a single flow vector per pixel, which cannot
model different motions of overlapping secondary effects.
To address the problem of multiple motion vectors per pixel,
c 2015 The Author(s)Computer Graphics Forum c 2015 The Eurographics Association and John Wiley & Sons Ltd.

H. Zimmer et al. / Path-space Motion Estimation and Decomposition for Robust Animation Filtering

Lochmann et al. [LRR∗ 14] compute approximate motion
vectors for diffuse reflections as well as specular reflections
and refractions in image space, which produces significant
artifacts when used for interpolating novel viewpoints, since
the continuous space-time behavior of the scene geometryis lost when operating in image space. In contrast, our motion estimation based on Manifold Exploration supports arbitrary chains of specular interactions and yields pixel-accurate
motion vectors by performing computations at render time,
where the 3D scene data is still available.
Denoising. There has been a strong renewed interest in
denoising Monte Carlo renderings recently and we refer thereader to the survey by Zwicker et al. [ZJL∗ 15] for a complete
overview. Many recent methods resort to generic image space
filters, which have proven to be surprisingly effective [SD12],
especially when performing joint filtering guided by auxiliaryfeatures, e.g. normals and reflectance. While normals capture
geometric detail and reflectance captures the appearance ofmaterials, they miss the aforementioned secondary effects
like shadows and reflections. To address these limitations,additional features have been proposed, such as a virtual flash
image to capture reflections [MJL∗ 13], or a direct visibility buffer to capture direct shadows [RMZ13]. Instead of
introducing additional features, we process each component
separately while using a component-specific set of auxiliaryfeatures. This not only preserves secondary effects, but also
allows denoising to adapt to the noise characteristics of eachcomponent, leading to significant quality improvements.
Spatial upsampling. To avoid the edge blurring of straightforward spatial upsampling techniques like bilinear interpolation, joint upsampling schemes have been proposed thatleverage edge information from a high-resolution guide image [HST13]. In the rendering context, such guide images
are readily obtained from cheap auxiliary features like depth
or normals [HEMS10]. We build on the approach of He etal. [HST13] and show that upsampling each component independently using the respective features as guides improves
results in the presence of complex secondary effects.
Temporal upsampling. There is a large body of workon image interpolation in video or for virtual view generation [CW93, ZKU∗ 04]. More recent work uses motion vectors to drive image warping for upsampling rendered footage
for higher frame rate displays [DER∗ 10]. However, such
image-based methods are prone to visual artifacts due tounavoidable errors in image correspondence estimation andambiguities caused by the typical restriction to a single motion vector or depth value per pixel. Frame interpolation canalso be seen as a special case of the more general concept ofleveraging temporal coherence, which has a long history in
rendering [SYM∗ 11]. In contrast to frame interpolation, thesemethods reuse shading information over time to bootstrap therendering of frames, as opposed to synthesizing them. This
produces approximate results at interactive rates, but mostly
does not meet the high quality standards of offline rendering.
c 2015 The Author(s)Computer Graphics Forum c 2015 The Eurographics Association and John Wiley & Sons Ltd.

3

Decomposition

The goal of our decomposition is the separation of several
different scattering effects like chains of specular reflections
and refractions so that they do not interfere in image-based
post-processing methods (denoising, spatial upsampling, andframe interpolation; see Section 5). To this end, we decompose the rendered image into disjoint path space components
using a tree of regular expressions, where each leaf nodecorresponds to an image buffer (Figure 2 and 3).
We use a decomposition based on light path analysis following Veach’s [Vea98] path-space formalism, which models
light transport using integrals over light paths. Each path isa piecewise linear trajectory between the camera and a lightsource, where each vertex represents an intermediate scattering interaction. In order to define our decomposition, werely on Heckbert’s [Hec90] regular expression notation toclassify light paths or families of similar light paths. The firstvertex on the camera sensor is labeled E, and each subsequent
scattering vertex encodes the underlying material: diffuse(D), specular or glossy reflection (R), and specular or glossy
transmission (T). We classify glossy light interactions (e.g.
scattering off rough metal or glass) as R or T if the roughness
is below a threshold (Beckmann roughness α < 0.1 in our
experiments), otherwise we consider them to be diffuse. Families of similar light transport paths can be expressed using a
regular expression syntax, as shown in the legend of Figure 3
and detailed in the supplementary material.
Many different decompositions are theoretically feasible
and it is important to find the right trade-off: decompositions
with few components may not provide much benefit, whereas
very detailed path classifications provide a rich source of information, but the storage and computational overhead eventually becomes prohibitive due to the exponential growth ofclasses with increasing path length. Our decomposition is
flexible in this regard and can be customized to focus on thesalient light transport effects in specific scenes. We found thedecomposition shown in Figures 2 and 3 to be adequate formost cases, but it is possible to enrich it with additional pathtypes. In the Vase scene (Figure 13), we additionally track
ETTTTD.* paths, i.e. a sequence of four specular refractions
showing the distorted background behind the vase.
Each component is associated with a color buffer, andthe pixel-wise sum of these buffers yields the final image.
The residual component captures all unmatched paths andusually contains only low-frequency, low-magnitude content.
However, it is still important for the final image synthesis.
Texture/lighting separation. We further decompose, similarly to previous work [WRC88], the individual components
into irradiance and reflectance, to separate texture from lighting (Figure 2, columns two and three). This separation isbeneficial for image-based methods as texture and lighting
exhibit fundamentally different characteristics in structure,noise, and motion. For instance, the motion of the reflectancebuffer is determined by the motion of shaded objects, whereas

H. Zimmer et al. / Path-space Motion Estimation and Decomposition for Robust Animation Filteringcolor= reflectance ×

irradiance

E

ED →

ED.+

⏚

ED
ET|ER

color

color

refl

⏚

mvec

ED.+ →ETTD.* → ERD.* →

⏚

color

ETT

LegendD diffuse scattering
R specular reflection
T specular transmission.* arbitrary length suffix (≥ 0 vertices).+ arbitrary length suffix (≥ 1 vertices)
captures unmatched energy

⏚

color

refl

ERD.*

⏚

ETTD.*
mvec

color

refl

mvec

→

Figure 3: A schematic illustration of the decomposition from
Figure 2, showing its tree-like structure. As rays are traced
from the camera/eye (“E”), their contribution and auxiliary
data is stored into one of several image buffers (green).

Figure 2: Decomposition of the Robot scene from Figure 1,
using the regular expression notation illustrated in Figure 3.
The components (from top to bottom) are: direct diffuse (ED),indirect diffuse (ED.+), specularly reflected (ERD.*), specularly transmitted (ETTD.*), and a residual ( ) that subsumes all previously unmatched paths. For each component
we extract the color (first column), as well as the reflectance(second column), and compute the effective irradiance (thirdcolumn) as the ratio between them. The final image is obtained by multiplying each component’s irradiance by its
reflectance, and adding up the resulting colors.

the apparent motion of the irradiance is also affected by the
motion of occluders and light sources.
We now detail the texture/lighting separation for the direct
diffuse component, but the approach can be trivially generalized to all other components. The direct diffuse component’s
observed color can be expressed as:
Z

ED→color =S2

ρ(ωi , ωo )Ld (ωi ) dω⊥i ,where we integrate the product of the BSDF ρ(ωi , ωo ) and the
direct incident radiance Ld (ωi ) over the space of projected
solid angles ω⊥
i for the outgoing direction ωo . As mentioned
previously, we additionally output a reflectance buffer during
rendering which contains a Monte Carlo estimate of
ED→reflectance = ρ(ωo ) =

1
π

Z
S2

ρ(ωi , ωo ) dω⊥i .

In the case of a perfectly diffuse Lambertian surface, ρ(ωo )is the standard directionless reflectance ρ of the surface, but
this expression nicely generalizes to an “effective reflectance”
for non-Lambertian materials containing glossy transmission
or reflection. Previous works then compute the irradiance as
the integral of all incoming radiance Ld (ωi ) and reconstruct

the component’s color as the product of reflectance and irradiance. However, this breaks down whenever the product of
the integrals of reflectance and irradiance over a pixel differsfrom the integral of the products, which occurs in the presence of non-Lambertian surfaces, distribution effects such as
depth-of-field or simple spatial anti-aliasing. Unlike previous
work, we do not directly evaluate the irradiance, but instead
compute an “effective” irradiance as the ratio between thecomponent’s color value and the effective reflectance,
ED→irradiance =ED→color.
ED→reflectanceAs very low reflectance values lead to numerical instability
we do not divide by the reflectance when it is below 10−3
and instead directly use the radiance as irradiance, which canbe done as these light paths do not contribute measurably.
This effective irradiance factorization circumvents the limitations of the standard irradiance factorization (see Figure 11
for an illustration) by enforcing a linear shading model on
all pixels. This ensures, by construction, that the component’s color can always be reconstructed as the product ofreflectance and effective irradiance.
Features. We also associate each component with a set offeatures that can tangibly improve the performance of imagebased methods [HEMS10, SD12]. This data is collected at
the first non-specular vertex of each path and can thus beeasily obtained as a byproduct of path tracing. We extractreflectance (as mentioned before), normal, object ID, face ID,
texture coordinates, and emitted radiance from visible lightsources (see Figure 9 for examples). The features capture
most of the characteristics of the corresponding components,
including the details of the geometry seen through reflection
or refraction.
We implemented our decomposition in the Mitsuba renderer [Jak10], as detailed in the supplementary material. The
results shown in Section 5 heavily use Mitsuba’s (smooth)plastic and microfacet-based roughplastic materials, which simulate a coated diffuse layer with a configurable
Fresnel transmission and a nonlinear dependence on albedo.
Similar coupled specular-matte materials are also used in
c 2015 The Author(s)Computer Graphics Forum c 2015 The Eurographics Association and John Wiley & Sons Ltd.

H. Zimmer et al. / Path-space Motion Estimation and Decomposition for Robust Animation Filtering

ED→reflectance(primary) motion vectors

Figure 4: The diffuse reflectance component and its (forward) motion vectors—which we refer to as primary motion
vectors—color coded as shown in the top right inset.
industrial rendering systems [HMD∗ 14]. However, their behavior cannot be handled with prior albedo-irradiance decompositions and requires our effective irradiance factorization.
4

Motion Estimation

Image-based methods such as frame interpolation and temporally stable denoising require accurate motion vectors foreach component of the decomposition. Disregarding the effects of shading and lighting, it is straightforward to extract
motion vectors of visible surface positions on the scene geometry by mapping the underlying intersections forward in timeand projecting the 3D motion into screen space (Figure 4). In
the remainder we refer to these as primary motion vectors.
4.1

Specular Flow via Temporal Manifold Exploration

Specular motion vectors are significantly more challenging
to extract due to the complex interaction of position, motion,orientation and curvature of the involved objects. In the following we show how to compute highly accurate specularmotion vectors (“specular flow”) directly within the renderer.
We propose a generalized version of the manifold exploration (ME) [JM12] technique to compute the apparent motion of objects observed through a sequence of specular reflection or refractions. ME is based on the observation thatlight, which undergoes specular scattering events, follows trajectories that lie on a lower-dimensional manifold of transport
paths akin to configuration spaces of a mechanical system.
By finding local parameterizations of this manifold, it is possible to explore it via a sequence of one or more local steps.
In the original method, this was used to answer questions
like: “if a 3D point seen through a static curved glass object
moves, how does the corresponding observed point on thesurface of the glass object shift?”. By a temporal extension of
the underlying manifolds, we can answer the same question
for specular motion from frame to frame in general dynamic
scenes. Instead of directly solving for the image-space motion, we solve a slightly more general problem: given a lightpath with vertices x1 , . . . , xN at time t, we evolve its configuration up to the next frame t +1. We can then project thedifferences in the two configurations onto the image planeand obtain the image-space motion vectors.
We now summarize the key ideas of our extension herec 2015 The Author(s)Computer Graphics Forum c 2015 The Eurographics Association and John Wiley & Sons Ltd.

Figure 5: A moving non-specular object observed through apair of moving specular refractions. To compute the effective
motion of the object in the rendered image, we fix vertices x1
and x4 of the light path and perform an implicit solve for thepath configuration at the next frame.

and refer the reader to Jakob’s thesis [Jak13] for a detailed
explanation of the original technique for static scenes.
Vertex x1 is assumed to be a position on the aperture of thecamera, and xn (n ≤ N, where N denotes the total number ofpath vertices) is an interaction with a non-specular material
that is observed through a chain of specular interactions. We
are only interested in the behavior up to the first non-specularinteraction or light source xn (e.g. x5 in Figure 5) and ignore
any subsequent vertices xn+1 , . . . , xN . Each specular interaction between x1 and xn can be interpreted as a constraintthat requires the incident and outgoing directions to satisfy ageometric relationship: in the case of a mirror, it requires theinclinations of the incident and outgoing directions to match.
These constraints effectively collapse the set of contributing
light paths to a lower-dimensional manifold, which can be
explored using a Newton-like root-finder involving derivative
computation and projection steps, which we describe in turn.
Our approach assumes the rendering system has the capability of querying the position of the path vertices overtime while keeping them rigidly attached to the underlying camera, shape, or light source. Hence, given the initial
vertices x1 (t), . . . , xn (t), we can find their future positions
x1 (t +1), . . . , xn (t +1). Generally, this new path is not in avalid configuration anymore, i.e. it does not satisfy the lawsof specular reflection or refraction everywhere. We therefore
derive a correction term that attempts to bring the vertices
back into a valid configuration by analyzing the geometricproperties of a local first-order approximation of the manifoldof valid light paths.
We assume that each vertex xi (t) has linearly independent
tangent vectors ∂u xi (t) and ∂v xi (t), and that its position canbe differentiated with respect to time, yielding a 3D motion
vector ∂t xi (t). We use these three quantities to define a Taylor
approximation x̂i centered around the current vertex positionxi (t) which parameterizes the vertex on a small neighborhoodin space (parameters u, v) and time (parameter t):x̂i (u, v,t) = xi + u · ∂u xi + v · ∂v xi + t · ∂t xi(1)

Our first change to the original ME derivation is the additionof the last temporal term, which introduces extra derivative
terms that propagate through the subsequent steps. For brevity,

H. Zimmer et al. / Path-space Motion Estimation and Decomposition for Robust Animation Filtering

we now assume that all accented quantities are parameterizedby (u, v,t). Similarly to the above equation, we define aninterpolated shading normal n̂i by replacing all occurrencesof xi by ni and normalizing the result of the interpolation.
Finally, we complete n̂i to an orthonormal three-dimensionalframe {ŝi , t̂i , n̂i }, where the ŝi is aligned with ∂u xi and wheret̂i = n̂i × ŝi .
Suppose now that a specular reflection or refraction with arelative index of refraction η takes place at vertex xi (η = 1 in
the case of reflection). If the vertex is in a valid specular configuration, its generalized half-direction vector [WMLT07]ĥi =x̂i−1 − x̂ix̂ − x̂i+ η i+1kx̂i−1 − x̂i k
kx̂i+1 − x̂i k(2)

is collinear with the normal n̂i . An equivalent way of stating
this property is that the projection of ĥi onto the interpolated
coordinate frameĥi · ŝiĉi =(3)ĥi · t̂ivanishes, i.e. ĉi = 0. A subpath x1 , . . . , xn with endpoints x1
and xn must then satisfy n−2 such constraints (one for each
specular scattering event), which we collect and jointly write
as ĉ(u1 , v1 , . . . , un , vn ,t) = 0, where ĉ : R2n+1→ R2(n−2) .
This equation describes a first-order approximation of an
implicitly defined 5-dimensional manifold over light paths
embedded in a (2n+1)-dimensional space. Of particular interest are the tangent vectors of this high-dimensional manifold,which express how infinitesimal movements of one vertex
affect the rest of the specular chain; these can be obtained via
a simple application of the implicit function theorem, which
we discuss next.
To concretely reason about tangent vectors, we must firstchoose coordinates in which they should be expressed. In thiscontext, the most useful coordinates parameterize the intermediate vertex positions in terms of the endpoint positions
u1 , v1 , un , vn (4 dimensions) and time t (1 dimension). Let
Jĉ be the (square) Jacobian matrix of ĉ with respect to theremaining coordinates (i.e. the intermediate vertex positions):∂ĉ(0) ∂ĉ(0)∂ĉ(0) ∂ĉ(0)
Jĉ =
,
,...,
,
.
(4)
∂u2
∂v2
∂un−1 ∂vn−1
Then the desired tangent vectors are given by the columns of−1 ∂ĉ(0) ∂ĉ(0) ∂ĉ(0) ∂ĉ(0) ∂ĉ(0)T = −Jĉ
. (5)
,
,
,
,
∂u1
∂v1
∂un
∂vn
∂t
The involved derivatives are simple to evaluate using automatic differentiation. We use a dense LU factorization forthe linear system solve in Eq. (5), which could be optimized
to take advantage of the block tridiagonal structure of Jĉ forlarge n. We did not find this necessary as n ≤ 8 in all of ourexperiments (n = 8 was required to track quadruple refraction
paths in Figure 13). Note that n is only related to the depthof the decomposition, not the total path depth used in the
scattering simulation.

ERD.*→color

specular motion vectors

Figure 6: Specular reflection component and matching motion vectors found using Manifold Exploration. White pixels
mark light paths which cease to exist or cannot be tracked
to the next frame. When interpolating frames, we flag and recompute these pixels using a second sparse rendering phase.
In principle, we could now use the entries of the matrix T
to estimate motion vectors for specular paths: for instance,
the fifth column specifies how the path vertices should be perturbed to maintain a valid configuration for an infinitesimalchange in time. However, specular flow is highly nonlinear, and this extrapolation is not accurate enough even ona frame-to-frame basis. ME uses a sequence of alternating
extrapolation and projection steps to accurately solve for pathconfigurations; the projection step effectively re-traces thelinearly extrapolated path starting from the camera x1 , which
either fails or produces a corrected light path that satisfies all
specular constraints. A simple repetition of these two steps
leads to a Newton-like method with quadratic convergenceclose to the solution. As with standard Newton methods, it
is helpful to use an adaptive step size criterion to ensure thatthe linearity assumption is sufficiently satisfied (please refer
to Jakob and Marschner [JM12] for details). On a typicalframe of our Robot scene, the average number of spatial andtemporal iterations were 3.72 and 1.37, respectively.
In our case we pursue two simultaneous goals that are
different from the original method: we want to evolve a lightpath x1 , . . . , xn from time t to t +1 such that the endpoints
x1 and xn remain firmly attached to the underlying objects.
We achieve this using two nested solves: the inner loop is astandard (i.e. non-temporal) manifold walk invoked at timet < t 0 ≤ t+1 to ensure that the endpoint vertices are at theirtarget positions x1 (t 0 ) and xn (t 0 ). The outer loop is a temporal
manifold walk, which advances the path forward in time andensures that it remains in a valid configuration (though theendpoints may shift). Combined, they lead to a final set of
positions at time t+1 which allows us to evaluate the change
in position of the first vertex as seen from the camera, i.e.
x2 (t +1) − x2 (t), and project it into image space to obtainthe final motion vector vt (p), where p ∈ Ω denotes a pixel
position in the 2D image domain Ω.
The entire process is fast compared to Monte Carlo rendering, since only a few rays need to be traced per pixel (Table 1).
Our approach results in highly accurate motion vectors withreprojection errors on the order of machine precision whenever a light path could be successfully tracked from one framec 2015 The Author(s)Computer Graphics Forum c 2015 The Eurographics Association and John Wiley & Sons Ltd.

H. Zimmer et al. / Path-space Motion Estimation and Decomposition for Robust Animation Filtering

keyframe 1

warped keyframe 2

optical flow

Figure 7: Optical flow-based motion estimation for the irradiance component (left). Using the primary motion vectors(Figure 4), we perform a motion compensation by warpingthe second keyframe (middle) where white pixels mark occlusions. The resulting flow (right) is added to the primary
motion vectors to obtain the final motion estimate.

Figure 8: We detect silhouettes in the scene and mark them
to be ignored by image-based methods since they containcompeting motion vectors.

4.3
to the other. We flag light paths that could not be tracked orthat do not exist in one of the frames (Figure 6), so thatimage-based methods can treat them accordingly, e.g. by recomputing the associated pixels in a final sparse rendering
pass following frame interpolation.
4.2

Image-based Irradiance and Residual Flow

Motion in the irradiance components is the result of time
variation in a complex multiple scattering process. For the
residual component, motion vectors are equally challenging
to compute within the renderer due to the large variety of averaged path space components. For both of these components
we therefore resort to image-based optical flow to estimate
motion vectors. We base our approach on the well-establishedmethod of Brox et al. [BBPW04] and propose several extensions to improve accuracy in our application scenario.
Most importantly, we need to handle large displacements
due to fast object or camera motion, which are known todegrade the robustness of optical flow estimation. To this end
we leverage the primary flow to bootstrap the flow computation via a motion compensation, as illustrated in Figure 7.
Here, the primary motion vectors are used to warp the second frame towards the first frame by performing a backward
lookup, which compensates for camera and object motion.
The remaining motion is due to secondary effects such as
moving shadows, and can be robustly estimated using opticalflow. Adding these motion vectors to the primary motion vectors then gives the final motion vectors between the frames.
We also detect occlusions by a forward-backward consistency check that tests whether by following the (forward)
motion vector and its corresponding backward motion vector
one ends more than 0.1 pixels away from the original position.
Here, we cannot hope to find a matching pixel and disable
the data matching term in the optical flow solver, resulting ina smooth filling-in of motion vectors due to the smoothness
assumptions imposed in the optical flow formulation. See thesupplementary material for more details.
For the irradiance components, we perform the optical flow
computation in the logarithmic domain, where shadows at
different brightness levels can be better distinguished.
c 2015 The Author(s)Computer Graphics Forum c 2015 The Eurographics Association and John Wiley & Sons Ltd.

Silhouette and Occlusion Handling

Silhouette pixels capture light from both foreground and background which makes it impossible to define a single uniquemotion vector there. We hence detect silhouettes (Figure 8)
and ignore the corresponding motion vectors for image-based
methods, i.e. we treat them like untracked motion vectors
in the specular flow (Figure 6). We find silhouettes as signdifferences in the dot product of adjacent faces with the direction to the camera and rasterize these as 3D lines. This isvery efficient, taking 6s at 1280x720 pixels.
A similar reasoning applies to occlusions, where imagespace motion vectors would describe the motion of the occluded background points, which leads to artifacts when used
for image-based methods. We hence detect occlusions asmentioned before and also ignore motion vectors there.
5

Applications

We now show how our decomposition and motion vectors canbenefit different image-based post-processing methods: denoising of low sample-count renderings (Section 5.1), spatial
upsampling to a higher resolution (Section 5.2), and temporal upsampling, i.e. interpolation of in-between frames(Section 5.3) to increase the frame rate.
5.1

DenoisingFor denoising we build upon the joint NL-means filtering
approach of Rousselle et al. [RMZ13], which computes thedenoised value û(p) of a pixel p as a weighted average over asquare neighborhood N (p) : û(p) = ∑ q∈N (p) w(p, q) u(q),where w(p, q) = min(wc (p, q), w f (p, q)) combines a color
weight wc computed on the color buffer and a feature weightw f . If multiple features are available (as in our case), then w f
is the minimum of the feature weights. A neighboring pixel
q is therefore given a high weight only if it is similar to p
according to the color and each feature.
We can directly leverage our decomposition for a joint
NL-Means denoising since we render, for each component,
the final color, as well as the reflectance, normal, and object
ID features. We separately denoise the effective irradiance,computed as described in Section 3, of each component. The

H. Zimmer et al. / Path-space Motion Estimation and Decomposition for Robust Animation Filtering
input (512 spp)

color only

decomposition

decomp. & features

7.7·10−3 / 9.4·10−3

1.9·10−3 / 7.3·10−3

1.9·10−3 / 4.5·10−3

0.78·10−3 / 2.2·10−3

ED.+→color

ED→reflectance

ED→normal

ERD.*→color

ground truth (16k spp)

ERD→reflectance

ERD→normal

Figure 9: Denoising is significantly more robust when leveraging our decomposition. Top row: NL-Means filtering the final
color (“color only”) yields a smooth but blurry output; NL-Means filtering each component separately (“decomposition”)
yields sharper reflections, but low-constrast texture details, such as the text printed on the torso, are still problematic; joint
NL-Means filtering of each component guided by our auxiliary features (“decomp. & features”) robustly recovers fine details inthe scene and yields a result close to the ground truth. We give the relative MSE of each image for the full frame (first value)
and the crop shown (second value). Second row: noisy data and corresponding reflectance and normal buffers for the indirect
diffuse component (left three images) and the specular reflection component (right tree images), showing how the decomposition
succeeds in capturing the distinct noise characteristic and spatial structure of the previously composited shading effects.
color weight wc is computed on the irradiance buffer, and thefeature weight w f is computed on the normal and object ID
buffers. Once the irradiance is denoised, we multiply back thecorresponding reflectance buffer to obtain the denoised component color. All denoised components are finally summed
up to yield the denoised rendering.
In the work of Rousselle et al., a filter bank of three joint
NL-Means filters was computed and combined on a per-pixel
basis. One filter was more sensitive to color differences, one
more sensitive to feature differences, and a third one morebalanced. In our implementation, we opted to use a single
balanced filter. Combined with our decomposition, we foundthis simplified denoising scheme offered a better trade-off
between quality and complexity (see Figures 9 and 10 for results using both our implementation and the original scheme
proposed by Rousselle et al.).
Temporal denoising. Residual low-frequency noise canlead to highly noticeable large-scale flickering in denoised
animation sequences. This problem can be alleviated by aspatio-temporal filtering [HEMS10]. Extending a joint NLMeans filter to the spatio-temporal domain is easily achievedby augmenting the filtering window to include data from
temporally adjacent frames [BCM08]. However, one needs
to account for camera and scene motion to be able to leverage the coherence from frame to frame in an optimal way.
To this end, we warp every component of adjacent frames,
as well as the corresponding feature buffers, using the computed per-component motion vectors, which aligns them tothe current frame. This works in the same manner as forthe motion compensation described in Section 4.2. Whendenoising the irradiance, we use the computed geometry mo-tion vectors of the corresponding component (ED.*, ERD.*,or ETTD.*), where the motion of moving shadows is not
captured over time. However, the robust NL-Means weights
ensure that shadows are not excessively blurred despite themisalignment. From the denoised irradiance components wecan then compute motion vectors, as described in Section 4.2,
to be used in other applications, such as frame interpolation.
The results of spatio-temporal filtering across 3 frames areincluded in our supplementary video.
Results. In Figure 9 we compare different denoising results leveraging progressively more information from ourdecomposition. Using our decomposition already recovers
much more details compared to denoising the final colors
only. When adding the features for a joint NL-Means filtering, we achieve denoising results that are visually very close
to a high sample-count ground truth rendering with a low
relative mean-square error (MSE). See our supplementarymaterial for the parameters used for the joint NL-Means filter,
results at lower sampling rates, and an equal error rendering
without denoising.
We also applied the original denoising algorithm proposedby Rousselle et al. [RMZ13] on our Robot scene, to illustrate
how an advanced denoising scheme can benefit from ourdecomposition. The results are given in Figure 10 and show asignificant improvement. Given that the choice of denoising
method is orthogonal to our decomposition, we would expect
other methods to benefit similarly.
Spatial anti-aliasing and distribution effects. Our effective irradiance factorization can be used in cases where
the standard irradiance factorization is invalid, as explained
c 2015 The Author(s)Computer Graphics Forum c 2015 The Eurographics Association and John Wiley & Sons Ltd.

H. Zimmer et al. / Path-space Motion Estimation and Decomposition for Robust Animation Filtering
RMZ13

RMZ13 + decomp.

1.2·10−3

0.75·10−3

Figure 10: Denoising the robot scene using the original
method proposed by Rousselle et al. [RMZ13] (left image),and the same method applied on top of our decomposition(right image). Even with this more advanced method, ourdecomposition brings a significant improvement, both qualitatively and quantitatively, as measured by the relative MSE(which we computed on the full frame).
input (512 spp)

5.6·10−3

irradiance9.9·10−3

eff. irradianceground truth

0.90·10−3

Figure 11: Denoising of the Robot scene using irradiancefactorization with added depth-of-field. Although this cropshows only diffuse materials, the linear shading assumption
at the core of the irradiance factorization is invalidated by
the coupling between irradiance and reflectance induced by
the depth-of-field effect (and to a lesser extent the spatial
anti-aliasing). Denoising using standard irradiance factorization (second column) actually degrades the MSE becauseof the excessive bias (most noticeable in the shrunken region
pointed by the arrow). Our effective irradiance factorization(third column) does not suffer from such issues.

in Section 3. This notably occurs when using spatial antialiasing and in the presence of distribution effects such as
depth-of-field or motion blur. All results presented use spatial
anti-aliasing, and, in Figure 11, we illustrate the output of ourdenoising technique when adding depth-of-field to the Robot
scene and compare it to the result obtained using standard
irradiance factorization. Note that in this case, the features
also exhibit a noticeable amount of noise which requires toprefilter them as proposed in [RMZ13].
5.2

Upsampling

Dramatic increases in the pixel count of cinema and home
TV have made it prohibitively expensive to render animations
for these new formats. This can be alleviated by rendering at
a lower resolution, followed by an upsampling step that produces the desired output resolution. Similarly to the improvements we achieve for denoising, upsampling each component
separately yields tangibly better results, as multiple shadingc 2015 The Author(s)Computer Graphics Forum c 2015 The Eurographics Association and John Wiley & Sons Ltd.

input

color only

decomp.

8.2·10−3

3.4·10−3

ground truth

Figure 12: Upsampling a 640×360 rendering at 512 spp(“input”) to 1280×720 pixels. Upsampling of the final pixel
color guided by the features of the diffuse component (“color
only”) blurs out secondary effects, such as the refractions inthe sphere and the reflections on the robot. We can robustly reconstruct the reflections and refractions by applying the same
upsampling scheme on top of our decomposition (“decomp.”),yielding a much lower relative MSE (which we computed on
the full frame). The ground truth shown is the denoised full
resolution rendering at 512 spp.

contributions that would normally interfere in the upsampling
process can be disambiguated. Additionally we can leverage
auxiliary features that can be cheaply computed at the hightarget resolution to guide the upsampling [HEMS10].
Our upsampling application is based on the guided image
filter [HST13]. We use a joint upsampling scheme, where thecolor is upsampled using the features (reflectance, normal
and emitted radiance) as a guide, and each component is
upsampled individually. We directly render the image at thetarget high resolution but with correspondingly fewer samples.
For all examples, we rendered at 1/4 of the samples. We thensubsample using a 2×2 box filter, which results in a lowresolution image with reduced noise. The subsampled images
are denoised, and then upsampled back to the full resolution.
This reduces the sampling rate by a factor of 4, while keeping
the same signal-to-noise ratio in the low-resolution image.
Due to its unconstrained optimization, the guided image
filter upsampling can produce negative pixel values, particularly along strong edges. Pixels with negative values are
flagged to be re-rendered with a higher sampling count anddenoised at the full resolution. In practice, the re-rendering
rate is very low. For the Robot scene it varies between 0.14%
and 0.28% per frame, with an average of 0.20%.
Results. In Figure 12, we show that directly upsampling thefinal colors loses a lot of detail, despite being a common approach [SGNS07]. While it correctly reconstructs the diffuse
shading, it fails for secondary effects, such as reflections andrefractions. In contrast, the same upsampling scheme appliedon each component of our decomposition and guided by thefeatures yields a significantly improved result, with correctreflections and refractions, and a drastically reduced relative
MSE. Please see the supplementary video for upsampling
results on the full sequence.

H. Zimmer et al. / Path-space Motion Estimation and Decomposition for Robust Animation FilteringRobotcolor only

ours

7.3·10−3

1.7·10−3

ground truth

color only: 6.2·10−2

Vase

1.1·10−1

0.1·10−1

Figure 13: Interpolation with challenging specular effects.
The crop of the Robot scene (top) shows the handling ofreflections (ERD.*) whereas the Vase scene (bottom) showcases a highly complicated fourfold specular transmission(ETTTTD.*). The color only baseline method (second column) interpolates the final colors using the primary motion
vectors, yielding strong artifacts and a high relative MSE(which we compute for the full frame). Our approach of interpolating each component separately using the corresponding
motion vectors significantly improves results. We interpolated
3 in-between frames for both scenes.

5.3

Frame Interpolation

Frame interpolation proceeds by projecting pixels rendered atsparse keyframes to in-between frames using corresponding
motion vectors. To improve results one can use motion vectors computed forward and backward in time, and average thecontributions from the two neighboring keyframes, weighted
by their distance to the interpolated frame [YTS∗ 11].
In our decomposition framework, we interpolate each component separately using the corresponding motion vectors.
This remedies ghosting artifacts that appear in the the presence of complex secondary effects such as reflections andrefractions where a single pixel receives contributions fromvarious sources that might all undergo different motions.
As we compute specular motion vectors at the final frame
rate, they are defined between subsequent frames. For interpolation, we however need motion vectors defined between keyframes and the current in-between frame, which
we achieve by concatenating the motion vectors. The same
applies to the primary motion vectors used for the diffuse
reflectance component. For the irradiance and residual components, we compute motion vectors between the keyframes
using optical flow (Section 4.2) and thus just scale thesemotion vectors w.r.t. the position of the in-between frame.
Re-rendering. There are several reasons why it can be necessary to re-render some pixels in the interpolated frames ina second sparse render pass: Some specular paths cease toexist or cannot be tracked over time, which leads to unknown
pixels in the specular motion vectors (Section 4.1). We also

ours: 1.4·10−2

ground truth

Figure 14: Interpolation with challenging moving shadows.
The baseline method (color only) leaves halos whereas ourapproach of interpolating the irradiance and residual components separately using the estimated motion vectors (Section 4.2) significantly reduces these artifacts and lowers therelative MSE (computed on the full frame).
ignore motion vectors of silhouette pixels as they may capture objects with conflicting motion. Such undefined motion
vectors result in holes in the interpolated frame, which need
to be filled by re-rendering. Holes are also caused by pixels
that are occluded in both keyframes, but become visible in
the in-between frames. Finally, if the illumination changes
noticeably, a seam can occur between disocclusions where
only a single keyframe contributes and the neighboring regions where contributions from two keyframes are averaged,
making it necessary to also re-render the disoccluded regionin such cases. For the Robot sequence, we re-render between
8% and 33% of pixels, with an average of 20%.
Results. In Figure 13 we show that using our decompositions and the corresponding motion vectors resolves theghosting at specular effects that occur when interpolating thefinal colors only using the primary motion vectors. Figure 14
shows results for a scene with complicated moving shadows.
These are captured in the diffuse irradiance and the residualcomponents and we estimate their motion vectors using optical flow (Section 4.2). Consequently we cannot match theaccuracy of the specular motion vectors obtained using Manifold Exploration, but still obtain visually appealing results.
The supplementary video shows interpolation results onfull sequences where the ghosting artifacts result in strongly
visible, temporal flickering.
Glossy objects. In the video we show additional scenes, including one featuring a highly glossy object (Glossy Sphere).
Here, we compute specular motion vectors the same way as
for ideally smooth ones, i.e. we pretend that they are smoothwhen running the Manifold Exploration. This is based on an
empirical observation that the effective motion in the smoothand glossy case is almost identical, even though the appearance in the rendering may significantly deviate. The resultsin the video validate that this approximation is sufficiently
accurate to obtain favorable interpolation results.
Motion blur. We can also use the computed motion vectors
to add motion blur as a post-process by integrating (splatting)c 2015 The Author(s)Computer Graphics Forum c 2015 The Eurographics Association and John Wiley & Sons Ltd.

H. Zimmer et al. / Path-space Motion Estimation and Decomposition for Robust Animation Filteringno motion bluradded motion blur(half exposure)added motion blur(full exposure)

Figure 15: We can use our per-component motion vectors tosimulate different amounts of motion blur as a post-process.
the pixel colors of each component along the motion vectors [CW93, BE01]. As shown in Figure 15, this not only
produces realistic motion blur, but also allows us to easily
control the amount of blur, i.e. the virtual exposure time, after
rendering. We fill pixels with undefined specular motion vectors (Figure 6) using standard image inpainting [BSCB00],
which we found sufficient to obtain visually plausible results.
5.4

Combination

To combine the previously described image-based methods into a single pipeline we first produce high resolution
keyframes by combining denoising and upsampling (Section 5.1 and 5.2). Then we interpolate in-between frames,
followed by re-rendering (Section 5.3). Corresponding results are shown in Figure 1 and in the supplementary video.
6

MethodRender Denoise Upsample Interpolate Resolution Overhead Rel. Cost

Base

178.0

–

–

–

–

–

–

GT
D
DI
DU
DUI

5696.0
208.2
210.8
61.7
64.3

–
5.5
5.4
1.5
1.4

–
–
–
12.2
11.1

–
–
172.1
–
131.4

1×
1×
4×
4×
16×

32.0×
1.2×
2.2×
1.7×
4.7×

32
1.2
0.55
0.43
0.29

Finally, we did not investigate volumetric effects in thispaper. Recent efforts in the movie industry [Kai12] involvethe use of “deep” buffers to capture and decompose the effects
of volumetric light transport for the purpose of compositing.
Extending our system with additional “deep” components
would be an interesting avenue of future work.

Performance

Table 1 shows the speedups achieved by several combinationsof our post-processing methods on the Robot scene, running a
CPU implementation on a cluster. Storing the decomposition
incurs a memory overhead proportional to its granularity. The
data is stored in compressed multi-channel EXRs, leveraging
that some components have many zero pixels. The frame
shown in Figure 2 is 29MB in total, where the final pixel
color accounts for 3.7MB.
7

Table 1: Average computation times per input frame (incore minutes) on the Robot scene (1280×720 pixels). The
noisy baseline render without extracting our decomposition(Base) used 512 spp, whereas an (almost) noise-free groundtruth render (GT) required 16k spp. The last three columns
contrast the gain in resolution (due to upsampling or interpolation) vs. the computational overhead compared tothe baseline, and give the relative cost as the ratio between
the two. Clearly, as we combine more post-processing steps(Denoising, Upsampling, Interpolation) we reduce the relative cost. Note that the Upsample and Interpolate columns
also include the re-rendering and denoising of missing pixels,
which for the interpolation make up for 90% of the time.

Limitations

Our decomposition cannot fully disambiguate motion in thecase of silhouette pixels overlapping regions with conflicting motions, which we handle by detecting and re-rendering
these pixels. However, this heuristic is very conservative andcould possibly be improved, leading to further performancegains. Similarly, our heuristic for handling occlusions is also
very conservative, and it may be possible to use more advanced blending techniques, e.g., Poisson blending [PGB03]
to handle occlusion seams with less re-rendering.
We also envision an automatic method to determine thecomponents that are needed to best represent a given scene,e.g. by analyzing the materials. This could be supported by acode generation tool that automatically instruments the pathtracing integrator for a given decomposition.
c 2015 The Author(s)Computer Graphics Forum c 2015 The Eurographics Association and John Wiley & Sons Ltd.

8

Conclusions

We have presented a general and flexible decomposition
framework for path tracing-based rendering, which resolves
ambiguities that have so far hampered the adoption of imagebased post-processing methods in scenarios with high quality
requirements. We demonstrated the benefits of our approachfor denoising, spatial upsampling and frame interpolation.
Our decomposition leverages two key contributions. First,
we provide motion vectors for all components of the decomposition, including specular paths for which we derived atemporal extension of manifold exploration, and the irradiance and residual components for which we used image-based
optical flow bootstrapped with the motion of visible objects.
Second, we proposed a simple modification of irradiancefactorization that can, once combined with our decomposition, handle general BRDFs, including specular surfaces, andchallenging distribution effects such as depth-of-field.
Acknowledgments. We thank Maurizio Nitti for creating
the Robot scene and Olesya Jakob for designing the vaseshown in Figure 13. We also thank our colleagues at theWalt Disney Animation Studios and Industrial Light & Magic
for fruitful discussions. Wenzel Jakob was supported by an
ETH/Marie Curie fellowship.

H. Zimmer et al. / Path-space Motion Estimation and Decomposition for Robust Animation Filtering

References[BBPW04] B ROX T., B RUHN A., PAPENBERG N., W EICKERTJ.: High accuracy optical flow estimation based on a theory for
warping. In ECCV (2004), pp. 25–36. 2, 7
[BCM08] B UADES A., C OLL B., M OREL J.: Nonlocal image andmovie denoising. International Journal of Computer Vision 76, 2(2008), 123–139. 8[BE01] B ROSTOW G. J., E SSA I.: Image-based motion blur forstop motion animation. In SIGGRAPH (2001), pp. 561–566. 11[BMS∗ 12] B OWLES H., M ITCHELL K., S UMNER R. W., M OOREJ., G ROSS M. H.: Iterative image warping. Comp. Graph. Forum31, 2 (2012), 237–246. 2[BSCB00] B ERTALMIO M., S APIRO G., C ASELLES V.,
BALLESTER C.: Image inpainting. In SIGGRAPH (2000),
pp. 417–424. 11[CW93] C HEN S. E., W ILLIAMS L.: View interpolation for image
synthesis. In SIGGRAPH (1993), pp. 279–288. 3, 11[DER∗ 10] D IDYK P., E ISEMANN E., R ITSCHEL T.,
M YSZKOWSKI K., S EIDEL H.-P.: Perceptually-motivatedreal-time temporal upsampling of 3D content for high-refresh-rate
displays. Comp. Graph. Forum 29, 2 (2010), 713–722. 3[DKR∗ 14]DABAŁA˛
Ł., K ELLNHOFER P., R ITSCHEL T., D IDYK
P., T EMPLIN K., M YSZKOWSKI K., ROKITA P., S EIDEL H.-P.:
Manipulating refractive and reflective binocular disparity. Comp.
Graph. Forum (Proc. Eurographics 2014) 33, 2 (2014), 53–62. 2[God14] G ODDARD L.: Silencing the noise on Elysium. In ACM
SIGGRAPH 2014 Talks (2014), ACM. 2[Hec90] H ECKBERT P. S.: Adaptive radiosity textures for bidirectional ray tracing. In SIGGRAPH (1990), pp. 145–154. 3[HEMS10] H ERZOG R., E ISEMANN E., M YSZKOWSKI K., S EI DEL H.-P.: Spatio-temporal upsampling on the GPU. In SI3D
(2010), pp. 91–98. 2, 3, 4, 8, 9
[HMD∗ 14]H ILL S., M C AULEY S., D UPUY J., G OTANDA Y.,
H EITZ E., H OFFMAN N., L AGARDE S., L ANGLANDS A., M EG IBBEN I., R AYANI F., DE ROUSIERS C.: Physically based shadingin theory and practice. In ACM SIGGRAPH 2014 Courses (NewYork, NY, USA, 2014), SIGGRAPH ’14, ACM. 2, 5

[HST13] H E K., S UN J., TANG X.: Guided image filtering. IEEE
Trans. Pattern Anal. Mach. Intell. 35, 6 (2013), 1397–1409. 3, 9
[Ige99] I GEHY H.: Tracing ray differentials. In SIGGRAPH(1999), pp. 179–186. 2[Jak10] JAKOB W.: Mitsuba renderer, 2010. http://www.
mitsuba-renderer.org. 4[Jak13] JAKOB W.: Light Transport on Path-Space Manifolds.
PhD thesis, Cornell University, Aug. 2013. 5[JM12] JAKOB W., M ARSCHNER S.: Manifold exploration: A
markov chain monte carlo technique for rendering scenes withdifficult specular transport. ACM Trans. Graph. 31, 4 (2012),
58:1–58:13. 2, 5, 6

[MJL∗ 13] M OON B., J UN J. Y., L EE J., K IM K., H ACHISUKA
T., YOON S.: Robust image denoising using a virtual flash image
for monte carlo ray tracing. Comp. Graph. Forum 32, 1 (2013),
139–151. 3[MMB97] M ARK W. R., M C M ILLAN L., B ISHOP G.: Postrendering 3D warping. In SI3D (1997), pp. 7–16, 180. 2[MYRD14] M EHTA S. U., YAO J., R AMAMOORTHI R., D URAND
F.: Factored axis-aligned filtering for rendering multiple distribution effects. ACM Trans. Graph. 33, 4 (July 2014). 2[NVI12] NVIDIA: Iray whitepaper, 2012. http://www.
nvidia-arc.com/fileadmin/user_upload/iray_
2013/documents/iray_whitepaper.121206.pdf. 2[PGB03] P ÉREZ P., G ANGNET M., B LAKE A.: Poisson image
editing. ACM Trans. Graph. 22, 3 (July 2003), 313–318. 11[RMZ13] ROUSSELLE F., M ANZI M., Z WICKER M.: Robust denoising using feature and color information. Computer Graphics
Forum 32, 7 (2013), 121–130. 3, 7, 8, 9
[SD12] S EN P., DARABI S.: On filtering the noise from the randomparameters in monte carlo rendering. ACM Trans. Graph. 31, 3(2012), 18. 2, 3, 4
[SGNS07] S LOANP.
J.,
G OVINDARAJUN.
K.,N OWROUZEZAHRAI D., S NYDER J.: Image-based proxyaccumulation for real-time soft global illumination. In PacificGraphics (2007), pp. 97–105. 9[Shi95] S HINYA M.: Improvements on the pixel-tracing filter:
Reflection/refraction, shadows & jittering. In Graphics Interface(1995), CANADIAN INFORMATION PROCESSING SOCIETY,
pp. 92–92. 2[SIMP06] S EGOVIA B., I EHL J. C., M ITANCHEY R., P ÉROCHE
B.: Non-interleaved deferred shading of interleaved sample patterns. In Proc. Graphics Hardware (2006), ACM. 2[ST90] S AITO T., TAKAHASHI T.: Comprehensible rendering of
3-d shapes. SIGGRAPH Comp. Graph. 24, 4 (Sept. 1990). 2[SYM∗ 11] S CHERZER D., YANG L., M ATTAUSCH O., N EHAB
D., S ANDER P. V., W IMMER M., E ISEMANN E.: A survey on
temporal coherence methods in real-time rendering. In EUROGRAPHICS 2011 State of the Art Reports (2011). 3[Vea98] V EACH E.: Robust Monte Carlo Methods for Light
Transport Simulation. PhD thesis, Stanford, CA, USA, 1998.
AAI9837162. 2, 3
[WKB∗ 02] WALD I., KOLLIG T., B ENTHIN C., K ELLER A.,
S LUSALLEK P.: Interactive global illumination using fast ray
tracing. In Proceedings of the 13th Eurographics Workshop on
Rendering (EGWR) (2002), pp. 15–24. 2[WMLT07] WALTER B., M ARSCHNER S. R., L I H., T ORRANCE
K. E.: Microfacet models for refraction through rough surfaces.
In Rendering Techniques (2007), pp. 195–206. 6[WRC88] WARD G. J., RUBINSTEIN F. M., C LEAR R. D.: A
ray tracing solution for diffuse interreflection. SIGGRAPH Comp.
Graph. 22, 4 (June 1988), 85–92. 2, 3

[Kai12] K AINZ F.: Interpreting OpenEXR deep pixels, 2012.
http://openexr.com/InterpretingDeepPixels.
pdf. 11[YTS∗ 11] YANG L., T SE Y.-C., S ANDER P. V., L AWRENCE J.,
N EHAB D., H OPPE H., W ILKINS C. L.: Image-based bidirectional scene reprojection. ACM Trans. Graph. 30, 6 (2011). 10[KKG∗ 14]

[ZJL∗ 15] Z WICKER M., JAROSZ W., L EHTINEN J., M OON B.,
R AMAMOORTHI R., ROUSSELLE F., S EN P., S OLER C., YOON
S.: Recent advances in adaptive sampling and reconstruction formonte carlo rendering. Comp. Graph. Forum, to appear (2015). 3

K RIVÁNEK J., K ELLER A., G EORGIEV I., K A PLANYAN A., FAJARDO M., M EYER M., NAHMIAS J.-D., K AR LÍK O., C ANADA J.: Recent advances in light transport simulation: Theory and practice. In ACM SIGGRAPH Courses (2014),
SIGGRAPH ’14, ACM. 1[LRR∗ 14] L OCHMANN G., R EINERT B., R ITSCHEL T., M ÜLLER
S., S EIDEL H.: Real-time reflective and refractive novel-view
synthesis. In VMV 2014 (2014), pp. 9–16. 2, 3

[ZKU∗ 04] Z ITNICK C. L., K ANG S. B., U YTTENDAELE M.,
W INDER S. A. J., S ZELISKI R.: High-quality video view interpolation using a layered representation. ACM Trans. Graph. 23, 3(2004), 600–608. 3

c 2015 The Author(s)Computer Graphics Forum c 2015 The Eurographics Association and John Wiley & Sons Ltd.