Depth-aware Motion Magnification
Julian F. P. Kooij1,2 and Jan C. van Gemert1
1

Delft University of Technology, Delft, The Netherlands{J.F.P.Kooij,J.C.vanGemert}@tudelft.nl
2
Leiden University Medical Center, Leiden, The Netherlands

Abstract. This paper adds depth to motion magnification. With therise of cheap RGB+D cameras depth information is readily available. We
make use of depth to make motion magnification robust to occlusion andlarge motions. Current approaches require a manual drawn pixel maskover all frames in the area of interest which is cumbersome and errorprone. By including depth, we avoid manual annotation and magnify
motions at similar depth levels while ignoring occlusions at distant depth
pixels. To achieve this, we propose an extension to the bilateral filter for
non-Gaussian filters which allows us to treat pixels at very different depth
layers as missing values. As our experiments will show, these missing
values should be ignored, and not inferred with inpainting. We show
results for a medical application (tremors) where we improve current
baselines for motion magnification and motion measurements.
Keywords: Motion magnification, bilateral filter, RGB+D

1

Introduction

Magnifying tiny motions in video [3,4] opened up a wealth of applications. Examples include: reconstructing speech exclusively from small visual vibrations [5],detecting a heart-beat either from blood flow [4] or from tiny head motions [6],magnifying muscle tremors [7], segmenting blood vessels [8] or estimating material properties by the way it moves [9]. In this paper we propose to only magnify
motion at selected depth ranges, which makes motion magnification robust to
occlusions and large motions at other depths. Robustness is especially important to open up new applications in the medical domain such as tremor assessment [10–12], where the interaction between doctor and patient should not be
disturbed, and prerequisites for video processing should not limit the poses andexercises dictated by the medical protocol.
Currently though, magnifying tiny motions requires that there are no occlusions or large motions [1, 3, 4]. A recent solution proposes to manually indicate
the large motions by drawing a binary pixel mask on the frames of interest [2].
While a mask indicates which pixels should be used, it does not solve how toignore the motion filter responses on the edge of the mask. Motion filters have acertain spatial extent and they ‘leak’ across the mask border. Moreover, manually

2

Julian F. P. Kooij and Jan C. van Gemert(a) Frame from sequence 1(c) original(d) magnified [1](b) Frame from sequence 2(e) CVPR’15 [2](f) ours

Fig. 1: Comparison of our and baseline magnification approaches when magnifying small motions in the background (here, body) behind moving occluders(here, trembling hands). (a),(b) For two sequences, the input image, depth map,and depth-dependent magnification matte of one frame (black/white is zero/full
magnification). (c)-(f) Space-time slices for the red lines in input images. Our
approach suppresses unwanted magnification artifacts from the foreground in
the magnified background. See supplementary material for videos.

drawing such a mask on a moving target is challenging and time-consuming. We
instead exploit depth to automatically define the mask. Furthermore, we preventthe ‘leaking’ by ignoring motion responses from very different depths whereas
filter responses from close-by depth layers are weighted.
Several techniques are available for weighting filter responses [13–16]. These
techniques allow weighted Gaussian smoothing or interpolation, for example,
on intensity differences resulting in edge-preserving smoothing. However, highquality motion magnification [1] depends on the complex steerable pyramid [17,18] which consists of non-Gaussian filters for which standard weighting of filter
responses [13,14,16] cannot be used. To illustrate, consider a Gaussian derivative
filter. Since it integrates to zero, it will give no response on a constant valued
input image. Intuitively, the response should not change if some parts of the
input are ignored, but reducing some filter weights to zero would now actually
yield non-zero output. In other words, the Gaussian derivative cannot be treatedas a weighted input aggregate. In this paper we therefore develop filter weightingof non-Gaussian filters, which can ignore input by treating it as missing values.
When images have missing values, there are several advanced inpainting techniques [19–23] available to estimate what is lost. It is not clear, however, howDepth-aware Motion Magnification3

inpainting can be used to infer missing values between multiple depth layers.
We propose a different goal. We do not want to recover what is lost: we want toignore what is there.
In the following sections we first discuss related work, then how to ignore
filter responses from different depth layers, and how this allows depth-aware
motion magnification. We experimentally compare against inpainting and show
example applications in the medical domain on hand tremors.

2

Related work

Motion can be magnified by explicitly tracking feature points with optical flow [24].
The motion is magnified by re-scaling the moving points and adding them backto the video. Optical flow is estimated locally between pairs of frames which isnoisy. This noise affects the motion magnification since local motion is represented by a single unique feature point. In contrast to feature point tracking,Eulerian video magnification [4] estimates motion frequency over longer time periods which is more stable. Thus, the method is well-suited for amplifying tiny
imperceptible motions. Impressive improvements [1] on the stability of linear
motion magnification [4] are made by relying on complex steerable pyramid filters [17,18]. A significant speedup without perceptual decrease in quality can beobtained by approximating the complex pyramid with the Riesz pyramid [25].
While extremely successful for clean video sequences, all these methods assume
that there are no occlusions or large motions present. Our method is specificallydesigned to deal with such cases.
With some help by the user, occlusion or large motions can be manuallyindicated. Examples of user input on video processing include de-animation [26],blending between face performances [27], video segmentation [28], and video
stabilization [29]. For motion magnification a manual drawn mask can specify
which pixels to magnify and which pixels to ignore [2]. In this paper we extend
this line of reasoning, replacing the manual drawn mask by a weighted mask
obtained from depth to ignore filter responses outside a target depth range.
Incorporating weighted responses in a filter is done with the bilateral filter [16]. It applies Gaussian blurring to an image, but locally adapts the Gaussian weights to suppress contributions of neighbourhood pixels with very different intensity levels. The fast bilateral filter [15] offers a significant speedup by
approximation. This is achieved by transforming the 2D input image into a 3D
sparse matrix, where the 3rd z-dimension is given by a pixel’s intensity level.
The speedup comes from allowing standard 3D convolutions to obtain intensityweighted responses. In this paper we begin with the fast bilateral filter [15] due
to its speed. However, where the bilateral filter only allows weighted Gaussiansmoothing or interpolation we require non-Gaussian filters: the complex steerable pyramid [17, 18] as used in high-quality motion magnification [1]. Insteadof weighting values, we adapt the bilateral filter so it can handle missing values.
Inferring missing values in images by inpainting typically exploits texture
synthesis and pixel consistency [19,30]. Strong step edges can be retained [21] and4Julian F. P. Kooij and Jan C. van Gemert

image statistics through patch-exemplars can give a good prior on what values toinfer [23]. Inpainting can be done efficiently [20], making it in principle suitablefor video processing. While inpainting could be used to fill in missing values forvery different depth layers, it is not clear how to use inpainting to combine closer
depth layers. In contrast to inpainting we do not wish to infer what should bepresent at all depth layers. Our goal is to remove all filter influences from pixels
at different depth layers.

3

ApproachThis section starts with the bilateral filter formulation [16], followed by our nonGaussian extension. We then apply the developed technique to complex steerable
pyramids and use these for occlusion-aware Eulerian motion magnification [1, 2,
31] and measurement [32]. We note that other image processing tasks could also
benefit from the non-Gaussian bilateral filter (see supplementary material for
examples), and for instance use intensity, optical flow, or color instead of depth
to filter micro-textures, stationaries, surfaces.
3.1

Bilateral filter

The bilateral filter [16] can be used for depth-aware smoothing. Given input
image I and corresponding depth image E, the bilateral filter computes output
image O. By defining y ∈ N (x) as the local a neighbourhood of 2D image
locations x = (u, v), and using O(x) as a shorthand for O(u, v), the bilateralfilter can be written as a weighted averageO(x) =

X1
w(|x − y|, E(x) − E(y)) I(y)
W (x)

(1)

y∈N (x)w(ds , dE ) = G(ds ; σs ) × G(dE ; σr )(2)Pwhere W (x) = y∈N (x) w(|x−y|, E(x)−E(y)) is the weight normalization termat x, and G(x; σ) = exp − 2σx2 is the Gaussian kernel. The positive weights
w(dI , dE ) approach zero as the spatial distance ds or the depth distance dE
increases. There are two smoothing parameters, the spatial standard deviation
σs , which controls the amount of spatial blurring as is in a normal Gaussian
image filter, and the depth standard deviation σr , which controls how strong
pixels on different depth layers are weighted.
3.2

Bilateral filter for non-Gaussian kernels

Consider some non-Gaussian kernel F with negative values, for instance F isan oriented band-pass filter used in a steerable pyramid [18], or a Gaussianderivative. As with the Gaussian bilateral filter, we would like to apply F toan input image I, but obtain filter responses representative of the local spatial

Depth-aware Motion Magnification

5neighbourhood with nearby depth values. While the bilateral filter with Gaussian
kernel can be seen as a weighted average, we cannot simply replace the kernel by
F . For instance, the integral of a Gaussian derivative kernel is zero, and would
yield a division by zero in normalization. Also, one cannot ignore part of the
input by reducing corresponding weights to zero, since this introduces unwantededge responses as if the input itself partly has zero values; our experiments in
Section 4.1 will illustrate this point.
Instead, we propose to reduce the influence of regions in distant depth layersby smoothly incorporating the spatial image structure at the local depth layer.
Using ξ = E(z) to denote the depth at the output location z, the non-Gaussianbilateral filter with output Q is written as
X
Q(z) =
F (|x − z|)O+ (x, E(z))(3)
x∈N (z)O+ (x, ξ) =1

X

W + (x, ξ)w(|x − y|, ξ − E(y))I(y)(4)

y∈N (z)with w(ds , dE ) again
P some weight function (which we will define in a moment),and W + (x, ξ) = y∈N (z) w(|x − y|, ξ − E(y)). Here the + suffix indicates that afunction operates on 3D space by extending the spatial domain with additional
depth information from E. Throughout this paper we shall use the + suffix
notation more frequently, and refer to it as an extended representation. Notethat Q(x) is not just a convolution with F after applying a bilateral filter, sinceO+ (x, ξ) is not only a function of x. But, our formulation does have the regularbilateral filter, Eq. (1), as special case when F (x) = 1 iff x = 0 and 0 otherwise.
We reformulate our filter to a 3D representation similar to the fast bilateral filter [15]. This has two benefits: One, as with the standard bilateral filter,
explicit evaluation of Equation (3) is inefficient, since filter coefficients need tobe reweighted at each spatial location. The 3D representation instead offers a
trade off between quality and speed [15]. Two, this 3D representation explicitly
keeps filter responses at different depths separated, which will be exploited in
our applications to perform depth-aware temporal filtering.
First, an extra dimension r is introduced, representing possible depth values
for E(y) in the domain of all depth values R. We rewrite Equation (4) asXX
W + (x, ξ)O+ (x, ξ) =w(|x − y|, ξ − r)δ(r, E(y)) I(y)(5)
r∈R y∈N

where δ(r, E(y)) = 1 iff r = E(y) and 0 otherwise. Next, the term δ(r, E(y))
and the 2D input image are used to define equivalent extended representations(indicated by the + suffix) I + for input, and V + to weight the input,
V + (y, r) = δ(r, E(y))+

I (y, r) = I(y)
+

(6)
(7)

+

δ(r, E(y))I(y) = V (y, r)I (y, r).

(8)

6

Julian F. P. Kooij and Jan C. van Gemert

We see that I + (y, r) indeed has 3D coordinates (y, r) = (uy , vy , r), and similarly, V + (y, r) constitutes a 3D binary mask indicating which part of the spacecontains valid input. We write out the terms of (5) as
X X
W + (x, ξ)O+ (x, ξ) =w(|x − y|, ξ − r) V + (y, r)I + (y, r),(9)r∈R y∈N (z)X X

W + (x, ξ) =w(|x − y|, ξ − r) V + (y, r).

(10)r∈R y∈N (z)Now we can recognize the nested summation as a 3D convolution over the extended representations using the weight function w as a 3D kernel, which is thefirst step in our non-Gaussian filtering method,
W + O+ = w ⊗ V + I +W+step 1:

3D Gauss convolution+=w⊗V .

(11)(12)

If we would use the bilateral filter weight function (2), the 3D convolution
will expand the local 3D neighbourhood of a target image location into regionswith different depth values. Thereby, increasing depth distances result in lesscontribution in the convolved result. However, this kernel also blurs the original
image values, inadvertently removing details from the input before the filter F
is applied in Equation (3), even at regions with uniform depth that should not
be affected. Therefore, we consider a weighting functionα
iff dI = 0 and dE = 0,
w(dI , dE ) =(13)
G(dI ; σs ) × G(dE ; σr )otherwise,
such that as α → ∞, the weight of the local image value I + (y, r) dominates allother weights in (9) and (10) when y = x, r = ξ, and when V + (y, r) = 1. Inother words, the 3D convolution will not blur the actual input values, and notaffect the filter response F in uniform depth regions. But the Gaussian weighting
is still used in include valid input from the neighbourhood when V + (y, r) = 0,i.e. at regions with missing values in the extended representation. In practice
we do not explicitly evaluate (13), but instead produce the result of α → ∞by applying the normal kernel first to a temporary result Θ+ , and then placing
back the original values to obtain the intended result. The remaining steps to
apply our filter method are therefore,
Θ+ = W + O+ /W +
+

+ +

+

O = V I + (1 − V )Θ+Q =F ⊗O++

Q(x) = Q (x, E(x))+

step 2:element-wise division(14)

step 3:

restore valid original data

(15)

step 4:apply F at all depth layers(16)

step 5:back-project to 2D(17)

The final step, (17), back-projects the extended space to the original 2D image
space, which completes the evaluation of Equation (3). Following [15], discretizing the depth dimensions r into D depth layers results in a fast approximation,and convolutions on the depth layers can be processed in parallel. Additionally,
we downsample the image instead of expanding the spatial Gaussian kernel [15].

Depth-aware Motion Magnification

3.3

7

Depth-aware video magnification

For phase-based motion magnification [1], the non-Gaussian complex steerable
pyramid is used. The principle behind this approach is that small temporalchanges in the spatial offset of edges translates to small temporal changes in
the phases of the complex filter responses in the pyramid. Likewise, augmenting
temporal phase variations results in magnifying periodic movements in the video.
With a magnification factor M , phases φt of the pyramid components pt at time
t are augmented with respect to the temporally low-pass filtered phases φ̄ toobtain magnified pyramid phase φbt = (1 + M ) · (φt − φ̄) + φ̄.
To exploit the depth information in the complex steerable pyramid, we apply the non-Gaussian bilateral filtering from Section 3.2. Figure 2 illustrates thesteps to construct a bilateral steerable pyramid from an input grey scale anddepth image pair (I, E). First, an extended representation is created following
Equations (11)-(15). This representation is then used in the pyramid construction by applying the low-pass and complex band-pass filters of [18] to each of
the depth layers, i.e. Equation (16). The result is an extended complex steerable
pyramid. The bottom row of Figure 2 illustrates that when the extended pyramidis back-projected using the depth map E (Equation (17)), the resulting pyramid
coefficients are depth-aware: filter responses of fore- and background edges are
separated as seen by the discontinuities. In the standard pyramid, strong filter
responses from the foreground (depicting two hands) ‘leak’ into the background,especially at higher pyramid levels where the filters have larger spatial extent.
Figure 3, depicts our proposed magnification pipeline. The phase augmentation principle is applied to components in all depth layers of the extended
pyramid. However, we adapt the factor per layer with a depth-dependent function, resulting in a spatiallyvarying magnification matte M (x) = Mmax ×exp −(E(x) − µd )/(2σd2 ) , parametrized by (µd , σd , Mmax ). In the last processing step, the magnified pyramids are back-projected to 2D frames, and the matteis used to smoothly blend the magnified results of the discrete depth layers.
The recently proposed method by [2] also considers magnification of subtle
motions that occur in videos with large movements, utilizing an opacity matte toblend selected regions of a magnified frame into the original unmagnified frame.
For our comparison, we adapted their method to our setting which entails that(a) instead of using a tool to manually select a binary foreground region, theopacity matte M is used, (b) we do not perform initial video stabilization as
the camera viewpoint is already static, (c) the motion of matte M itself is notmagnified as we do not wish to magnify the motion of the occluding object, butrather that of the occluded region. Figure 3 also shows the difference between
both approaches. The baseline introduces the depth information at the last step
only. Our approach uses depth from the start to obtain and operate on a depthaware representation. Section 4.2 shall empirically compare both approaches.
3.4

Motion measurements with a bilateral pyramid

In addition to magnification, another use for complex steerable filters is to measure subtle periodic motions in the video. In [32], an image is first down scaled

8Julian F. P. Kooij and Jan C. van Gemert(a) initial data and weights(b) after spatial+depth conv.

(c) normalized(d) Bilateral steerable pyramid: first band at four pyramid levels(e) Bilateral pyramid (back-projected)(f) Standard steerable pyramid

Fig. 2: Constructing a bilateral steerable pyramid on the frame from Figure 1awith the steps in Section 3.2. (a) the input image and depth map are used
to construct a 3D image representation image I + and input weight map V +by discretizing the depth into multiple layers. (b) step 1: Both representations
are filtered in 3D (2D and the image coordinates + 1D depth coordinate). (c)step 2,3: The filtered 3D image is normalized using the filtered weights, andvalid input is restored. (d) step 4: the steerable pyramid is constructed on eachdiscrete depth layer. Here, the result is shown at various levels in the pyramidof a single orientation band. (e) step 5: The 3D representation can be backprojected to a normal pyramid using depth map, resulting in an edge-aware
steerable pyramid. Note how the responses of edges in the nearby hand remain
within the foreground region, resulting in hard edges (e.g. see red arrows). (f)In contrast, a normal steerable pyramid induces soft object edges which ‘leak‘from the foreground into the surrounding background, especially at the higher
pyramid levels (e.g. see red arrows).

Depth-aware Motion Magnification
Baseline[CVPR’15]

Compose using matte[pt ... pT] (NxT)

pt (Nx1)

image topyramid

per input frame

temporal filter ofpyramid coefficientspyramid
to image

temporal filter over all frames[p+t ... p+T] (NxDxT) Magnify using matte

Proposed9

blend

per output frame

p+t

(NxD)
temporal filtering ofpyramid coefficients

image to bilateralpyramidbilateral pyramid
to image

blend

Fig. 3: Video magnification pipelines. (Top) baseline approach from [2], which
composes a magnified and unmagnified version of each frame. The compositionis based on an opacity matte, based on the depth map. (Bottom) our approachinstead uses the depth map directly in the pyramid construction/deconstruction.

and filtered with B = 2 bands for the u and v direction, i.e. b ∈ {0◦ , 90◦ }.
Changes in phase can be translated to a local motion estimate (∆u, ∆v), as◦∆ut (u, v) = −∂u
φ0t (u, v)
φt (u, v)∂t0◦◦∆vt (u, v) = −∂v
φ90t (u, v).
φt (u, v)
∂t
90◦

(18)

In each equation, the first r.h.s. term is the inverse of a spatial derivative, andthe second term is a temporal derivative.
For a depth-aware version, we can use the bands of our bilateral complex
pyramid, using B = 2 bands, and select a particular layer l for scale. To ensurethat the spatial and temporal derivatives are depth-aware, we compute Equa+tion (18) in the extended space φl,b+(u, v, r) and obtain ∆u+tt (u, v, r), ∆vt (u, v, r).
Only afterwards are these back-projected to 2D motion maps ∆ut and ∆vt .

4

Experiments

We first evaluate against inpainting techniques. Then, we introduce a novel
RGB+Depth dataset targeting tremor analysis, which is an important medical application [10, 12]. On this dataset we compare our depth-aware motion
magnification against the state-of-the-art [2], and we show the effect of bilateralfiltering on motion measurements in fore- and background.
4.1

Filtering near missing values

Consider that we wish to convolve filter F on image I for which we have a binary
mask M whose pixel values should be ignored. This situation corresponds to theextreme case of the bilateral filter where foreground and background are far apart10

Julian F. P. Kooij and Jan C. van Gemert

such that all weights are either 0 or 1. Our approach of Section 3.2 weighs inneighborhood values in ignored regions before applying F . Here we compare ourapproach to image inpainting techniques that intent to reconstruct the regions.
Let g be a filling technique that replaces the values in the masked region,
Ig,M = g(I, M ). The filled image can then be filtered with convolutional filterF
F , resulting in Ig,M
= Ig,M ⊗ F . Ideally the masked pixels are ignored and do
not have a response at R, i.e. at the region of pixels just outside M but where
the filter still covers masked out pixels. As error measure, we therefore report
the L1 norm over the pixels in R. Let L1 (g, I, M, F ) be the norm for a particulartechnique g on image I and mask M after applying filter F , then error(g) is thetotal norm over all tested images, masks and filters, i.e.
XXXF
L1 (g, I, M, F ) =
|Ig,M
(x)|
error(g) =L1 (g, I, M, F ).
(19)
x∈R

I

M

We evaluate on a public inpainting dataset [22], which contains 17 images of
640 × 480 pixels (all images are converted to grayscale), 4 image masks, andalso provides on each image-mask pair state-of-the-art inpainting results for
Bugeau [19], Herling [20], Total Variation (TV) [21] and Xu [23]. We also compare against replacing the missing region with zeros, or the actual pixel values(an ideal inpainting algorithm). First, we use the 7 filters used in the construction of the bilateral pyramid, and tested varying the spatial parameters σs , but
found that σs = 1 performed best on the steerable pyramids features. The results in Figure 4a show that our proposed approach results in lower errors thanthe other inpainting techniques. As expected, the naive approach of replacing
the masked region with zeros results in strong responses near the mask border,
as shown by the error plots. One of the better results is obtained with TV [21],which in fact produces quite bland areas. Indeed, even using an ideal inpainting
algorithm (i.e. actual ) would introduce more unwarranted filter responses.
To test if these results generalize, we also use 64 filters from the trained VGG
convolutional neural network [33] (normalized by subtracting DC components
divided by norm). Figure 4b shows that similar results were obtained.
4.2

Depth-aware motion magnification and measurements

The next section describes our novel tremor dataset, and then the experiments on
motion magnification and motion measurement.1 Please see the accompanying
videos in the supplementary material.
RGB+D tremor dataset Tremors are manifestations of periodic movementsin the body, and assessing their properties (frequency, amplitude) is criticalfor health monitoring [11, 12]. Since in practice only few accelerometers canbe placed on the body they are typically placed where the amplitude of the1

The bilateral pyramid, depth-aware magnification code and dataset (RGB, Depth,
skeleton) can be found at https://github.com/jkooij/depthaware-momag

Depth-aware Motion Magnification

11

×104
4155

3

4145

error

error

4150

4140
4135

2.5

2

4130
ours

zeros

actual Bugeau Herling

TV

Xu

ours

zeros

actual Bugeau HerlingTVXu(a) Pyramid filters [18] (lower is better) (b) 64 ConvNet filters [33] (lower is better)(c) mask(d) fill with ours(e) fill with zeros(f) actual values(g) Bugeau [19](h) Herling [20](i) TV [21](j) Xu [23]Fig. 4: (a)-(b) Errors for steerable pyramid and ConvNet features when filtering
around a masked region. Insets show visualizations of (some of) these filters (c)one of the masks, black indicates missing values. (d)-(j) Output examples offilling methods (includes inpainting results by [22]). Red shows the evaluation
region R of Eq. (19) where the filters will be affected by missing values.

tremor is most clearly visible, e.g. on the hand and arm. Video based measuringand magnification could help discover more subtle occurrences, visualize where
tremors originate or how they move trough the body, and even make objective
tremor assessment possible without expensive hospital equipment.
We therefore collected a novel dataset with the Microsoft Kinect 2 to study
visual tremor assessment using (1) visualization, and (2) by measuring frequency.
The dataset contains 4 RGB+Depth sequences of subjects with a simulated
tremor in the hand, observed with their arms extended for several seconds. Thisis a common task in tremor assessment, intended to induce a postural tremor(i.e. a tremor which occurs due to subject trying to actively maintain a certainpose) [11]. The subjects are movement scientists, experienced in working withpatients of the neurology department at the Leiden University Medical Center.

12

Julian F. P. Kooij and Jan C. van Gemert

Using [34], we recorded the high-res RGB video (1920 × 1090, encoded in
H.264, 4:2:0 YUV), low-res depth video (512 × 450, lossless H.264, 0 - 4 meter distance mapped to 8-bit greyscale), and the Kinect 2’s estimated skeleton
data [35], all at 30 fps. Afterwards, the Kinect’s mapping API was used toproject the recorded depth frames to the RGB image space. The alignment of
the depth image with the colour images is not perfect, however: the video anddepth camera have slightly different viewpoints, the recording of the depth video
loses some quality due to the 8-bit greyscale conversion, and quick motions may
result in motion blur in the video that is not observed in the depth. For thesereasons, the depth data is pre-processed by first running a 2D median filter toremove noise, and then a 2D max filter. This extends the occluding regions andensure that foreground in the video is also fully enclosed in the nearby regionsof the depth image. The supplementary material demonstrates how depth noise,
temporal misalignment, and pre-processing affect the magnification results.
Motion magnification behind moving occluder On the first three sequences, each 91 frames (= 3 seconds), we compare our depth-aware video magnification to the the baseline approach from [2], as described in Section 3.3.
Instead of specifying specific frequencies to magnify [1], we use the mean phaseover the whole sequence as the low-pass φ̄ in order to magnify all periodic motion variations, and to avoid tuning temporal bandwidth parameters. The spatial
deviation parameter σs = 1, and depth deviation σr = 0.1 meter.
Examples of the input image region, corresponding depth map, and the usedmagnification matte (which in all cases has been set to magnify the body’s depth
range) can be seen in Figure 1a and 1b. The results of the various magnification
methods are visualized as space-time slices of Figure 1. Figure 5 shows additional
single frame comparisons. On the third sequence the clothing is very dark. Herethe intensity channel has been enhanced to more clearly show the details in the
body. The figure illustrates that compositing the magnified and original image,as in the baseline [2], results in notable artifacts in both textured on non-textured
backgrounds. We conclude that the approach in [2], which is designed to magnify
foreground under heavy camera motion, does not properly magnify backgroundbehind non-static occluders. Our approach instead suppresses the artifacts.
Motion frequency for overlapping body regions We applied the bilateralmotion measurement on the 4th and longer sequence (∼ 17 s.) to determine
the vertical motion in the hand (foreground) and chest region (background)surrounding the hand. In each frame, the measured motion is averaged over abody part mask automatically extracted using the Kinect 2’s built-in skeleton
estimate, resulting in a single temporal signal for each body region. The timealigned groundtruth data of an accelerometer on the chest demonstrates that thissequence contains two breathing cycles of about 8.5 s., see Figure 6a. When weapply a low-pass filter to only keep frequencies in the 0-0.2 Hz range, we observethat without the bilateral filter the measurements in the chest are virtually
the same as those in the hand, see dashed blue lines Figures 6b and 6c. With

13

sequence 3

sequence 2

sequence 1

Depth-aware Motion Magnification(a) magnified [1](b) CVPR’15 [2](c) ours

Fig. 5: Frames from motion magnification results on three sequences (top-tobottom Mmax = 10, 3, 5). The method of [2] blends the standard magnification
result [1] with the original frame using an opacity matte, but this does notprevent unwanted artifacts of the foreground occurring the magnified background(see red arrows), even though the (unmagnified) foreground is corrected. Our
approach using the bilateral filtered pyramid does avoids such artifacts.

bilateral filtering we obtain the same motion measurements in the foreground,but discover two periodic cycles in the background (see red lines).
The corresponding video magnification results in Figure 6d again demonstrate that the moving foreground ‘leaked’ into the background. Our bilateralpyramid yields more robust phase-based measurements in such situations.

5

Conclusions

Our work exploits depth to make motion magnification robust against moving occluders. To construct depth-aware steerable pyramids, the bilateral filter
was adapted to non-Gaussian kernels, such that filter responses can ignore local
image values at distant depth layers. We proposed a simple and efficient filling technique that is less prone to introducing additional filter responses thanJulian F. P. Kooij and Jan C. van Gemert∆ y (px)

acceleration

0.02

baselinebilateral

0

-0.02
0

2

4

6

8

10

time (seconds)

12

14

16

0.02

∆ y (px)

14baselinebilateral

0

-0.02
2

4

6

8

10

12

14

16

time (seconds)(a) GT accelerometer chest (b) Velocity signal of chest

2

4

6

8

10

12

14

16

time (seconds)(c) Velocity signal of hand(d) Depth, baseline, and our magnification results on sequence 4

Fig. 6: Measuring motion in chest behind moving hand on sequence 4. (a) Accelerometer on chest shows that there are 2 respiration cycles, each taking 9
seconds. (b) Low-passed velocity measurements on the chest, obtained with standard pyramid [1] (blue), and our bilateral pyramid (red). Our method measures
two full up-down cycles of breathing, while the baseline shows the same motion
pattern as measured in the occluding hand (c). (d) This effect is also observed
when using these pyramids for motion magnification: the baseline (middle) contains movement of the hand in the background, unlike our pyramids (right).

state-of-the-art image inpainting techniques. Depth-aware motion magnification
was demonstrated on a novel RGB+D dataset recorded with Microsoft Kinect
2 for tremor assessment, an important application in the medical domain. On
this dataset with small motions in the background behind large motions in theforeground, we show improved qualitative motion magnification results with lessvisual artifacts compared to a state-of-the-art magnification baseline, which only
exploits depth information as a final processing step. The bilateral pyramid also
resulted in improved phase-based motion measurements.
Future work includes extending the dataset with more subjects, extract moremeasures used in medical practice, and investigate application to computationally efficient Riesz pyramids [25]. Our aim is to develop the explored methods
into cheap and objective techniques to discover, monitor and classify tremors
and other movement disorders (e.g. dystonia’s) all over the body. Other uses of
the non-Gaussian bilateral filter are also considered.
Acknowledgments. This work is part of the research programme Technology
in Motion (TIM [628.004.001]), financed by the Netherlands Organisation for
Scientific Research (NWO).

Depth-aware Motion Magnification

15

References
1. Wadhwa, N., Rubinstein, M., Durand, F., Freeman, W.T.: Phase-based video
motion processing. ACM Trans. Graph. (Proceedings SIGGRAPH 2013) 32(4)(2013)
2. Elgharib, M.A., Hefeeda, M., Durand, F., Freeman, W.T.: Video magnification inpresence of large motions. In: Computer Vision and Pattern Recognition (CVPR),
2015 IEEE Conference on, IEEE (2015) 4119–4127
3. Rubinstein, M., Wadhwa, N., Durand, F., Freeman, W.T.: Revealing invisible
changes in the world. Science 339(6119) (2013) 519–519
4. Wu, H.Y., Rubinstein, M., Shih, E., Guttag, J., Durand, F., Freeman, W.T.: Eulerian video magnification for revealing subtle changes in the world. ACM Trans.
Graph. (Proceedings SIGGRAPH 2012) 31(4) (2012)
5. Davis, A., Rubinstein, M., Wadhwa, N., Mysore, G., Durand, F., Freeman, W.T.:
The visual microphone: Passive recovery of sound from video. ACM Transactions
on Graphics (Proc. SIGGRAPH) 33(4) (2014) 79:1–79:10
6. Balakrishnan, G., Durand, F., Guttag, J.: Detecting pulse from head motions invideo. In: Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition. (2013) 3430–3437
7. Aziz, N.A., Tannemaat, M.R.: A microscope for subtle movements in clinicalneurology. Neurology 85(10) (2015) 920–920
8. Amir-Khalili, A., Peyrat, J.M., Abinahed, J., Al-Alao, O., Al-Ansari, A.,
Hamarneh, G., Abugharbieh, R.: Auto localization and segmentation of occluded
vessels in robot-assisted partial nephrectomy. In: Medical Image Computing andComputer-Assisted Intervention–MICCAI 2014. Springer (2014) 407–414
9. Davis, A., Bouman, K.L., Chen, J.G., Rubinstein, M., Durand, F., Freeman, W.T.:
Visual vibrometry: Estimating material properties from small motions in video. In:
Computer Vision and Pattern Recognition (CVPR), 2015 IEEE Conference on,IEEE (2015) 5335–5343
10. Bain, P.G.: Tremor. Parkinsonism & related disorders 13 (2007) S369–S374
11. Deuschl, G., Bain, P., Brin, M.: Consensus statement of the movement disordersociety on tremor. Movement Disorders 13(S3) (1998) 2–23
12. Schwingenschuh, P., Katschnig, P., Seiler, S., Saifee, T.A., Aguirregomozcorta, M.,
Cordivari, C., Schmidt, R., Rothwell, J.C., Bhatia, K.P., Edwards, M.J.: Moving
toward laboratory-supported criteria for psychogenic tremor. Movement Disorders
26(14) (2011) 2509–2515
13. Fattal, R.: Edge-avoiding wavelets and their applications. ACM Trans. Graph.
28(3) (2009) 1–10
14. He, K., Sun, J., Tang, X.: Guided image filtering. Pattern Analysis and MachineIntelligence, IEEE Transactions on 35(6) (2013) 1397–1409
15. Paris, S., Durand, F.: A fast approximation of the bilateral filter using a signal
processing approach. International journal of computer vision 81(1) (2009) 24–52
16. Tomasi, C., Manduchi, R.: Bilateral filtering for gray and color images. In: Computer Vision, 1998. Sixth International Conference on, IEEE (1998) 839–846
17. Freeman, W.T., Adelson, E.H.: The design and use of steerable filters. IEEE
Transactions on Pattern Analysis & Machine Intelligence (9) (1991) 891–906
18. Simoncelli, E.P., Freeman, W.T.: The steerable pyramid: A flexible architecture
for multi-scale derivative computation. In: icip, IEEE (1995) 3444
19. Bugeau, A., Bertalmı́o, M., Caselles, V., Sapiro, G.: A comprehensive framework
for image inpainting. Image Processing, IEEE Transactions on 19(10) (2010) 2634–
2645

16

Julian F. P. Kooij and Jan C. van Gemert

20. Herling, J., Broll, W.: Pixmix: A real-time approach to high-quality diminishedreality. In: Mixed and Augmented Reality (ISMAR), 2012 IEEE International
Symposium on, IEEE (2012) 141–150
21. Getreuer, P.: Total variation inpainting using split bregman. Image Processing On
Line 2 (2012) 147–157
22. Tiefenbacher, P., Bogischef, V., Merget, D., Rigoll, G.: Subjective and objective
evaluation of image inpainting quality. In: Image Processing (ICIP), 2015 IEEE
International Conference on, IEEE (2015) 447–451
23. Xu, Z., Sun, J.: Image inpainting by patch propagation using patch sparsity. Image
Processing, IEEE Transactions on 19(5) (2010) 1153–1165
24. Liu, C., Torralba, A., Freeman, W.T., Durand, F., Adelson, E.H.: Motion magnification. ACM transactions on graphics (TOG) 24(3) (2005) 519–526
25. Wadhwa, N., Rubinstein, M., Durand, F., Freeman, W.T.: Riesz pyramids for fastphase-based video magnification. In: Computational Photography (ICCP), 2014
IEEE International Conference on, IEEE (2014) 1–10
26. Bai, J., Agarwala, A., Agrawala, M., Ramamoorthi, R.: Selectively de-animating
video. ACM Trans. Graph. 31(4) (2012) 66–1
27. Malleson, C., Bazin, J.C., Wang, O., Bradley, D., Beeler, T., Hilton, A., SorkineHornung, A.: Facedirector: Continuous control of facial performance in video. In:
Proceedings of the IEEE International Conference on Computer Vision. (2015)
3979–3987
28. Shankar Nagaraja, N., Schmidt, F.R., Brox, T.: Video segmentation with just a
few strokes. In: Proceedings of the IEEE International Conference on Computer
Vision. (2015) 3235–3243
29. Bai, J., Agarwala, A., Agrawala, M., Ramamoorthi, R.: User-assisted video stabilization. Computer Graphics Forum 33(4) (2014) 61–70
30. Efros, A.A., Leung, T.K.: Texture synthesis by non-parametric sampling. In: Computer Vision, 1999. The Proceedings of the Seventh IEEE International Conference
on. Volume 2., IEEE (1999) 1033–1038
31. Liu, C., Torralba, A., Freeman, W.T., Durand, F., Adelson, E.H.: Motion magnification. ACM transactions on graphics (TOG) 24(3) (2005) 519–526
32. Chen, J.G., Wadhwa, N., Cha, Y.J., Durand, F., Freeman, W.T., Buyukozturk,
O.: Modal identification of simple structures with high-speed video using motion
magnification. Journal of Sound and Vibration 345 (2015) 58–71
33. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale
image recognition. arXiv preprint arXiv:1409.1556 (2014)
34. Kooij, J.F.P.: SenseCap: synchronized data collection with Microsoft Kinect2 andLeapMotion. In: Proceedings of the 22nd ACM international conference on Multimedia. ACM, 2016, ACM (to appear)
35. Shotton, J., Sharp, T., Kipman, A., Fitzgibbon, A., Finocchio, M., Blake, A.,
Cook, M., Moore, R.: Real-time human pose recognition in parts from single
depth images. Communications of the ACM 56(1) (2013) 116–124