Supplementary Material
Depth-aware Motion Magnification
Julian F. P. Kooij1,2 and Jan C. van Gemert1,2
1

Delft University of Technology, Delft, The Netherlands{J.F.P.Kooij,J.C.vanGemert}@tudelft.nl
2Leiden University Medical Center, Leiden, The Netherlands

This document provides additional experiments for our paper [1].
– Section 1 investigates the effect of depth image quality and alignment onmotion magnification.
– Section 2 demonstrates other uses of the non-Gaussian bilateral filter, unrelated to motion magnification.

1Effect of depth image quality and alignment

The paper describes that the depth images are pre-processed to remove artifacts.
The pre-processing step consists of a applying 2D median and 2D max filter on
the depth images. In this section we investigate how pre-processing affects thealignment with the color image, and how it removes noise. We first look at theeffect of pre-processing on a single frame under various conditions, and then at
how it affects the motion magnification results.
1.1

Single frame depth and color image alignment

Figures 1 and 2 compare the effect of pre-processing on a single frame fromsequence 2 in our paper under various conditions. Each depth/color image pairdemonstrates how the depth aligns with the color image by highlighting in red
in the color image the ‘foreground’ region at a depth distance below 2.5 meters.
Pairs in the top row use the original depth input, the bottom row the depth
after pre-processing. The following conditions are shown:
Normal depth image First, 1a shows the original depth image. It contains
some depth outliers (e.g. bright yellow blocks near armpit) and missing depthvalues (e.g. around the arm). These are natural effect of the depth sensor, butartifacts can also occur due to the depth-to-image projection, and upscaling of
the low-res depth image to the color HD format. Without pre-processing (seetop row), the foreground region is affected by noise and outliers and fits too tightaround the arm, hence various border pixels of the arm fall in the background.
The proposed pre-processing step (bottom row) removes the noise and fills mostof the holes. The foreground region is also slightly enlarged such that the moving
left arm is now fully contained.

2Julian F. P. Kooij and Jan C. van Gemert

Additional depth noise Then, 1b and 1c demonstrate the effect of adding
zero-mean normally distributed noise,  ∼ N (0, σn ), to the original depth image,using low σn = 50mm, and high σn = 100mm variance respectively. However,the bottom rows show that the proposed pre-processing yields almost identicalresults as the original input with respect to the moving arm.
Additional depth outliers Figures 2a and 2b consider additional extreme
values. Since normally the depth-to-image projection upscales the depth image,an individual outlier can create a large block in the mapped depth image. We
therefore add artificial outliers as 4×4 square regions with extreme values. In 2a,about ∼ 1% of depth pixels is made an outlier, and in 2b this number is increasedto ∼ 20%.
We can see that in the first case, pre-processing still removes the moderate amount of ouliers, thus results remain mostly unaffected. But, the secondcase shows that if there are too many outliers close together, the proposed preprocessing can result in fewer but even larger regions in the filtered result.
Temporal misalignment Finally, we consider the case that the depth andcolor frames are misaligned. Each depth frame t is therefore now combined withcolor frame t + 3, which increases the risk of the foreground inadvertently moving into the magnified region. At ∼ 30Hz, 3 frames corresponds to about 1/10
seconds. As can be seen in the top part of Figure 2c, the finger tips (which tremble with largest amplitude) are not contained in the foreground region anymore
without pre-processing. The bottom part of the figure shows however that by
expanding the foreground in the pre-processing step the hand is still fully contained, in this case.
We conclude that median filtering suffices to remove the noise in most conditions, and foreground expansion using the 2D max filter increases the robustness
to spatial misalignment resulting from depth-to-color mapping, upscaling, andtemporal offsets. Of course, if the motion of the foreground is too large, thenthe spatial misalignment resulting from a frame offset could be too big for thesimple foreground expansion.
1.2

Effect on depth-aware motion magnification

Next, we magnify the same depth region of sequence 2 as described in the paper.
Figure 3 illustrates again the temporal effects on a single vertical slice of pixels
containing the trembling left arm (foreground which should remain unmagnifed),
and the background which should be magnified. For reference, Figures 3a shows
again the selected spatial slice, 3b the magnification matte, and 3c is the spacetime slice of the original unmagnified video.
Figures 3d-3i demonstrate the effect of depth-aware motion magnificationon the slice under the conditions explored in the previous section. For eachSupplementary Material: Depth-aware Motion Magnification

3condition, the left space-time slice show the result without pre-processing, andthe right slice with pre-processing.
As we found in the previous section, the pre-processing step ensures that theartifacts are mitigated in most conditions. In our test cases, the only exception
is the condition containing 20% additional depth outliers, Figure 3h. As wasshown in Figure 2b, these outliers are not fully removed by the used filtering
strategy and the presence of these artifacts results in irregular and unrealistic
magnification patterns, see 3h (right).

2

Other uses of non-Gaussian bilateral filtering

In the paper, the non-Gaussian bilateral filtering approach has only been demonstrated in the context of motion magnification. This section presents examples
of some other possible image filtering applications.
2.1

Suppressing object edges from depthFirst, we study extracting texture edges in a single RGB+D image taken froma frame of the Sintel dataset [2]. The input contains various occluding objects
and an accurate depth map such that no depth pre-processing is needed, see theimage in Figure 4a and its depth channel in Figure 4b. We consider using thedepth channel to guide a bilateral filter on the color image.
While standard bilateral filtering with a Gaussian filter removes texture edges
but keeps edges at object boundaries (see Figure 4c), our non-Gaussian bilateralfilter can instead be used to detect texture edges while suppressing edges at theobject boundaries. To demonstrate this, we filter the intensity of the RGB image
with a steerable filter, and visualize in Figures 4d-4f the edge orientation (colorcoded, see color wheel in upper-left corner) and magnitude (saturation).
The effect of the depth on the filtered results can be adjusted by altering thedepth range σr . In the extreme case with large σr , the non-Gaussian bilateralfilter reduces to a standard non-Gaussian filter which does not use depth information, see Figure 4d. But, as σr decreases, the filter increasingly ignores intensity
differences that coincide with depth differences, see Figure 4e (σr = 1m → 10
depth layers) and 4f (σr = 0.1m → 100 depth layers). Note how object edges inthe black encircled regions are present in the standard filter, but removed by ourbilateral non-Gaussian filter. On the other hand, texture edges (such as thoseon the bamboo and face in the white encircled regions) are kept since the depth
around these edges is nearly uniform.
2.2

Non-Gaussian bilateral filtering with intensity only

Instead of relying on a related depth map to filter an input image, one could also
consider using the intensity channel of the input itself as illustrated in Figure 5.
Figure 5a shows a grayscale input image. While the original bilateral filter
performs edge-aware smoothing (Figure 5b), the non-Gaussian bilateral filter can4Julian F. P. Kooij and Jan C. van Gemert

be used as an edge detector that ignores strong edges, emphasizing microtexturesinstead. On the input image, we build a normal steerable pyramid, and ourproposed bilateral steerable pyramid using the intensity image itself to build thedepth layers.
As in the previous subsection, we visualize the extract edge orientation, but
now do not encode magnitude with saturation anymore as only low magnitudeedges are kept.
In Figure 5c with the normal steerable pyramid, we see that large intensity
differences dominate the edge orientation in a large spatial extent (determinedby the size of the filter). For example, where the dark body outline contrasts
against the lighter flat background, the computed orientation does not show anyof the subtle details of the textures (e.g. inside the body, legs). The affected local
neighborhood grows in size at subsequent pyramid levels, see Figure 5d.
In contrast, our bilateral non-Gaussian filter suppresses the influence of thelarge intensity differences, uncovering the texture orientation within regions evenclose to the object border, see Figures 5e and 5f. In Figures 5e for instance, one
can see that more texture orientations in the legs and elbows than in 5c. Likewise,
one can see more details in the body in 5f compared to 5d.

References
1. Kooij, J.F.P., van Gemert, J.C.: Depth-aware motion magnification. In: European
Conference on Computer Vision (ECCV), 2016, Springer (to appear)
2. Butler, D.J., Wulff, J., Stanley, G.B., Black, M.J.: A naturalistic open source movie
for optical flow evaluation. In A. Fitzgibbon et al. (Eds.), ed.: European Conf. on
Computer Vision (ECCV). Part IV, LNCS 7577, Springer-Verlag (October 2012)
611–625

5with depth pre-processing

without depth pre-processing

Supplementary Material: Depth-aware Motion Magnification(a) proposed(b) white noise σn = 50mm (c) white noise σn = 100mmFig. 1: Depth/color image pairs to demonstrate alignment, where red regions incolor image corresponds to depth distance < 2.5 meter. Each column corresponds
to different depth image quality, where top row is without depth pre-processing,and bottom row with. Comparing the bottom row, pre-processing ensures thatthe visual fore- and background are separably by depth in all cases.
(a) Original depth image as used in the paper. (b) With additional zero-mean
Gaussian noise added to the depth σn = 50mm. (c) With more zero-mean Gaussian noise added to the depth σn = 100mm.

Julian F. P. Kooij and Jan C. van Gemert

with depth pre-processing

without depth pre-processing

6(a) 1% outliers(b) 20% outliers(c) temporal offset 3 frames

Fig. 2: Depth/color image pairs to demonstrate alignment, where red regions incolor image corresponds to depth distance < 2.5 meter. Each column corresponds
to different depth image quality, where top row is without depth pre-processing,and bottom row with. Comparing the bottom row, pre-processing ensures thatthe visual fore- and background are separably by depth, except in extreme caseswith too many outliers.
(a) With about 1% of depth pixels set to extreme values, pre-processing still
removes all artifacts. (b) With about 20% outliers pre-processing cannot remove
all artifacts. (c) With temporal misalignment of 3 frames between depth andcolor image. Without pre-processing the fingertips of the trembling hand crossthe foreground border.

Supplementary Material: Depth-aware Motion Magnification(a) sample region (b) magnify matte7(c) no magnification(d) proposed method (left: without, right: with pre-processing)(e) depth white noise σ = 50mm (left: without, right: with pre-processing)(f) depth white noise σ = 100mm (left: without, right: with pre-processing)(g) 1% depth outliers (left: without, right: with pre-processing)(h) 20% depth outliers (left: without, right: with pre-processing)(i) temporal offset 3 frames (left: without, right: with pre-processing)Fig. 3: Pre-processing the depth input ensures that motion magnification remains
large unaffected, even with additional artificial depth errors and temporal offsets.

8Julian F. P. Kooij and Jan C. van Gemert(a) input image “bamboo 2”(b) depth image(c) standard bilateral filter(d) standard non-Gaussian filter(e) bilateral non-Gaussian filter,(f) bilateral non-Gaussian filter,σr = 1 meterσr = 0.1 meter

Fig. 4: (a) input frame from Sintel dataset [2], and (b) its depth frame. (c)A standard bilateral filter for edge-aware Gaussian blurring. (d) A standard
steerable filter. Edges are color coded: orientation by color hue (see color-key in
top-left corner), edge magnitude by saturation. (e), (f) Our method suppresses
strong image edges that coincide with large depth differences (see black encircledregions), while keeping edges within objects where depth differences are small(see white encircled regions). The sensitivity for depth increases as σr decreases.

Supplementary Material: Depth-aware Motion Magnification

9(a) input image, “cameraman” (b) standard bilateral Gauss
blur(c) standard pyramid, level 2(d) standard pyramid, level 3(e) bilateral pyramid, level 2(f) bilateral pyramid, level 3(ours)(ours)

Fig. 5: Using intensity instead of depth for non-Gaussian bilateral filtering. (a)input image. (b) applying a standard bilateral filter blurs the image while maintaining strong edges. (c), (d) edge orientations at various levels (scales) in astandard steerable pyramid (upper-left corner displays the orientation color-key).
(e), (f) in our bilateral steerable pyramid, guided by the intensity image itself,
detailed orientation are visible of textures, even nearby strong object edges.