Gene Cheung
National Institute of Informatics
2nd October, 2013

3D visual communication:
media representation, transportand rendering.
MMSP'13 Plenary 10/02/2013

1

Acknowledgement
Collaborators:• Yu Mao (NII, Japan)• Wei Hu, Wenxiu Sun, Wei Dai, Prof. Oscar Au (HKUST, HK)• Prof. Antonio Ortega (USC, USA)
• Dr. Dinei Florencio, Cha Zhang, Phil Chou (MSR)• Xiaoyu Xiu, Hadi Hadizadeh, Prof. Jie Liang, Prof. Ivan Bajic (SFU, Canada)• Prof. Ngai-Man Cheung (SUTD, Singapore)• Prof. Bruno Machiavello, Camilo Dorea, Mintsu Hung (UofBrasilia, Brazil)• Dr. Wai-tian Tan (HPL, now Cisco)

Slides Contributors:• Dr. Philipp Merkle (HHI, Germany)• Prof. Minh Do (UIUC, USA)
• Prof. Patrick Le Callet (UofNantes, France)• Dr. Thomas Maugey, Prof. Pascal Frossard (EPFL, Switzerland)• Mr. Hiroshi Sankoh (KDDI Labs, Japan)
MMSP'13 Plenary 10/02/2013

2

Presentation Outline• Background & Motivation (3D, not your mother’s 2D)• 3D Video representation / coding:• Depth map coding• HEVC tools for depth maps
• Graph-based Transform (GBT) for depth maps

• Depth map denoising• Denoising + compression?
• Why code depth images?

• 3D Video streaming:• Video compression with flexible decoding for interactive streaming• Loss-resilient texture-plus-depth video streaming (skip)• 3D view synthesis:• Robust view synthesis for free viewpoint video• Synthesized image interpolation for z-dimension camera movement
MMSP'13 Plenary 10/02/2013

3

Presentation Outline• Background & Motivation (3D, not your mother’s 2D)• 3D Video representation / coding:• Depth map coding• HEVC tools for depth maps
• Graph-based Transform (GBT) for depth maps

• Depth map denoising• Denoising + compression?
• Why code depth images?

• 3D Video streaming:• Video compression with flexible decoding for interactive streaming• Loss-resilient texture-plus-depth video streaming (skip)• 3D view synthesis:• Robust view synthesis for free viewpoint video• Synthesized image interpolation for z-dimension camera movement
MMSP'13 Plenary 10/02/2013

4

Biography (how I got started in 3D)• MS from UC Berkeley in EECS in 1998.
• Thesis: Joint source / channel coding for wireless video.

• PhD from UC Berkeley in EECS in 2000.
• Thesis: Computation / memory / distortion tradeoff in signal compression.

• Senior researcher in HP Labs Japan from 2000 to 2009.
• Topic 1: 2D video coding & streaming optimization (2000~2007).
• Topic 2: Interactive multiview video, w/ Prof. Ortega (2007~).

• Faculty in NII from 11/2009 to now.
• Topic 1: Immersive visual communication:• Free viewpoint video coding, streaming, view synthesis.

• Topic 2: Visual saliency & gaze analysis.
MMSP'13 Plenary 10/02/2013

5

Biography (how I got started in 3D)• MS from UC Berkeley in EECS in 1998.

2D video• Thesis: Joint source / channel coding for wireless video.
Communication(12 yrs)• PhD from UC Berkeley in EECS in 2000.

• Thesis: Computation / memory / distortion tradeoff in signal compression.

• Senior researcher in HP Labs Japan from 2000 to 2009.
• Topic 1: 2D video coding & streaming optimization (2000~2007).
• Topic 2: Interactive multiview video, w/ Prof. Ortega (2007~).

• Faculty in NII from 11/2009 to now.
• Topic 1: Immersive visual communication:• Free viewpoint video coding, streaming, view synthesis.

• Topic 2: Visual saliency & gaze analysis.
MMSP'13 Plenary 10/02/2013

5

Biography (how I got started in 3D)• MS from UC Berkeley in EECS in 1998.

2D video• Thesis: Joint source / channel coding for wireless video.
Communication(12 yrs)• PhD from UC Berkeley in EECS in 2000.

• Thesis: Computation / memory / distortion tradeoff in signal compression.

• Senior researcher in HP Labs Japan from 2000 to 2009.
• Topic 1: 2D video coding & streaming optimization (2000~2007).
• Topic 2: Interactive multiview video, w/ Prof. Ortega (2007~).

• Faculty in NII from 11/2009 to now.
3D video• Topic 1: Immersive visual communication:
Communication(7 yrs)• Free viewpoint video coding, streaming, view synthesis.

• Topic 2: Visual saliency & gaze analysis.
MMSP'13 Plenary 10/02/20135Video Communication: 2D to 2.5D to 3D• 2D Video• 1 capturing camera @ sender.
• 1 2D display @ receiver (non-interactive).

senderreceiverVideo Communication: 2D to 2.5D to 3D• 2D Video• 1 capturing camera @ sender.
• 1 2D display @ receiver (non-interactive).

senderreceiversenderreceiver• 2.5D Video (stereoscopic)• 2 capturing cameras @ sender.
• 1 stereoscopic display @ receiver (non-interactive).

Video Communication: 2D to 2.5D to 3D• 2D Video• 1 capturing camera @ sender.
• 1 2D display @ receiver (non-interactive).

senderreceiversenderreceiver• 2.5D Video (stereoscopic)• 2 capturing cameras @ sender.
• 1 stereoscopic display @ receiver (non-interactive).
• 3D Video (multiview, free viewpoint)• Multiple capturing cameras @ sender.
• 1 2D / stereoscopic display @ receiver (interactive):• Receiver observes subset of high dimension
media available @ sender!

receiversender FB channelVideo Communication: 2D to 2.5D to 3D• 2D Video• 1 capturing camera @ sender.
• 1 2D display @ receiver (non-interactive).

senderreceiversenderreceiver• 2.5D Video (stereoscopic)• 2 capturing cameras @ sender.
• 1 stereoscopic display @ receiver (non-interactive).
• 3D Video (multiview, free viewpoint)• Multiple capturing cameras @ sender.
receiver• 1 2D / stereoscopic display @ receiver (interactive):• Receiver observes subset of high dimension
media available @ sender!

sender FB channel

texture map

depth mapVideo Communication: 2D to 2.5D to 3D• 2D Video• 1 capturing camera @ sender.
• 1 2D display @ receiver (non-interactive).

senderreceiversenderreceiverMultiview Camera Setup @ Nagoya U.

• 2.5D Video (stereoscopic)• 2 capturing cameras @ sender.
• 1 stereoscopic display @ receiver (non-interactive).
• 3D Video (multiview, free viewpoint)• Multiple capturing cameras @ sender.
receiver• 1 2D / stereoscopic display @ receiver (interactive):• Receiver observes subset of high dimension
media available @ sender!

sender FB channel

texture map

depth mapVideo Communication: 2D to 2.5D to 3D• 2D Video• 1 capturing camera @ sender.
• 1 2D display @ receiver (non-interactive).

senderreceiversenderreceiver• 2.5D Video (stereoscopic)• 2 capturing cameras @ sender.
• 1 stereoscopic display @ receiver (non-interactive).
• 3D Video (multiview, free viewpoint)• Multiple capturing cameras @ sender.
receiver• 1 2D / stereoscopic display @ receiver (interactive):• Receiver observes subset of high dimension
media available @ sender!

sender FB channel

texture map

depth map

Multiview Video Streaming• Interactive view-switches among captured camera viewpoints.

MMSP'13 Plenary 10/02/2013

7

Free Viewpoint Video Streaming• Interactive view-switches to any virtual camera viewpoints.

*Courtesy of KDDI Laboratories, Japan

MMSP'13 Plenary 10/02/2013

8

Immersive Communication• Goal: ultra-realistic networked visual communication.
• Application: HQ teleconferencing, tele-medicine,distance learning.

• Features:
1. Gaze-corrected view.

xz

2. Motion Parallax: fast, smooth interactive viewswitching triggered by tracked observer’s head
position.
3. Low-delay, loss-resilient network transmission.

MMSP'13 Plenary 10/02/2013

9

Potential Impact• Immersive Communication ≠ Skype calls!
• Non-verbal means (postures, gestures) are important.
• Eye-contact is important.
• Depth perception via motion parallax.

• Substitute for face-to-face meetings.
• Reduce travel cost, improve productivity.
• Reduce carbon footprints.
• Example apps: HQ teleconferencing, tele-medicine.

• Enhance Virtual Reality is 1 of 14 grand challenges chosen
by National Academy of Engineering for 21st century.
• Treatment of social anxieties, phobias, children autism.
• Training & teaching: virtual surgeries, etc.
MMSP'13 Plenary 10/02/2013

10

Presentation Outline• Background & Motivation (3D, not your mother’s 2D)• 3D Video representation / coding:• Depth map coding• HEVC tools for depth maps
• Graph-based Transform (GBT) for depth maps

• Depth map denoising• Denoising + compression?
• Why code depth images?

• 3D Video streaming:• Video compression with flexible decoding for interactive streaming• Loss-resilient texture-plus-depth video streaming (skip)• 3D view synthesis:• Robust view synthesis for free viewpoint video• Synthesized image interpolation for z-dimension camera movement
MMSP'13 Plenary 10/02/2013

11

3D Video Representation• Texture + depth maps from 1 or morecamera viewpoints.
• Texture map: color image like RGB.
• Depth map: per-pixel distance bt’n captured
objects in 3D scene & capturing camera.

• Synthesis of intermediate views via depth-imagebased rendering (DIBR).
• Computation-efficient.
• Unlike model-based approach, complexity notscene-dependent.
texture map

depth map

MMSP'13 Plenary 10/02/2013

12

3D Video Representation• Texture + depth maps from 1 or morecamera viewpoints.
• Texture map: color image like RGB.
• Depth map: per-pixel distance bt’n captured
objects in 3D scene & capturing camera.

• Synthesis of intermediate views via depth-imagebased rendering (DIBR).
• Computation-efficient.
• Unlike model-based approach, complexity notscene-dependent.
texture map

depth map

MMSP'13 Plenary 10/02/2013

12

Fraunhofer
Heinrich Hertz Institute

Coding of depth maps

Coding of depth or disparity maps
• Inter-view and additionallyinter-component correlations are
exploited by prediction-based coding• Tools:− Disparity-compensated prediction for
dependent view
− Depth modeling modes− Motion parameter inheritance− Synthesized view distortion
optimization

view 0view 1© Müller, “3D High-Efficiency Video Coding for Multi-View Video and Depth Data,” IEEE Transactions on Image Processing, Sep. 2013.
K.
08.10.2013
Courtesy of Fraunhofer HHI, Berlin, Germany.

13

Depth Modeling Modes

Fraunhofer
Heinrich Hertz InstituteDepth map properties:•

Sharp edges
representing object
borders•

Large areas of slowly
varying values
representing object
areas

New intra prediction modes• Representation of depth edges• Partition block into two regions with constant sample •values• Prediction based on co-located texture block• Optional transform coding of residual

Edges in depth maps arecorrelated with edges invideo pictures© Merkle et al., “Coding of depth signals for 3D video using wedgelet block segmentation with residual adaptation,“ ICME 2013.
P.
Courtesy of Fraunhofer HHI, Berlin, Germany.

08.10.201314

Depth Modeling Modes – Intra Wedgelet

Fraunhofer
Heinrich Hertz InstituteExplicit Wedgelet signaling•

Wedgelet partition of current block is estimatedat the encoder by minimum distortion searchusing original depth signal•

Pre-defined lists of Wedgelet patterns forfast search and efficient signaling

Intra-predicted Wedgelet partitioning•

Separation line for current block is predicted fromneighboring blocks•

Prediction from Wedgelet block bycontinuing separation line in current block•

Prediction from conventional intra block by
combining direction and maximum slope point•

Transmission of line end refinement© Merkle et al., “Coding of depth signals for 3D video using wedgelet block segmentation with residual adaptation,“ ICME 2013.
P.
Courtesy of Fraunhofer HHI, Berlin, Germany.

08.10.201315

Depth Modeling Modes – Inter-component

Fraunhofer
Heinrich Hertz InstituteInter-component prediction of Wedgelet•

Wedgelet partition of current block is predictedfrom co-located block of reconstructed video
picture by minimum distortion search•

Disable mode when co-located texture block
has insignificant texture information(using mean absolute difference)

Inter-component prediction of Contour•

Contour partition of current block is predictedfrom co-located block of reconstructed video
picture by thresholding segmentation•

Disable mode when co-located texture block
has insignificant texture information© Merkle et al., “3D video: Depth coding based on inter-component prediction of block partitions,“ PCS 2012.
P.
Courtesy of Fraunhofer HHI, Berlin, Germany.

08.10.2013

16

Motion Parameter Inheritance

Fraunhofer
Heinrich Hertz InstituteInheritance of partitioning and motion data from co-located video block• Block-adaptive signalling• Use merge syntax: Insert as first entry in candidate list
• Only supported if complete co-located video block is inter-codedreference picture

current
picture© Winken et al., “Motion vector inheritance for high efficiency 3D video plus depth coding,“ PCS 2012.
M.
Courtesy of Fraunhofer HHI, Berlin, Germany.

inheritance ofpartitioning andmotion datatransmission of newpartitioning andmotion data

08.10.201317

Fraunhofer
Heinrich Hertz Institute

Encoder control

Synthesized view distortion optimization

video pictures…
3D video
decoder…
depth maps
camera parameterssynthesisofintermediateviews

N-view
display
3D receiverstereodisplay© Tech et al., “3D video coding using the synthesized view distortion change,“ PCS 2012.
G.
Courtesy of Fraunhofer HHI, Berlin, Germany.

08.10.201318

Fraunhofer
Heinrich Hertz Institute

Encoder control

Synthesized view distortion optimization• Coding artifacts in depth data are only indirectly perceivable in synthesizedvideo data
• Decoded depth map itself is not visible
video pictures…
3D video
decoder…
depth maps
camera parameterssynthesisofintermediateviews

N-view
display
3D receiverstereodisplay© Tech et al., “3D video coding using the synthesized view distortion change,“ PCS 2012.
G.
Courtesy of Fraunhofer HHI, Berlin, Germany.

08.10.2013

18

Encoder control

Fraunhofer
Heinrich Hertz Institute

Synthesized view distortion optimization• Coding artifacts in depth data are only indirectly perceivable in synthesizedvideo data
• Decoded depth map itself is not visible
 Consider errors in synthesized views in encoder© Tech et al., “3D video coding using the synthesized view distortion change,“ PCS 2012.
G.
Courtesy of Fraunhofer HHI, Berlin, Germany.

08.10.201318

Presentation Outline• Background & Motivation (3D, not your mother’s 2D)• 3D Video representation / coding:• Depth map coding• HEVC tools for depth maps
• Graph-based Transform (GBT) for depth maps

• Depth map denoising• Denoising + compression?
• Why code depth images?

• 3D Video streaming:• Video compression with flexible decoding for interactive streaming• Loss-resilient texture-plus-depth video streaming (skip)• 3D view synthesis:• Robust view synthesis for free viewpoint video• Synthesized image interpolation for z-dimension camera movement
MMSP'13 Plenary 10/02/2013

19

Graph-Based Transform (GBT)graph connectivity•

An adaptive transform that avoids filtering across edges•

Equal to KLT under some specific statistic model whenrepresents pixel correlation

G. Shen, W.-S. Kim, S.K. Narang, A. Ortega, J. Lee, and H. Wey, “Edge-adaptive transforms for efficient depth map
coding,” IEEE Picture Coding Symposium, Nagoya, Japan, December 2010.
D. Shuman, S.K. Narang, P. Frossard, A. Ortega, and P. Vandergheynst, “The Emerging Field of Signal Processing on Graphs,”IEEE Signal Processing Magazine, pp.83-98, May 2013.
20

Depth Map Coding using Graph-Based Transform•

Depth map: Piecewise Smoothness (PWS)•

GBT gives compact compression for depth maps
- sparse transform domain representation(avoid filtering across edges)- simple transform description(the statistics of depth maps is simple: pixel correlation is either 0 or 1)•

ExampleGBT

DCT

a 4x4 block•

Complexity issue: real-time eigen-decomposition, only operate on small blocks

21

Multi-resolution Graph-based Transform• Objective: Encode large blocks with GBT in low complexity• Key Idea
- Encode sharp edges in original high resolution:
preserve sharpness- Encode smooth surfaces in low-pass-filtered anddown-sampled low resolution:
save bits & reduce complexity- At the decoder, the LR surfaces are up-sampled and interpolated while respecting
the losslessly encoded HR edges.
W. Hu, G. Cheung, X. Li and O. Au, "Depth Map Compression using Multi-resolution Graph-based Transform for Depth-image-based
Rendering," IEEE International Conference on Image Processing, Orlando, FL, September 2012.

22

Experimentation
Experimental Setup- H.264/AVC Reference Software JM17.1- Test images: Middlebury multiview image sets
- QP: 24, 28, 32, 36- Distortion metric: PSNR of synthesized views
LR-DCT

reduce bitrate by 68% compared to HR-DCT
and 55% compared to HR-GBT

MR-GBT

23

Presentation Outline• Background & Motivation (3D, not your mother’s 2D)• 3D Video representation / coding:• Depth map coding• HEVC tools for depth maps
• Graph-based Transform (GBT) for depth maps

• Depth map denoising• Denoising + compression?
• Why code depth images?

• 3D Video streaming:• Video compression with flexible decoding for interactive streaming• Loss-resilient texture-plus-depth video streaming (skip)• 3D view synthesis:• Robust view synthesis for free viewpoint video• Synthesized image interpolation for z-dimension camera movement
MMSP'13 Plenary 10/02/201324

Video Enhancement for Depth Camera Problem: Depth images
from ToF camera are lowresolution, blurred, noisy
 Setting: Given a noisy, lowresolution depth map DL anda registered noise-free,high-solution color image I Estimate DH*Courtesy of Prof. M. Do, UIUC, USA25

Proposed Method: Weighted Mode Filtering Generating joint histogram• g(p): color value at pixel p• f(p): depth value at pixel p• fG(p): enhanced depth value at pixel p• GI, GS, Gr: Gaussian function
color Gaussian

pixel pspatial Gaussianerr Gaussian

dth binneighbors of pixel p

D. Min, J. Lu, and M. N. Do, “Depth video enhancement based on weighted mode filtering,” IEEE Trans. on
Image Processing, 2012.
*Courtesy of Prof. M. Do, UIUC, USA
26

Result Comparison

Up-sampling results for low-quality depth image (from ‘Mesa
Imaging SR4000’, 176x144) with corresponding color image(from ‘Point Grey Flea’, 1024x768).
*Courtesy of Prof. M. Do, UIUC, USA

Presentation Outline• Background & Motivation (3D, not your mother’s 2D)• 3D Video representation / coding:• Depth map coding• HEVC tools for depth maps
• Graph-based Transform (GBT) for depth maps

• Depth map denoising• Denoising + compression?
• Why code depth images?

• 3D Video streaming:• Video compression with flexible decoding for interactive streaming• Loss-resilient texture-plus-depth video streaming (skip)• 3D view synthesis:• Robust view synthesis for free viewpoint video• Synthesized image interpolation for z-dimension camera movement
MMSP'13 Plenary 10/02/2013

28

Depth Processing in 3D Video
Communication• Pipeline of 3D Video Communication System• At encoder, depth processing means denoising &compression.
3D scene

Depth Capturing

Data Compression & transmission

3D imaging tasks

MVC Decoder

MVC Encoder

Arbitrary
View Synthesis

Renderingunderstanding*W. Sun et al, “Rate-distortion Optimized 3D Reconstruction from Noise-corrupted Multiview Depth
Videos,” ICME, 2013.

29

Depth Processing in 3D Video
Communication• Pipeline of 3D Video Communication System• At encoder, depth processing means denoising &compression.
3D scene

Depth Capturing

Data Compression & transmission

3D imaging tasks

MVC Decoder

MVC Encoder

Arbitrary
View Synthesis

Renderingunderstanding*W. Sun et al, “Rate-distortion Optimized 3D Reconstruction from Noise-corrupted Multiview Depth
Videos,” ICME, 2013.

29

Two Practical Problems• Two related but different processing problems concerning
depth maps (after acquisition):
1. Denoising
MVC Encoder–

Acquired depth maps inherently noisy.

30

Two Practical Problems• Two related but different processing problems concerning
depth maps (after acquisition):
1. Denoising
MVC Encoder–

Acquired depth maps inherently noisy.

2. Compression– Reduce transmission bandwidth.

30

Separate vs. Joint Approach• Separate 2-step approach:
1. Denoise depth maps optimally (e.g.
high noise variance samples
MAP formulation) regardless of
low noise variance samples
rep. size;x(t)2. compress computed MAP surface
in deterministic way via
conventional codec.

• Joint approach by performing
denoising / compression as one:compressed signal

MAP solution– Problem inherently probabilistic.
– Can compress large noise variancesamples aggressively.
31xRate-constrained Estimation• Given observed depth maps y = [y1, y2, …], find optimal 3D surface s.
Bayes RulePr( y | s) Pr(s)Pr(s | y) =Pr( y)

Rate-constrained
MAP

max Pr( y | s) Pr(s)sRate Terms.t.

R(s) ≤ R

min − log Pr( y | s) − log Pr(s) + λR(s)sRate-constrained
search space for s• Distortion term: select s to agree w/ observations.
search space• Prior term: select s to agree w/ prior.
for s• Rate term: select s that requires few bits for representation.
32

Experimentation
Up to 2.42dB gain in PSNR

Fig. PSNR of synthesized virtual
views at decoder versus coding
rate for Lovebird1 (top) and

Balloons (bottom).

Improved virtual viewFig. Top Row (Lovebird1): synthesized virtual view 5 using texture and depth
maps at view 4 and 6. Depth maps are of 48kbps: Unprocessed (left), MLsolution (center), RD-optimized (right). Bottom Row (Balloons): synthesizedvirtual view 2 using texture depth maps at view 1 and 3. Depth maps are of
100kbps: Unprocessed (left), ML-solution (center), RD-optimized (right).
33

Presentation Outline• Background & Motivation (3D, not your mother’s 2D)• 3D Video representation / coding:• Depth map coding• HEVC tools for depth maps
• Graph-based Transform (GBT) for depth maps

• Depth map denoising• Denoising + compression?
• Why code depth images?

• 3D Video streaming:• Video compression with flexible decoding for interactive streaming• Loss-resilient texture-plus-depth video streaming (skip)• 3D view synthesis:• Robust view synthesis for free viewpoint video• Synthesized image interpolation for z-dimension camera movement
MMSP'13 Plenary 10/02/2013

34

35

Graph-based representation(Collaboration, Antonio Ortega, USC,
USA)

depth + color
capture

Find an alternative to depth-based representation:



Main idea-

describe the inter-view pixel connections as links in a graph

EPFL – Signal Processing Laboratory (LTS4)
http://lts4.epfl.ch

36

Framework
luminance
rate

luminance
Representation

Codinggeometry

N multiview
images

N depthimagesdecoding
Reconstruction

geometryrate

channelKnown at the encoder

N reconstructed
imagesdistortion D

EPFL – Signal Processing Laboratory (LTS4)
http://lts4.epfl.ch

36

Framework
luminance
rate

luminance
Representation

Codinggeometry

decoding
Reconstruction

geometry
rate

Most common approach:depth images for:- view synthesis prediction- virtual view synthesis
N multiview
images

N depthimageschannelKnown at the encoder

N reconstructed
imagesdistortion D

EPFL – Signal Processing Laboratory (LTS4)
http://lts4.epfl.ch

36

Framework
luminance
rate

luminance
Representation

Codinggeometry

Most common approach:depth images for:- view synthesis prediction- virtual view synthesis
N multiview
images

N depthimagesdecoding
Reconstruction

geometryrate

In that case, luminance is made of:- some reference frame- some residuals after view
synthesis prediction
channelKnown at the encoder

N reconstructed
imagesdistortion D

EPFL – Signal Processing Laboratory (LTS4)
http://lts4.epfl.chDepth images
characteristics

3D space
depth error

37

3D point

pixel

true pixel projection
shifted pixel projection

Captured luminance + depth images
viewpoint 1
Depth-based schemes:-Interpolated image
viewpoint 2

captured luminance and depth signals at several reference viewpoints
depth-based interpolation of intermediate viewpoint at decoder side

Depth-based representation drawbacks:-

an error in depth signal (estimation, compression) leads to spatial shift on the synthesized
viewpoint
the induced error is difficult to model
EPFL – Signal Processing Laboratory (LTS4)
http://lts4.epfl.ch

38

Idea
luminance
rate

luminance
Representation

Codinggraph

N multiview
images

N depthimagesdecoding
Reconstruction

graph
rate

replace depth values byconnections between pixels ofdifferent views
channelN reconstructed
imagesdistortion D

EPFL – Signal Processing Laboratory (LTS4)
http://lts4.epfl.ch

38

Idea
luminance
rate

luminance
Representation

Codinggraph

decoding
Reconstruction

graph
rate

- one reference frame- innovation pixels whose
positions are given by the graph
N multiview
images

N depthimagesreplace depth values byconnections between pixels ofdifferent views
channelN reconstructed
imagesdistortion D

EPFL – Signal Processing Laboratory (LTS4)
http://lts4.epfl.ch

39

Motivation: Pixel Classification
Pixels categories-(a) : appearing pixels-(b) : disoccluded pixels-(c) : occluded pixels-(d) : disappearing pixels



Warped image description-

links between these pixels and the reference image

Proposed graph-based representation-

links back to previous frames
OR explicit new pixels
EPFL – Signal Processing Laboratory (LTS4)
http://lts4.epfl.ch40

Graph-based representation
GRAPH RULES

Only new pixels appear in higher levels
Connections link these pixels with their neighbor in the previouslevelDescribe right view with:- a maximum of references to left view pixels- Only « new » pixels



The (a) appearing and (b) occluded pixels are described in
the first image/level they appear
The (c) disoccluded and (d) disappearing pixels are
represented in the graph by connections with noluminance values

EPFL – Signal Processing Laboratory (LTS4)
http://lts4.epfl.ch41

View reconstruction

Reconstruction policy:-

start at the level that is to be reconstructed and to fill all theappearing pixels-

follow the connections to upper levels when they occur-

go down to lower level when it is not possible to continue in thecurrent level
EPFL – Signal Processing Laboratory (LTS4)
http://lts4.epfl.ch

42

SummaryGraph links between views:-

Provide a description of the geometry-

Give an information of neighborhood between pixels-

Permits a better control of compresion error

Summary:
3D Video Representation / Coding• Geometry Representation of 3D scene for Image
Synthesis at Receiver.
• Depth Images:• Piecewise smooth. Compact representation?
• Auxiliary info. How to characterize err?
• Joint denoising / compression?

• Graph-based representation?

MMSP'13 Plenary 10/02/2013

43

Presentation Outline• Background & Motivation (3D, not your mother’s 2D)• 3D Video representation / coding:• Depth map coding• HEVC tools for depth maps
• Graph-based Transform (GBT) for depth maps

• Depth map denoising• Denoising + compression?
• Why code depth images?

• 3D Video streaming:• Video compression with flexible decoding for interactive streaming• Loss-resilient texture-plus-depth video streaming (skip)• 3D view synthesis:• Robust view synthesis for free viewpoint video• Synthesized image interpolation for z-dimension camera movement
MMSP'13 Plenary 10/02/2013

44

Background to Interactive Multiview Video Streaming Multiview Video• Closely spaced cameras capturing pictures
periodically and synchronously.
• The perception of depth via motion parallax.
Interactive Multiview Video Streaming (IMVS) A client can periodically request one of many captured views, as
video is played back in time.
 To reduce transmission BW, transmit only views interactively
selected by client.
 The encoding is done once at the server for a possibly large groupof clients.

*G. Cheung, A. Ortega, N.-M. Cheung, “Interactive Streaming of stored multiview video
Using redundant frame structures,” TIP, March 2011.

45

Background to Interactive Multiview Video StreamingMultiview Video Coding (MVC) Strong correlation both in temporal and inter-view domains.
 Efficiently encoding frames of all views in rate-distortion manner.
Are MVC frame structures suitable for IMVS?
 Insufficient decoding flexibility forinteractive view-switching.
 Multiple views transmitted but only one
single view displayed.

46

IMVS: 1st attempt w/ I + P-frames

Frame Structure Optimization [G. Cheung MMSP’08, PV’09]
 Using I- and P-frames, design redundant structures trading off
transmission rate and storage.
 Create multiple decoding paths for likely view transitions.
time

view

(a)

(b)
(c)

47

IMVS: 2nd attempt w/ merge frame

Merge Frame (M-frame) Identical reconstruction: an identical decoded frame for a set of
possible predictors at streaming time.
 Two novel DSC-based implementations of M-frame [N.-M. Cheung
PCS’09, G. Cheung ICIP’09].
 Application of M-frame in IMVS scenario, with superior performanceover I-frame [G. Cheung TIP’11].

same instantsame view

M

(a)

(b)

*W. Dai, G. Cheung, N.-M. Cheung, A. Ortega, O. Au, “Rate-Distortion Optimized Merge Frame
Using Piecewise Constant Functions,” ICIP’13.

48

IMVS: 2nd attempt w/ merge frame

Merge Frame (M-frame) Identical reconstruction: an identical decoded frame for a set of
possible predictors at streaming time.
 Two novel DSC-based implementations of M-frame [N.-M. Cheung
PCS’09, G. Cheung ICIP’09].
 Application of M-frame in IMVS scenario, with superior performanceover I-frame [G. Cheung TIP’11].

Recent Advance: developed RD-optimal merge frame without bit-plane
coding and channel coding in conventional DSC. (best student paper @ ICIP 2013).
same instantsame view

M

(a)

(b)

*W. Dai, G. Cheung, N.-M. Cheung, A. Ortega, O. Au, “Rate-Distortion Optimized Merge Frame
Using Piecewise Constant Functions,” ICIP’13.

48

IMVS: 3rd attempt w/ network delay

IMVS with fixed network delay Problem: view-switch request suffers one RTT delay.
 Key idea: upon each feedback, additional data are sent to cover all
view positions client could select when the data arrive at client.

*Xiaoyu Xiu, G. Cheung, A. Ortega, Jie Liang, “Delay-Cognizant Interactive Streaming of Multiview
Video using Free Viewpoint Synthesis,” TMM, March 2012.

49

IMVS: 3rd attempt w/ network delay

IMVS with fixed network delay Problem: view-switch request suffers one RTT delay.
 Key idea: upon each feedback, additional data are sent to cover all
view positions client could select when the data arrive at client.

*Xiaoyu Xiu, G. Cheung, A. Ortega, Jie Liang, “Delay-Cognizant Interactive Streaming of Multiview
Video using Free Viewpoint Synthesis,” TMM, March 2012.

49

Summary:
3D Video Streaming• High dimensional media navigation problem• Asymmetric info:• Sender knows statistical model for navigation.
• Receiver knows exact navigation path.

• Compression with decoding flexibility

MMSP'13 Plenary 10/02/2013

50

Presentation Outline• Background & Motivation (3D, not your mother’s 2D)• 3D Video representation / coding:• Depth map coding• HEVC tools for depth maps
• Graph-based Transform (GBT) for depth maps

• Depth map denoising• Denoising + compression?
• Why code depth images?

• 3D Video streaming:• Video compression with flexible decoding for interactive streaming• Loss-resilient texture-plus-depth video streaming (skip)• 3D view synthesis:• Robust view synthesis for free viewpoint video• Synthesized image interpolation for z-dimension camera movement
MMSP'13 Plenary 10/02/2013

51

Conversion (FTV context)
 Lossy Conversion• Depth Image Based Rendering (DIBR)• Depth Estimation from single or multiple viewpoints*Courtesy of Prof. P. Le Callet, UofNantes, France

52

DIBR: artefacts*Courtesy of Prof. P. Le Callet, UofNantes, France

53

DIBR: current quality metrics are uselessTowards a new quality metric for 3D synthesized views assessment – in IEEE ICIP 2011
Emilie Bosc, R.pépion, P. Le Callet, M. Köppel, P. Ndjiki-Nya, M. Pressigout, L. Morin*Courtesy of Prof. P. Le Callet, UofNantes, France54

Saliency-based Error Concealment

network
server

Goal: Packets are dropped in network during
video streaming. Reconstruct a missing pixel blockb by minimizing some cost function:

min 𝑓𝑓𝑓_𝑒𝑒𝑒(𝒃)𝒃Problem: The problem is under-determined.
Solution: Add a convex saliency term as follows:clientApply low-saliency priorduring EC

video frame

lost MBs

high saliency
regionmin{𝑓𝑓𝑓_𝑒𝑒𝑒 𝒃 + 𝜆 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠(𝒃)}𝒃Advantages:
1. Potential wrong candidates become less attention-grabbing.
2. It serves as a true prior in an ROI-based streaming application.

H. Hadizadeh, I. Bajic, G. Cheung, “Saliency-cognizant Error Concealment in Loss-corrupted Streaming Video”, ICME’2012
55(Best paper runner-up award), “Video Error Concealment Using a Computation-efficient Low Saliency Prior,” accepted to TMM, June 2013.

Saliency-based Error Concealment

network
server

Goal: Packets are dropped in network during
video streaming. Reconstruct a missing pixel blockb by minimizing some cost function:

min 𝑓𝑓𝑓_𝑒𝑒𝑒(𝒃)𝒃Problem: The problem is under-determined.
Solution: Add a convex saliency term as follows:clientApply low-saliency priorduring EC

video frame

lost MBs

high saliency
regionmin{𝑓𝑓𝑓_𝑒𝑒𝑒 𝒃 + 𝜆 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠(𝒃)}𝒃Advantages:
1. Potential wrong candidates become less attention-grabbing.
2. It serves as a true prior in an ROI-based streaming application.

H. Hadizadeh, I. Bajic, G. Cheung, “Saliency-cognizant Error Concealment in Loss-corrupted Streaming Video”, ICME’2012
55(Best paper runner-up award), “Video Error Concealment Using a Computation-efficient Low Saliency Prior,” accepted to TMM, June 2013.

Saliency-based Error Concealment

network
server

Goal: Packets are dropped in network during
video streaming. Reconstruct a missing pixel blockb by minimizing some cost function:

min 𝑓𝑓𝑓_𝑒𝑒𝑒(𝒃)𝒃Problem: The problem is under-determined.
Solution: Add a convex saliency term as follows:clientApply low-saliency priorduring EC

video frame

lost MBs

high saliency
regionmin{𝑓𝑓𝑓_𝑒𝑒𝑒 𝒃 + 𝜆 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠(𝒃)}𝒃Advantages:
orig. image less attention-grabbing.
our saliency map
IKN saliency map
1. Potential wrong candidates become
2. It serves as a true prior in an ROI-based streaming application.

H. Hadizadeh, I. Bajic, G. Cheung, “Saliency-cognizant Error Concealment in Loss-corrupted Streaming Video”, ICME’2012
55(Best paper runner-up award), “Video Error Concealment Using a Computation-efficient Low Saliency Prior,” accepted to TMM, June 2013.

Saliency-based Error Concealment

network
server

Goal: Packets are dropped in network during
video streaming. Reconstruct a missing pixel blockb by minimizing some cost function:

min 𝑓𝑓𝑓_𝑒𝑒𝑒(𝒃)𝒃Problem: The problem is under-determined.
Solution: Add a convex saliency term as follows:clientApply low-saliency priorduring EC

video frame

lost MBs

high saliency
regionmin{𝑓𝑓𝑓_𝑒𝑒𝑒 𝒃 + 𝜆 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠(𝒃)}𝒃Advantages:
1. Potential wrong candidates become less attention-grabbing.
2. It serves as a true prior in an ROI-based streaming application.

H. Hadizadeh, I. Bajic, G. Cheung, “Saliency-cognizant Error Concealment in Loss-corrupted Streaming Video”, ICME’2012
55(Best paper runner-up award), “Video Error Concealment Using a Computation-efficient Low Saliency Prior,” accepted to TMM, June 2013.

Saliency-based Error Concealment
Experiment: up to 3.6dB improvement in PSNR.

RECAP

Our Proposal
MMSP'13 Plenary 10/02/2013

56

Background: Free Viewpoint Video Streaming
1. encode texture + depth of 2 views (left &right) out of N, according to the desired
virtual view at receiver.

3D
scene

channel

3. Synthesize virtual
view from decoded left
and right views.

2. transmit overbandwidthconstrained, lossprone network.

texture +depth
SALIENCY-COGNIZANT ROBUST VIEW SYNTHESIS IN
FREE VIEWPOINT VIDEO STREAMING

BackgroundBackground: Packet Loss
V2

V0

Correlated
Loss

V1|0

Uncorrelated
Loss

Virtual View

SALIENCY-COGNIZANT ROBUST VIEW SYNTHESIS IN
FREE VIEWPOINT VIDEO STREAMING

BackgroundV1|2

Background: Packet Loss
V2

V0

Q: What is a good view
synthesis strategy given
losses in reference views?
Correlated
Loss

V1|0

Uncorrelated
Loss

Virtual View

SALIENCY-COGNIZANT ROBUST VIEW SYNTHESIS IN
FREE VIEWPOINT VIDEO STREAMING

BackgroundV1|2

System Assumption
Retransmission of lost packets (ARQ) leads to interactive
delay.

Foward error correction (FEC) code is used.
Unequal error protection (UEP) is applied, where more
important regions are protected more using FEC.

Low salient region: weak FEC
High salient region: strong FEC

SALIENCY-COGNIZANT ROBUST VIEW SYNTHESIS IN
FREE VIEWPOINT VIDEO STREAMING

System Overview

Formulation
1. Identify lost
pixels.
2. For each lost pixel patch p,
construct two patch candidates:• Weighted Pixel Blendng
(WPB)
1

ψp

• Examplar-Based Matching
(EPM)

ψ p2

SALIENCY-COGNIZANT ROBUST VIEW SYNTHESIS IN
FREE VIEWPOINT VIDEO STREAMING

Formulation

3. Select between 2
candidates:

min D(ψ pg ) + λZ (ψ pg )g∈{1, 2}

D – Expected Distortion
Z – Computed Saliency

Formulation
Low-saliency prior

1. Identify lost
pixels.
2. For each lost pixel patch p,
construct two patch candidates:• Weighted Pixel Blendng
(WPB)
1

ψp

• Examplar-Based Matching
(EPM)

ψ p2

SALIENCY-COGNIZANT ROBUST VIEW SYNTHESIS IN
FREE VIEWPOINT VIDEO STREAMING

Formulation

3. Select between 2
candidates:

min D(ψ pg ) + λZ (ψ pg )g∈{1, 2}

D – Expected Distortion
Z – Computed Saliency

Weighted Pixel Blending

X t0

X t1

Stv (i, j ) = (1 − v) X t0 (i, j 0 ) + vX t1 (i, j1 )SALIENCY-COGNIZANT ROBUST VIEW SYNTHESIS IN
FREE VIEWPOINT VIDEO STREAMING

WPBWeighted Pixel Blending

X t0

X t1

Stv (i, j ) = (1 − v) X t0 (i, j 0 ) + vX t1 (i, j1 )Key idea: adjust weights based on pixel reliability
SALIENCY-COGNIZANT ROBUST VIEW SYNTHESIS IN
FREE VIEWPOINT VIDEO STREAMINGWPBExamplar-Based Patch Matching
A similar algorithm as [8] is applied.
[8] A. Criminisi, P. Perez and C. Gomila., “Region filling and object removal by examplar-based image
inpaiting”, in IEEE Transactions on Image Processing, September 2004, vol 13., no 9, pp 1-13.

The order in which patches in the target region Ω is filled is done
according to a priority factor P(p).

P( p) = C ( p) D( p)C(p) denotes
confidence

SALIENCY-COGNIZANT ROBUST VIEW SYNTHESIS IN
FREE VIEWPOINT VIDEO STREAMING

EPM

D(p) is the data term
which is a function of thestrenght of isophotes.

Low-Saliency Prior
We determine the patch around a missing pixel with thehighest priority.
Then, the two possible candidates using WPB and EPM
are seltected based on:

min D (ψ ) + λZ (ψ )g∈{1, 2}gpgp

 D() for WPB is the average estimated distortion of pixel in patch D() for EPM is the average estimated distortion of the copied patch

SALIENCY-COGNIZANT ROBUST VIEW SYNTHESIS IN
FREE VIEWPOINT VIDEO STREAMING

Low-Saliency Prior

Experimental Results
Packet Losses manifest themselves as isolated MBs
due to FMO.
Packet Losses occur only in low-saliency regions (black
regions in the image) due to UEP.

SALIENCY-COGNIZANT ROBUST VIEW SYNTHESIS IN
FREE VIEWPOINT VIDEO STREAMING

Experimental Results

Results (uncorrelated losses)

SALIENCY-COGNIZANT ROBUST VIEW SYNTHESIS IN
FREE VIEWPOINT VIDEO STREAMING

Experimental Results

Results (correlated losses)

SALIENCY-COGNIZANT ROBUST VIEW SYNTHESIS IN
FREE VIEWPOINT VIDEO STREAMING

Experimental Results

Experimental Results

Using Co-located Blocks.
PSNR 34.70 dB
SALIENCY-COGNIZANT ROBUST VIEW SYNTHESIS IN
FREE VIEWPOINT VIDEO STREAMING

Proposed
PSNR 35.39 dB
Experimental Results

Presentation Outline• Background & Motivation (3D, not your mother’s 2D)• 3D Video representation / coding:• Depth map coding• HEVC tools for depth maps
• Graph-based Transform (GBT) for depth maps

• Depth map denoising• Denoising + compression?
• Why code depth images?

• 3D Video streaming:• Video compression with flexible decoding for interactive streaming• Loss-resilient texture-plus-depth video streaming (skip)• 3D view synthesis:• Robust view synthesis for free viewpoint video• Synthesized image interpolation for z-dimension camera movement
MMSP'13 Plenary 10/02/2013

68

DIBR and its difficulty with z-movement
DIBR
1. Texture + Depth
2. DIBR to project known pixels
3. Inpainting at decoder or intra-coded
blocks sent from server to fill in pixels in
disoccluded regions.

Referenceview

Requestedvirtual view

View-switch along the z-dimension is very
natural, but it is missing in the current
systems.
Difficulty:
Pixels get scattered far apart
P. Merkle, A. Smolic, K. Mueller, and T. Wiegand, “Multiview video plus depth
representation and coding” in IEEE International Conference on Image Processing, San
Antonio, TX,2007

69

Our Work• Goal:Design a new interpolation methodthat supports z-dimension
navigation
With better quality of interpolation,less information is need to be sent toenhance the quality• Challenges:
1.
2.

Distinguish between expansion
holes and disocclusion holes
How to interpolate the hole area

Example of expansion holes

Disocclusion: region not
visible in reference
Expansion: low sample
rate

70

Distinguish between expansion holes and disocclusion holes

Block based processing :1. construct a
histogram of depth
values of thesynthesized pixels inthe block,2. separate depth
pixels into layers
3. Convex set based
identification
71

Expansion Hole Interpolation
Interpolation:
1. Construct a Graph G, with pixels as its vertices, and connect thevertices with weighted edges
2. Use the eigen-vectors of the Graph Laplacian as the transform
matrixCalculation of Graph Laplacian72

Expansion Hole Interpolation
Non-local means: exploit the self-similarity in the images

Experiment results73Summary:
3D View Synthesis• Inverse 3D imaging problem• Not enough info for perfect reconstruction• Leverage on image interpolation, inpainting, super-resolution• Co-design with signal representation at sender?

MMSP'13 Plenary 10/02/2013

74

Presentation Summary• 3D Video representation / coding:• Depth map coding: standard + non-conventional coding tools.
• Depth map denoising• Q: Denoising + compression?
• Q: Why code depth images?

• 3D Video streaming:• Video compression with flexible decoding for interactive streaming• Q: High-dimensional media navigation problem?

• 3D view synthesis:• Robust view synthesis w/ low-saliency prior• Synthesized image interpolation using graph transform• Q: Inverse 3D imaging problem? Co-design w/ representation?
MMSP'13 Plenary 10/02/2013

75

Q&A
• Contact me at:• Email: cheung@nii.ac.jp• Homepage: http://research.nii.ac.jp/~cheung• CfP for Special Issue on “Interactive Media Processing
for Immersive Communication” in IEEE Journal on
Selected Topics in Signal Processing.
• Submission deadline: April 2nd, 2014
• Guest Editors: Gene Cheung, Dinei Florencio, Patrick Le Callet, ChiaWen Lin, Enrico Magli

MMSP'13 Plenary 10/02/201376