Gene Cheung
National Institute of Informatics
28th November, 2017

Interactive Media Streaming
Applications Using Merge Frames.
BJTU Visit 11/28/2017

1

Acknowledgement
Collaborators:• M. Kaneko (NII, Japan)• A. Ortega (USC, USA)• D. Florencio (MSR, USA)
• P. Frossard (EPFL, Switzerland)• J. Liang, I. Bajic (SFU, Canada)• V. Stankovic (U of Strathclyde, UK)• X. Wu (McMaster U, Canada)• P. Le Callet (U of Nantes, France)• X. Liu (HIT, China)• W. Hu, J. Liu, Z. Guo (Peking U., China)• L. Fang (Tsinghua, China)• C.-W. Lin (National Tsing Hua University, Taiwan)
BJTU Visit 11/27/2017

2

NII Overview• National Institute of Informatics• Chiyoda-ku, Tokyo, Japan.
• Government-funded research lab.

• Offers graduate courses & degrees
through The Graduate University for
Advanced Studies (Sokendai).
• 60+ faculty in “informatics”:
quantum computing, discretealgorithms, database, machinelearning, computer vision, speech &audio, image & video processing.
• Get involved!
• 2-6 month Internships.
• Short-term visits via
MOU grant.
• Lecture series,Sabbatical.

BJTU Visit 11/28/2017

3Introduction to APSIPA and APSIPA DL

APSIPA Mission: To promote broad spectrum of research and education
activities in signal and information processing in Asia Pacific
APSIPA Conferences: ASPIPA Annual Summit and Conference
APSIPA Publications: Transactions on Signal and Information Processingin partnership with Cambridge Journals since 2012; APSIPA Newsletters
APSIPA Social Network: To link members together and to disseminate
valuable information more effectively
APSIPA Distinguished Lectures: An APSIPA educational initiative to reachout to the community

APSIPA Distinguished Lecture Series
www.apsipa.org4

Outline• What is interactive media navigation?
• e.g. Multiview / free-viewpoint video• Merge frame for interactive media navigation• Previous works• Merge frame / block overview• Fixed target merging• Optimized target merging• Interactive Virtual Reality Video Streaming

Wei Dai, Gene Cheung, Ngai-Man Cheung, Antonio Ortega, Oscar Au, "Merge Frame Design for Video Stream Switching
using Piecewise Constant Functions," IEEE Transactions on Image Processing, vol. 25, no.8, August 2016
Gene Cheung, Zhi Liu, Zhiyou Ma, Jack Z. G. Tan, "Multi-Stream Switching for Interactive Virtual Reality Video
Streaming," IEEE International Conference on Image Processing, Beijing, China, September, 2017.

5

Outline• What is interactive media navigation?
• e.g. Multiview / free-viewpoint video• Merge frame for interactive media navigation• Previous works• Merge frame / block overview• Fixed target merging• Optimized target merging• Interactive Virtual Reality Video Streaming

BJTU Visit 11/28/2017

6

What is interactive media navigation /streaming?
• Server: a very large correlated media data set.
server• e.g., multiview video, light field data, etc.

• Client: can observe only small data subset at a time.

• Network: cannot deliver whole dataset before start ofnavigation.
network• Interactive navigation: client requests data, server sends
data. Repeat.
request data

clientdeliver dataserverclientG. Cheung, A. Ortega, N.-M. Cheung, B. Girod, “On Media Data Structures for Interactive Streaming in Immersive
Applications,” in SPIE Visual Communications and Image Processing, Huang Shan, China, July, 2010.

7

Interactive Multiview Video Streaming(IMVS)• Server: multiple views of same video captured synchronously in time.
• Client: can observe only 1 view at a time.
• Interactive navigation:• Client plays back video in time uninterrupted.
• Client requests view, server sends view. Repeat.

G. Cheung, A. Ortega, N.-M. Cheung, "Interactive Streaming of Stored Multiview Video using Redundant
Frame Structures," IEEE Transactions on Image Processing, vol.20, no.3, pp.744-761, March 2011.

8

Outline• What is interactive media navigation?
• e.g. Multiview / free-viewpoint video• Merge frame for interactive media navigation• Previous works• Merge frame / block overview• Fixed target merging• Optimized target merging• Interactive Virtual Reality Video Streaming

BJTU Visit 11/28/2017

9

Merge Frame for Media Navigation:
conflicting coding requirements• Inherent tension between coding efficiency &flexible decoding.
• Differential coding assumes single
order of frame decoding.
1

2

3• Flexible decoding assumes several
orders (paths) of frame decoding.

• Other examples:• Interactivebrowsingof JPEG2000
ResearchQuestion:
Howto enableimages.
flexible decoding• Single-video w/ forward & backward playback.
without• InteractiveLightField performance?
Streaming (ILFS).
greatsacrifice ofcoding
BJTU Visit 11/28/201710

Merge Frame for Media Navigation:
previous works 1• SP frames (H.264 extended profile):• Primary SP-frame: motion prediction + extra
quantization. (small).
• Secondary SP-frame: motion prediction +lossless encoding. (large).

1

2

1

2

3
3
344

• Pros: small primary SP-frame.
• Cons:• very large secondary SP-frames.
• As many secondary SP-frames as decoding paths.
M. Karczewicz and R. Kurceren, “The SP- and SI-frames design for H.264/AVC,” in IEEE Transactions
on Circuits and Systems for Video Technology, vol. 13, no.7, July 2003, pp. 637–644.
X. Sun, F. Wu, S. Li, G. Shen, and W. Gao, “Drift-free switching of compressed video bitstreams at predictive frames,”in IEEE Transactions on Circuits and Systems for Video Technology, vol. 16, no.5, May 2006, pp. 565–576.

11

Merge Frame for Media Navigation:
previous works 2• DSC frames:• Key Idea: treat merging as noise removal.
• Divide side information (SI) frames into block,perform DCT, quantization.

12124• Examine bit-planes of quantized coefficients.
• If bit-planes different from target, channel coding
to “denoise” SI bit-planes to target bit-planes.

34• Pros: one merge frame for many decoding paths.
• Cons:• Bit-plane / channel coding are complex.
• Channel coding works well only for average statistics.
P. Ramanathan, M. Kalman, and B. Girod, “Rate-distortion optimized interactive light field streaming,”
in IEEE Transactions on Multimedia, vol. 9, no.4, June 2007, pp. 813–825.
N.-M. Cheung, A. Ortega, and G. Cheung, “Distributed source coding techniques for interactivemultiview video streaming,” in 27th Picture Coding Symposium, Chicago, IL, May 2009.

12

Merge Frame for Media Navigation:
definition• Interactive Video Stream Switching (IVSS)• Multiple related pre-encoded video streams.
• Designated switching points to switch from one to another.

• Picture Interactive Graph

Dynamic View Switching:switch to neighboring viewof next time instant.

No loops in PIG.

Optimized target merging.

W. Dai, G. Cheung, N.-M. Cheung, A. Ortega, O. Au, “Rate-distortion Optimized Merge Frame using Piecewise
Constant Functions,” IEEE Int’l Conf. on Image Processing, Melbourne, Australia, September, 2013. (Best student paper award)
Wei Dai, Gene Cheung, Ngai-Man Cheung, Antonio Ortega, Oscar Au, "Merge Frame Design for Video Stream Switching
using Piecewise Constant Functions," IEEE Transactions on Image Processing, vol. 25, no.8, August 201613

Merge Frame for Media Navigation:
definition• Interactive Video Stream Switching (IVSS)• Multiple related pre-encoded video streams.
• Designated switching points to switch from one to another.

• Picture Interactive Graph

Static View Switching:switch to neighboring viewof same time instant.

Loops in PIG.

Fixed target merging.

J.-G. Lou, H. Cai, and J. Li, “A real-time interactive multi-view video system,” in ACM International
Conference on Multimedia, Singapore, November 2005.
N.-M. Cheung and A. Ortega, “Compression algorithms for flexible video decoding,” in IS&T/SPIE Visual
Communications and Image Processing (VCIP’08), San Jose, CA, January 2008.

14

Merge Frame for Media Navigation:framework• Switching Mechanism• Side Information (SI) frame: P-frame predicted from diff. streams.
• Merge frame: merge diff. among SI frames into same frame.
• Interactive Transmission: transmit one SI frame + merge frame according
to chosen decoding path.

BJTU Visit 11/28/201715

Merge Frame for Media Navigation:
merge frame (M-frame) overview
1. Each decoded SI frame is divided into 8x8 blocks, DCT transform andcoefficient quantized (q-coeff).
2. Given block b, if q-coeffs of SI frames very different, use I-block.
3. If q-coeffs of SI frames the same, use skip block.
4. If q-coeffs of SI frames slightly different, use merge block.

SI1

SI2

M-frameBJTU Visit 11/28/201716

Merge Frame for Media Navigation:
merge block overview• Use piecewise constant function (pcf) for merging of SI’s q-coeffs:••

Q-coeff’s must land on the same “step” for identical merging.

pcf defined by step size W and shift c:•

Choose W per frequency of all merge blocks (cheap).

•

Choose c per block per frequency (expensive).

W x  cf x   Wc
2
 W pcfq-coeffs

17Merge Frame for Media Navigation:
2 merging problems
Static view switching
Fixed Target Merging:•Find M-frame M to reconstruct any SI
frame Sn, n=1,…,N, identically to a fixedtarget T.

•

Difficult to optimize M-frame parameters.

Optimized Target Merging:•

Dynamic view switching

Find M-frame M to reconstruct any SI
frame Sn, n=1,…,N, identically to afloating target T M  , such that:M *  arg min D T, T M    R M M•Optimize M-frame parameters in RDmanner.
BJTU Visit 11/28/201718

Merge Frame for Media Navigation:
step W, shift c (fixed target merging)• Choosing step size W for given freq k:• Compute max diff. from target q-coeff in each block b:

Z b  max X b0  X bnn1,, N X b0

X b1

Zb

• Choose step size W to be roughly 2 * max diff:pcf:
W x  cf x   Wc
2
 W W  2Zb  2#b• Choosing shift c for each block b:• Choose shift:

X b2cb  Wb# / 2  X b0, 2 , where X b, 2  X b modWb00#• Lemma V.1: given this choice of step and shift,f X bn   X b0 , n  0,, N • Merge block group Bm, use a bigger step:

Z BM  max Z bbBMWB#m  2 Z Bm  2

X b0, 2  X b0 mod WB#m

19

Merge Frame for Media Navigation:
step W, shift c (optimized target merging)• Choosing step size W for given freq k:• Compute max diff. bet’n 2 q-coeffs in block b, then block-wise max diff.:Z  max X  X*bi , j0 ,, N ibjb

Z*BM

 max Z

• Choose step size W to be roughly max diff:bBM*bX b2

X b0

X b1

Z b*WBM  Z B* M  1• Choosing shift c for each block b:• Given step W, range Fb of shifts c can leadto identical merging.
• Choose c in Fb to min RD cost:min0 cb WB M | cbFbd b    log P cb peaks + uniform

continuous• Initialize P(cb):• Initialize a “peaks + uniform” distribution.
• Rate-constrained LM till convergence.

20Comparison with Coset Coding
X b2• Coset Coding:• SI values X b are noisy observations of target X b0nX b1X b0Zb

• Compute first largest difference w.r.t. to target:

Z b  max X bn  X b0n• Encoder: select coset size W  2 Z b , transmit coset index ib  X b mod W0• Decoder: compute Xˆ b  arg min X bn  X
X Zs.t. ib  X modW

• Fixed Target Merging:• Step W is roughly 2 Z b : Wb  2 Z b  2#• Shift c given W is remainder of target: cb  Wb / 2  X b , 2 , where#

0X b0, 2  X b0 mod Wb#• Expect the same coding rate as coset coding!
BJTU Visit 11/28/2017

21

Comparison with Coset Coding
X b2• Optimized Target Merging:

X b0

X b1ZbZ b*Wb  Z  1 , where Z b  Z• Compared to coset size W  2 Z b , nearly half the step size!
• Step W is roughly Z b :*b*b• Feasible range of shifts to select from via RD optimization:min0 cb WB M | cbFbd b    log P cb • Expect significant coding gain, especially at low rates.

BJTU Visit 11/28/2017

22

Merge Frame for Media Navigation:
experiments• Exp Setup: Static view switching• Fixed target merging: 3 views with the same QP.
• H.264 for I- and P-frames.
• Compared w/ DSC frames.

BJTU Visit 11/28/201723

Merge Frame for Media Navigation:
experiments 2• Exp Setup: Bit-rate adaptation• Optimized target merging: 3 streams of same sequence at diff. rates (TFRC).
• H.264 for I- and P-frames.
• vs. DSC frames, SP-frames.

• Worst case plots.

BJTU Visit 11/28/201724

Merge Frame for Media Navigation:
experiments 3• Exp Setup: Dynamic view switching• Optimized target merging: 3 views with the same QP.
• H.264 for I- and P-frames.
• vs. DSC frames, SP-frames.

• Worst case plots.

BJTU Visit 11/28/201725

Outline• What is interactive media navigation?
• e.g. Multiview / free-viewpoint video• Merge frame for interactive media navigation• Previous works• Merge frame / block overview• Fixed target merging• Optimized target merging• Interactive Virtual Reality Video Streaming

BJTU Visit 11/28/201726

Interactive Virtual Reality VideoStreaming• Virtual reality (VR): immersive 360 video
w/ headsets.
• Diff. fields-of-view (FoV) rendered onheadset, as user’s head rotates L / R.

• Transmit only FoV:
• reduces BW, but• results in stream-switch delay due toserver-to-client RTT.

Research problem:

Design redundant video streams covering diff. viewing ranges, accounting
for RTT EXPLICITLY, given storage and network constraints,Gene Cheung, Zhi Liu, Zhiyou Ma, Jack Z. G. Tan, "Multi-Stream Switching for Interactive Virtual Reality VideoStreaming," IEEE International Conference on Image Processing, Beijing, China, September, 2017.

27

Round-trip Time Interactive Delay

Sever / Client Interaction Model:• Client: transmits head coordinate θ per frame.
• Server: transmits corresponding video stream f(θ).
BJTU Visit 11/28/201728

Redundant Frame Structure Design
Objective Function:{di} are distortion vectors for streams i’s
mapping function from angle to stream

angle transition matrix

RTT in number of
total number of viewing angles
video frames
steady state prob for angle kcanonical row vector for angle kdistortion vector for diff.
angles for stream f(k)binary circulant matrix to account for FoV

BJTU Visit 11/28/2017

29

Redundant Frame Structure Design
Constraints:

encoding rate of stream igiven distortion vector di• Storage
constraint:storage budget
video length

channelbandwidth• Bandwidth
constraint:

BJTU Visit 11/28/201730

Experimental Setup• 2 video sequences: indoor concert and outdoor walking• FoV is 90◦, maximum switch each time is 5◦• FoV resolution is 512×512, two quantization parameters used, frame rate is30fps• #discrete view angles K=60, #switch during each T is 3
• View popularity: transition probabilities from i to j linearly decreases with |i− j|, the slope of decrease is steeper at π/2 and 3π/2
• Comparison scheme: ‘static’, a non-switching scheme, which always
sends an encoded video covering the entire 360 angles

BJTU Visit 11/28/201731

Simulation Results• outperforms ‘static’ by up to 2.9dB in PSNRBJTU Visit 11/28/201738

Summary• Interactive media navigation• Difficult to achieve to good compression efficiency & flexible
decoding.

• Merge frame to facilitate interactive navigation• Fixed target merging• Optimized target merging• Interactive virtual reality video streaming• Redundancy to overcome stream-switch delay due to RTT

BJTU Visit 11/28/2017

33

Q&A

• Email: cheung@nii.ac.jp• Homepage: http://research.nii.ac.jp/~cheung/

BJTU Visit 11/28/201734