# Traffic and Quality Characterization of the H.264/AVC Scalable Video Coding Extension

Authors: Geert Van der Auwera; Prasanth T. David; Martin Reisslein; Lina J. Karam
Journal: Advances in Multimedia
 (2008)

---

## Abstract

The recent scalable video coding (SVC) extension to the H.264/AVC video coding standard has unprecedented compression efficiency while supporting a wide range of scalability modes, including temporal, spatial, and quality (SNR) scalability, as well as combined spatiotemporal SNR scalability. The traffic characteristics, especially the bit rate variabilities, of the individual layer streams critically affect their network transport. We study the SVC traffic statistics, including the bit rate distortion and bit rate variability distortion, with long CIF resolution video sequences and compare them with the corresponding MPEG-4 Part 2 traffic statistics. We consider (i) temporal scalability with three temporal layers, (ii) spatial scalability with a QCIF base layer and a CIF enhancement layer, as well as (iii) quality scalability modes FGS and MGS. We find that the significant improvement in RD efficiency of SVC is accompanied by substantially higher traffic variabilities as compared to the equivalent MPEG-4 Part 2 streams.  We find that separately analyzing the traffic of temporal-scalability only encodings gives reasonable estimates of the traffic statistics of the temporal layers embedded in combined spatiotemporal encodings and in the base layer of combined FGS-temporal encodings. Overall, we find that SVC achieves significantly higher compression ratios than MPEG-4 Part 2, but produces unprecedented levels of traffic variability, thus presenting new challenges for the network transport of scalable video.


---

## Body

## 1. Introduction

We study the video traffic generated by the scalable
video coding (SVC) extension [1, 2]
of the H.264/MPEG-4 advanced video coding standard [3] (H.264 SVC for brevity).
This extension is expected to have a broad application domain for heterogeneous
wired and wireless video transmission to various terminals. Indications of the
growing acceptance of H.264/AVC are its adoption in application standards and
industry consortia specifications, such as DVB, ATSC, 3GPP, 3GPP2, MediaFLO,
DMB, DVD Forum (HD-DVD), and Blu-Ray Disc Association (BD-ROM). At the same time,
mobile TV technologies are made widely available. IPTV, mobile TV, satellite
TV, and video surveillance are considered key applications that can make
H.264/AVC and its SVC extension the dominant video encoder in the professional
and consumer markets.In order to examine the fundamental traffic
characteristics of H.264 SVC's scalability modes, we focus on encodings with
fixed quantization scales, that is, with variable bit rate (VBR). An additional
motivation for the focus on VBR video is that the VBR streams allow for
statistical multiplexing gains that have the potential to improve the
efficiency of video transport over communication networks [4–9]. The development of video network transport mechanisms
that meet the strict playout deadlines of the video frames and efficiently
accommodate the variability of the video traffic is a challenging problem.
Based primarily on the characteristics of MPEG-4 Part 2 single-layer and
scalable video, transport mechanisms have been developed for a wide range of
network transport scenarios, including video transport over the Internet (see,
e.g., [10–16]) over wireless networks
(see, e.g., [17–24]) over peer-to-peer networks (see, e.g., [25–32]) and over sensor networks
[33–35]. The widespread adoption of
the new H.264/AVC video standard necessitates the careful study of the traffic
characteristics of video coded with the new H.264/AVC codec and its extensions.
Recent traffic studies [36]
indicate that despite the lower average bit rate of H.264/AVC and H.264 SVC
single-layer video, elementary bufferless
multiplexing of a small number of video streams can be more efficient with
MPEG-4 Part 2 encoding than with H.264/AVC or H.264 SVC encoding due to the
significantly higher traffic variability of H.264/AVC and H.264 SVC. Therefore,
it is necessary to thoroughly examine the new SVC extension's statistical
traffic characteristics from a communication network perspective.The traffic characterizations and network transport
mechanisms for scalable video encoded with MPEG-4 Part 2 and older codecs have
received significant attention in the literature (see, e.g., [37–54]). The traffic
characterization of H.264/AVC and H.264 SVC nonscalable (single-layer)
traffic is studied in [36, 55, 56]. The study of network transport mechanisms in the
context of H.264/AVC (see, e.g., [57–59]) and H.264 SVC (see, e.g., [60–64]) has begun to attract
interest. To the best of our knowledge, the traffic of H.264
SVC-encoded
scalable video is for the first
time examined in the present study. Existing studies of the H.264/AVC codec and
its SVC extension, such as [3, 65, 66], focus primarily on the bit rate-distortion (RD)
performance, that is, the video quality (PSNR) as a function of the average bit rate, and typically consider only short video sequences up to a few hundred
frames. In contrast, for the transport over communication networks, the traffic
variability is also a key concern [5, 9, 14]. Therefore, we examine in the present study the joint
characterization of bit rate-distortion and higher order bit rate
statistics, such as the variability of the bit rate, as a function of
the distortion. We perform a detailed analysis of elementary statistics of the
scalable video traffic. We study statistics of frame sizes, group of picture
(GoP) sizes, as well as frame and GoP qualities. We use bit rate-distortion
(RD) and bit rate variability-distortion (VD) curves to compare the H.264
SVC-layered traffic to the equivalent traffic of MPEG-4 Part 2 [67], which is the predecessor
of H.264/AVC and which supports temporal, spatial, and FGS scalability. In
order to obtain reliable and meaningful statistical estimates of the traffic
variability and other properties, it is necessary to examine long video
sequences with several thousand frames, as we do in this study.All encodings of this study are publicly available as
video traces athttp://trace.eas.asu.edu/. Video traces [47] are files mainly containing
video frame time stamps, frame types (e.g., I, P, or B), encoded frame sizes
(in bits), and frame qualities (PSNR). Video traces are employed in simulation
studies of transport of scalable video over communication networks (see, e.g.,
[37–41, 44, 46, 52, 53, 54]). Key advantages of simulating with video traces over
experiments with actual video are that only very basic knowledge of video
encoding is required for simulations with video traces and that video traces are
freely available without copyright protection. Also, network simulations with
video traces can be conducted with standard network simulation programs and
integrated in network simulation modules (see, e.g., [68]), whereas experiments with
actual video require in-depth video coding expertise and large computational
resources for the encoding of many long video sequences.The paper is organized as follows. We provide a brief
overview of the scalability modes of the H.264 SVC extension in Section2. In
Section 3, we describe the video test sequences, encoding tools, and video
traffic metrics employed in our study. In Section 4, we analyze the traffic
characteristics of the individual temporal scalability layers of long CIF
videos. In Section 5, we study spatial scalability mode traffic with the same
long CIF sequences and their QCIF subsampled versions. In Sections 6
and 7, we examine SVC's fine granularity scalability (FGS) traffic and medium
granularity scalability (MGS) traffic, respectively. In Section 8, we
consider the combined spatiotemporal and FGS-temporal scalabilities, which
permit us to examine the separability of the combined scalability modes into
the basic modes from a video traffic analysis perspective. We summarize our
conclusions in Section 9.


## 2. Overview of H.264 Scalable Video  Coding (SVC)

In this section, we briefly introduce the scalable video coding (SVC) extension of
H.264/AVC. For a detailed discussion of the video technologies in the MPEG-4
family, such as MPEG-4 Part 2 [67] and H.264/AVC [3], we refer to [69]. At the end of 2007, the SVC scalability extension
was added to the H.264/AVC standard. The SVC extension provides temporal
scalability, spatial scalability, coarse (CGS) and medium (MGS) granularity
scalability, as well as combined spatiotemporal SNR scalability (restricted set
of spatiotemporal-SNR points can be extracted from a global scalable bit
stream). The fine granularity scalability (FGS) mode was initially intended to
be part of the SVC extension, however, FGS was not included in the initial
version of SVC. Presently, investigations are ongoing to include FGS in a
followup of the SVC extension.While earlier scalable video encoders and receivers,
such as MPEG-4 Part 2, did not gain wide market deployment, the H.264 SVC
scalability extension is expected to play a major role in providing video
services over heterogeneous networks due to the significantly improved rate-distortion
efficiency of the H.264 SVC scalability encoding tools (with respect to MPEG-4
Part 2) and the growing industrial acceptance of H.264/AVC as the successor of
the pervasive MPEG-2 standard.In the following subsections, we briefly discuss the main
scalability modes of this new H.264 SVC scalability amendment and refer to
[2] for detailed
information.


### 2.1. Temporal Scalability with Hierarchical B Frames

The introduction of hierarchical B frames has allowed the H.264 SVC encoder to
achieve temporal scalability while at the same time improving RD efficiency as
compared to the classical B frame prediction method, employed by the older MPEG
standards (MPEG-1/2/4 Part 2) and used by default in H.264/AVC. Figure1(a)
depicts the classical B frame prediction structure, where each B frame is
predicted only from the preceding I or P frame and from the subsequent I or P
frame. Figure 1(b) depicts the hierarchical B frame structure [70] which uses B frames to
predict B frames. The illustrated case is the dyadic hierarchy of B frames,
meaning that the number n of B frames in between the key pictures (I or
P frames) must equal n=2k−1. (We do not consider low-delay or constrained delay B frame prediction
structures, for which we refer to [2].)B frame prediction structures.
(a)
Classical B frame prediction structure(b)
Hierarchical B frame prediction structureWe depict the hierarchy with 3 B frames (I frame
period is 16) in Figure1(b). Temporal layer 0 consists of I and P key
pictures, which are used to predict the B frames of temporal layer 1 (the
temporal layer is indicated by the subscript of the I, P, and B symbols). The B
frames of temporal layer 1 together with the key pictures predict the B frames
of the second temporal layer. This halving of the prediction distance between
frames in each prediction step is called dyadic hierarchy, with each
splitting step resulting in one temporal layer, that is, the hierarchy with 15
B frames supports 5 temporal layers.Underneath Figures1(a) and 1(b), we provide for each
frame the preferred encoding order with the smallest decoding delay. We observe
that the encoding orders are identical for temporal layer 0, since the
prediction dependencies of the key pictures are identical in both cases. With
hierarchical B frames, the middle B frame is predicted first, while in the
classical approach, the first B frame is predicted first.The coding efficiency of hierarchical B frames depends
on the choice of the quantization parameters for each B frame. H.264 SVC
introduces cascading quantization scales which assign a higher quantization
parameter value (lower quality) to B frames belonging to higher temporal
layers.


### 2.2. Spatial Scalability

A spatial
scalable bit stream implies that streams with different frame resolutions, such
as QCIF (176×144 pixels), CIF (352×288), and 4CIF (704×576), are extractable from a single bit
stream. In this example, the QCIF layer would be the spatial base layer, and
the CIF and 4CIF layers the spatial enhancement layers. An important new
property of H.264 SVC is that a spatial layer is decodable with a single
motion-compensation loop.Besides the encoding mechanisms that we described in
Section2.1, the tools that exploit the interlayer
redundancies between spatial layers are interlayer motion prediction, interlayer
residual prediction, and interlayer intra prediction [2]. Figure 2 depicts the intra- and interlayer
prediction dependencies for two spatial layers (base and enhancement),
illustrating that the interlayer prediction mechanisms operate in a bottom-up
fashion, that is, the base layer is used for the prediction of the spatial
enhancement layer.Figure 2
Two-layer spatial
scalability intra- and interlayer prediction dependencies.


### 2.3. SNR Scalability, Including Fine and Medium Granularity Scalability

With SNR (quality) scalability, the quality of
the video frames is improved for a given spatial resolution and frame rate. The
main quality scalability modes, although not all are part of the SVC amendment,
are coarse granularity scalability (CGS), medium granularity scalability (MGS),
and fine granularity scalability (FGS). In our traffic study, we focus on MGS
(included in first SVC) and FGS (not included in first SVC), which we now
briefly review.H.264 FGS supports single-loop decoding. The I/P key
pictures of the quality base layer are predicted from one another as in Figure1(b), but the B frames can be predicted using all quality refinements available
in the higher quality layers, as illustrated in Figure 3(a). This prediction
using the quality refinements of the enhancement layer improves the coding
efficiency since the highest quality representation is used for prediction, but
results in a decoding drift error, that is only stopped at the next I/P key
picture [71]. Alternatively, the quality base layer
prediction structure can be based on the hierarchical B frames of the quality
base layer only, with identical dependencies in the quality refinement layer,
as illustrated in Figure 3(b). This prediction structure is also known as
closed-loop motion compensated prediction at low and high bit rates, and we
consider this structure in our traffic study.Fine
granularity scalability (FGS) prediction structures.
(a)
SVC FGS(b)
SVC FGS alternative(c)
MPEG-4 Part 2 FGSIn MPEG-4 Part 2 FGS, closed-loop motion compensation
is adopted only for the quality base layer while for the quality enhancement
layer, a bit-plane technique is used to code the difference between the
original picture and the picture reconstructed from the quality base layer, as
illustrated in Figure3(c). However, not exploiting the temporal redundancies
between the adjacent pictures in the FGS enhancement layer incurs a
considerable loss in coding efficiency, which schemes, such as PFGS [72], tried to alleviate.In H.264 FGS, hierarchical B frames are used to
efficiently exploit the temporal redundancy among adjacent pictures in the FGS
enhancement layer. Using a different coding technique (requantization of
quantization error) instead of bit-plane coding in MPEG-4 Part 2 FGS, H.264 FGS
codes the enhancement layer information in progressive refinement (PR) slices
that can be truncated with byte granularity. Furthermore, motion refinement is
allowed in the FGS enhancement layer, as detailed in [1].SVC MGS similarly encodes additional quality layers
that each consist of disposable quantities that are coarser than the byte
truncation offered by FGS. One MGS quality enhancement layer, for example,
increases the base layer quality corresponding to quantization parameter QP to the quality of an encoding with parameterQP−6. The information in each MGS enhancement layer can additionally be represented
with a maximum granularity of 1/16 or equivalently up to 16 refinements included
in the enhancement layer. This medium granularity enables network mechanisms to
drop MGS enhancement packets in a simplified manner compared to FGS, which
requires truncation.


### 2.4. Combined Scalability

H.264 SVC supports spatiotemporal-SNR scalability, also referred to ascombined scalability. This means that one global bit stream supports spatial, temporal,
and SNR scalability. Depending on the encoding configuration, several
individual bit streams with different spatial resolutions, frame rates, and SNR
enhancement layers are extractable from the global bit stream. The SNR
enhancement can be provided by CGS, MGS, or FGS. Note that not all scalability
modes are necessarily supported by a combined scalable bit stream.


## 2.1. Temporal Scalability with Hierarchical B Frames

The introduction of hierarchical B frames has allowed the H.264 SVC encoder to
achieve temporal scalability while at the same time improving RD efficiency as
compared to the classical B frame prediction method, employed by the older MPEG
standards (MPEG-1/2/4 Part 2) and used by default in H.264/AVC. Figure1(a)
depicts the classical B frame prediction structure, where each B frame is
predicted only from the preceding I or P frame and from the subsequent I or P
frame. Figure 1(b) depicts the hierarchical B frame structure [70] which uses B frames to
predict B frames. The illustrated case is the dyadic hierarchy of B frames,
meaning that the number n of B frames in between the key pictures (I or
P frames) must equal n=2k−1. (We do not consider low-delay or constrained delay B frame prediction
structures, for which we refer to [2].)B frame prediction structures.
(a)
Classical B frame prediction structure(b)
Hierarchical B frame prediction structureWe depict the hierarchy with 3 B frames (I frame
period is 16) in Figure1(b). Temporal layer 0 consists of I and P key
pictures, which are used to predict the B frames of temporal layer 1 (the
temporal layer is indicated by the subscript of the I, P, and B symbols). The B
frames of temporal layer 1 together with the key pictures predict the B frames
of the second temporal layer. This halving of the prediction distance between
frames in each prediction step is called dyadic hierarchy, with each
splitting step resulting in one temporal layer, that is, the hierarchy with 15
B frames supports 5 temporal layers.Underneath Figures1(a) and 1(b), we provide for each
frame the preferred encoding order with the smallest decoding delay. We observe
that the encoding orders are identical for temporal layer 0, since the
prediction dependencies of the key pictures are identical in both cases. With
hierarchical B frames, the middle B frame is predicted first, while in the
classical approach, the first B frame is predicted first.The coding efficiency of hierarchical B frames depends
on the choice of the quantization parameters for each B frame. H.264 SVC
introduces cascading quantization scales which assign a higher quantization
parameter value (lower quality) to B frames belonging to higher temporal
layers.


## 2.2. Spatial Scalability

A spatial
scalable bit stream implies that streams with different frame resolutions, such
as QCIF (176×144 pixels), CIF (352×288), and 4CIF (704×576), are extractable from a single bit
stream. In this example, the QCIF layer would be the spatial base layer, and
the CIF and 4CIF layers the spatial enhancement layers. An important new
property of H.264 SVC is that a spatial layer is decodable with a single
motion-compensation loop.Besides the encoding mechanisms that we described in
Section2.1, the tools that exploit the interlayer
redundancies between spatial layers are interlayer motion prediction, interlayer
residual prediction, and interlayer intra prediction [2]. Figure 2 depicts the intra- and interlayer
prediction dependencies for two spatial layers (base and enhancement),
illustrating that the interlayer prediction mechanisms operate in a bottom-up
fashion, that is, the base layer is used for the prediction of the spatial
enhancement layer.Figure 2
Two-layer spatial
scalability intra- and interlayer prediction dependencies.


## 2.3. SNR Scalability, Including Fine and Medium Granularity Scalability

With SNR (quality) scalability, the quality of
the video frames is improved for a given spatial resolution and frame rate. The
main quality scalability modes, although not all are part of the SVC amendment,
are coarse granularity scalability (CGS), medium granularity scalability (MGS),
and fine granularity scalability (FGS). In our traffic study, we focus on MGS
(included in first SVC) and FGS (not included in first SVC), which we now
briefly review.H.264 FGS supports single-loop decoding. The I/P key
pictures of the quality base layer are predicted from one another as in Figure1(b), but the B frames can be predicted using all quality refinements available
in the higher quality layers, as illustrated in Figure 3(a). This prediction
using the quality refinements of the enhancement layer improves the coding
efficiency since the highest quality representation is used for prediction, but
results in a decoding drift error, that is only stopped at the next I/P key
picture [71]. Alternatively, the quality base layer
prediction structure can be based on the hierarchical B frames of the quality
base layer only, with identical dependencies in the quality refinement layer,
as illustrated in Figure 3(b). This prediction structure is also known as
closed-loop motion compensated prediction at low and high bit rates, and we
consider this structure in our traffic study.Fine
granularity scalability (FGS) prediction structures.
(a)
SVC FGS(b)
SVC FGS alternative(c)
MPEG-4 Part 2 FGSIn MPEG-4 Part 2 FGS, closed-loop motion compensation
is adopted only for the quality base layer while for the quality enhancement
layer, a bit-plane technique is used to code the difference between the
original picture and the picture reconstructed from the quality base layer, as
illustrated in Figure3(c). However, not exploiting the temporal redundancies
between the adjacent pictures in the FGS enhancement layer incurs a
considerable loss in coding efficiency, which schemes, such as PFGS [72], tried to alleviate.In H.264 FGS, hierarchical B frames are used to
efficiently exploit the temporal redundancy among adjacent pictures in the FGS
enhancement layer. Using a different coding technique (requantization of
quantization error) instead of bit-plane coding in MPEG-4 Part 2 FGS, H.264 FGS
codes the enhancement layer information in progressive refinement (PR) slices
that can be truncated with byte granularity. Furthermore, motion refinement is
allowed in the FGS enhancement layer, as detailed in [1].SVC MGS similarly encodes additional quality layers
that each consist of disposable quantities that are coarser than the byte
truncation offered by FGS. One MGS quality enhancement layer, for example,
increases the base layer quality corresponding to quantization parameter QP to the quality of an encoding with parameterQP−6. The information in each MGS enhancement layer can additionally be represented
with a maximum granularity of 1/16 or equivalently up to 16 refinements included
in the enhancement layer. This medium granularity enables network mechanisms to
drop MGS enhancement packets in a simplified manner compared to FGS, which
requires truncation.


## 2.4. Combined Scalability

H.264 SVC supports spatiotemporal-SNR scalability, also referred to ascombined scalability. This means that one global bit stream supports spatial, temporal,
and SNR scalability. Depending on the encoding configuration, several
individual bit streams with different spatial resolutions, frame rates, and SNR
enhancement layers are extractable from the global bit stream. The SNR
enhancement can be provided by CGS, MGS, or FGS. Note that not all scalability
modes are necessarily supported by a combined scalable bit stream.


## 3. Study Setup: Video Sequences, Encoding  Tools, and Video Traffic Metrics

In this section, we introduce the setup used for
obtaining the video traffic and quality characterizations presented in the
subsequent sections.


### 3.1. Video Sequences

The Common Intermediate Format (CIF,352×288 pixels) video sequences used for the
statistics presented in this study are the ten-minute Sony Digital Video
Camera Recorder demo sequence (17,682 frames at 30 frames/sec), which we
refer to as Sony Demo sequence, the first half hour of the Silence of
the Lambs movie (54 000 frames at 30 frames/sec), the Star Wars IV movie (54 000 frames at 30 frames/sec), and the first hour of the Tokyo
Olympics video (133 128 frames at 30 frames/sec). We also use about 30
minutes of the NBC 12 News (49 523 frames at 30 frames/sec), including
the commercials. The video sequences Silence of the Lambs, Star Wars
IV, Tokyo Olympics, and NBC 12 News can, respectively, be
described as drama/thriller, science fiction/action, sports, and news video.
The Sony Demo sequence consists of 29 scenes with varying texture and
motion complexities. Due to space constraints, we present in this paper only
illustrative plots for encodings with Silence of the Lambs and Star
Wars IV. The corresponding plots for the other video sequences are
available in [73, 74].


### 3.2. Encoding Tools

We used the MEncoder tool to decode the sequences into uncompressed YUV format and to
subsample the originally higher resolution sequences to CIF resolution. We used
the MPEG-4 Part 2 Microsoft v2.3.0 software, and the SVC reference software,
named JSVM, version 5.9 for the temporal layer evaluations, and versions 7.10
and 7.13, respectively, for studying FGS and spatial scalability.


### 3.3. Encoding Setup

We employ four
GoP structures in our study of temporal scalability layers, namely,IBPBPBPBPBPBPBPB (16 frames, with 1 B frame per I/P frame), which we denote by G16-B1, IBBBPBBBPBBBPBBB (16 frames, with 3 B frames per I/P frame) denoted by G16-B3, IBBBBBBBPBBBBBBB (16 frames, with 7 B frames per I/P frame) denoted by G16-B7, and IBBBBBBBBBBBBBBB (16 frames, with 15 B frames per I frame) denoted by G16-B15. In the
context of SVC, these four GoP structures are, respectively, designated by their
“GoP size” which is the number of hierarchical B frames plus one key picture,
either of type I or P. Hence, G16-B1 has GoP size 2, G16-B3 has
GoP size 4, G16-B7 has GoP size 8, and G16-B15 has GoP size 16.
In the following, we employ our own GoP structure notation to emphasize the
repetitive I-P-B frame type patterns in the encodings. These four GoP
structures are natural structures for hierarchical B frames and allow us to
compare temporal layer statistics across encoders based on identical underlying
GoP patterns.Due to space constraints, we primarily focus in this
paper on the temporal scalability layers for theG16-B3 GoP structure,
which supports three temporal layers. The other GoP structures are presented in
[73]. In our study of
the spatial and FGS scalability layers, we focus on the GoP structure G16-B3 since the RD efficiency of MPEG-4 Part 2 deteriorates for more B frames, making
a comparison across encoders less useful.


### 3.4. Video Traffic Metrics

We summarize the video traffic and quality metrics,
which are all defined with respect to a given video sequence encoded with a
fixed quantization scale, in Table1. We remark that the coefficient of
variation of the frame sizes CoVX is widely employed as a measure of the
variability of the frame sizes, that is, the bit rate variability of the
encoded video. Plotting the CoV as a function of the quantization scale (or
equivalently, the PSNR video quality) gives the rate variability-distortion
(VD) curve [48].
Alternatively, the peak-to-mean (Peak/Mean or PtM) ratio of the frame sizes is commonly used to
express the traffic variability.Table 1
Video traffic and quality metrics for encoding with given quantization scale.
MetricDefinitionFrame size metricsMNumber of frames in video sequenceXmSize [bits] of encoded video framem, m=1,…,MX¯=(1/M)∑m=1MXmMean frame size of encoded video sequenceSX2=(1/(M−1))∑m=1M(Xm−X¯)2Variance of frame sizes (SX is the standard deviation [bits])CoVX=SX/X¯Coefficient of variation of frame sizes
[unit free]Xmax=maxm=1,…,MXmMaximum frame size [bits]PtMX=Xmax/X¯Peak-to-mean frame size ratio [unit free]Bit rate metricsTFrame period [s]R¯=X¯/TMean bit rate [bits/s]Rmax=Xmax/TPeak bit rate [bits/s]GoP size metricsNNumber of frames in GoPYnSize of GoPn, n=1,…,M/N [bits]Y¯=(M/N)∑n=1M/NYnMean GoP size [bits]PtMY=Ymax/Y¯Peak-to-mean GoP size ratio [unit free]CoVY=SY/Y¯Coefficient of variation of GoP sizes [unit
free]Frame quality metricsQmPSNR quality of framem as defined in (2)Q¯=(1/M)∑m=1MQmAverage PSNR quality of video sequenceCoQV=SQ/Q¯Coefficient of quality variationRegarding the bit rate metrics, we note that if each
video frame is transmitted during one frame periodT (e.g., 33 milliseconds for 30 frames/s), then
the bit rate Rm [bits/s] required to transmit frame Xm is Rm=Xm/T. The corresponding mean bit rate R¯ and peak bit rate Rmax [bits/s] are defined in Table 1.We define aGroup of Pictures (GoP) of an
encoded video stream as one I frame and all subsequent P and B frames before
the next I frame in the stream. The size Yn [bits] of GoP n equals the sum of the sizes of the N frames that belong to the GoP.We use the peak signal-to-noise ratio (PSNR) as the
objective measure of the quality of a reconstructed video frameR(x,y) with respect to the uncompressed video frame F(x,y). The larger the difference between R(x,y) and F(x,y), or equivalently, the lower the quality of R(x,y), the lower the PSNR value. The PSNR is expressed in decibels [dB] to accommodate
the logarithmic sensitivity of the human visual system. The PSNR is typically
obtained for the luminance video frame and in case of a Nx×Ny frame consisting of 8-bit pixel values, it is
computed as a function of the mean squared error (MSE) as
(1)MSE=1Nx⋅Ny∑x=0Nx−1∑y=0Ny−1[F(x,y)−R(x,y)]2,(2)PSNR=10⋅log102552MSE.
We denote the PSNR quality of a
video frame m by Qm.
For a detailed definition of all statistics used in this study, we refer to
[75].


## 3.1. Video Sequences

The Common Intermediate Format (CIF,352×288 pixels) video sequences used for the
statistics presented in this study are the ten-minute Sony Digital Video
Camera Recorder demo sequence (17,682 frames at 30 frames/sec), which we
refer to as Sony Demo sequence, the first half hour of the Silence of
the Lambs movie (54 000 frames at 30 frames/sec), the Star Wars IV movie (54 000 frames at 30 frames/sec), and the first hour of the Tokyo
Olympics video (133 128 frames at 30 frames/sec). We also use about 30
minutes of the NBC 12 News (49 523 frames at 30 frames/sec), including
the commercials. The video sequences Silence of the Lambs, Star Wars
IV, Tokyo Olympics, and NBC 12 News can, respectively, be
described as drama/thriller, science fiction/action, sports, and news video.
The Sony Demo sequence consists of 29 scenes with varying texture and
motion complexities. Due to space constraints, we present in this paper only
illustrative plots for encodings with Silence of the Lambs and Star
Wars IV. The corresponding plots for the other video sequences are
available in [73, 74].


## 3.2. Encoding Tools

We used the MEncoder tool to decode the sequences into uncompressed YUV format and to
subsample the originally higher resolution sequences to CIF resolution. We used
the MPEG-4 Part 2 Microsoft v2.3.0 software, and the SVC reference software,
named JSVM, version 5.9 for the temporal layer evaluations, and versions 7.10
and 7.13, respectively, for studying FGS and spatial scalability.


## 3.3. Encoding Setup

We employ four
GoP structures in our study of temporal scalability layers, namely,IBPBPBPBPBPBPBPB (16 frames, with 1 B frame per I/P frame), which we denote by G16-B1, IBBBPBBBPBBBPBBB (16 frames, with 3 B frames per I/P frame) denoted by G16-B3, IBBBBBBBPBBBBBBB (16 frames, with 7 B frames per I/P frame) denoted by G16-B7, and IBBBBBBBBBBBBBBB (16 frames, with 15 B frames per I frame) denoted by G16-B15. In the
context of SVC, these four GoP structures are, respectively, designated by their
“GoP size” which is the number of hierarchical B frames plus one key picture,
either of type I or P. Hence, G16-B1 has GoP size 2, G16-B3 has
GoP size 4, G16-B7 has GoP size 8, and G16-B15 has GoP size 16.
In the following, we employ our own GoP structure notation to emphasize the
repetitive I-P-B frame type patterns in the encodings. These four GoP
structures are natural structures for hierarchical B frames and allow us to
compare temporal layer statistics across encoders based on identical underlying
GoP patterns.Due to space constraints, we primarily focus in this
paper on the temporal scalability layers for theG16-B3 GoP structure,
which supports three temporal layers. The other GoP structures are presented in
[73]. In our study of
the spatial and FGS scalability layers, we focus on the GoP structure G16-B3 since the RD efficiency of MPEG-4 Part 2 deteriorates for more B frames, making
a comparison across encoders less useful.


## 3.4. Video Traffic Metrics

We summarize the video traffic and quality metrics,
which are all defined with respect to a given video sequence encoded with a
fixed quantization scale, in Table1. We remark that the coefficient of
variation of the frame sizes CoVX is widely employed as a measure of the
variability of the frame sizes, that is, the bit rate variability of the
encoded video. Plotting the CoV as a function of the quantization scale (or
equivalently, the PSNR video quality) gives the rate variability-distortion
(VD) curve [48].
Alternatively, the peak-to-mean (Peak/Mean or PtM) ratio of the frame sizes is commonly used to
express the traffic variability.Table 1
Video traffic and quality metrics for encoding with given quantization scale.
MetricDefinitionFrame size metricsMNumber of frames in video sequenceXmSize [bits] of encoded video framem, m=1,…,MX¯=(1/M)∑m=1MXmMean frame size of encoded video sequenceSX2=(1/(M−1))∑m=1M(Xm−X¯)2Variance of frame sizes (SX is the standard deviation [bits])CoVX=SX/X¯Coefficient of variation of frame sizes
[unit free]Xmax=maxm=1,…,MXmMaximum frame size [bits]PtMX=Xmax/X¯Peak-to-mean frame size ratio [unit free]Bit rate metricsTFrame period [s]R¯=X¯/TMean bit rate [bits/s]Rmax=Xmax/TPeak bit rate [bits/s]GoP size metricsNNumber of frames in GoPYnSize of GoPn, n=1,…,M/N [bits]Y¯=(M/N)∑n=1M/NYnMean GoP size [bits]PtMY=Ymax/Y¯Peak-to-mean GoP size ratio [unit free]CoVY=SY/Y¯Coefficient of variation of GoP sizes [unit
free]Frame quality metricsQmPSNR quality of framem as defined in (2)Q¯=(1/M)∑m=1MQmAverage PSNR quality of video sequenceCoQV=SQ/Q¯Coefficient of quality variationRegarding the bit rate metrics, we note that if each
video frame is transmitted during one frame periodT (e.g., 33 milliseconds for 30 frames/s), then
the bit rate Rm [bits/s] required to transmit frame Xm is Rm=Xm/T. The corresponding mean bit rate R¯ and peak bit rate Rmax [bits/s] are defined in Table 1.We define aGroup of Pictures (GoP) of an
encoded video stream as one I frame and all subsequent P and B frames before
the next I frame in the stream. The size Yn [bits] of GoP n equals the sum of the sizes of the N frames that belong to the GoP.We use the peak signal-to-noise ratio (PSNR) as the
objective measure of the quality of a reconstructed video frameR(x,y) with respect to the uncompressed video frame F(x,y). The larger the difference between R(x,y) and F(x,y), or equivalently, the lower the quality of R(x,y), the lower the PSNR value. The PSNR is expressed in decibels [dB] to accommodate
the logarithmic sensitivity of the human visual system. The PSNR is typically
obtained for the luminance video frame and in case of a Nx×Ny frame consisting of 8-bit pixel values, it is
computed as a function of the mean squared error (MSE) as
(1)MSE=1Nx⋅Ny∑x=0Nx−1∑y=0Ny−1[F(x,y)−R(x,y)]2,(2)PSNR=10⋅log102552MSE.
We denote the PSNR quality of a
video frame m by Qm.
For a detailed definition of all statistics used in this study, we refer to
[75].


## 4. Temporal Scalability Traffic Analysis

We examine the
traffic characteristics of the temporal layers embedded in video streams
encoded with H.264 SVC and MPEG-4 Part 2. We demonstrate that the traffic
variability of H.264 SVC temporal layers is significantly higher than the
variability of the corresponding MPEG-4 Part 2 temporal layers. For a fair
comparison, we assume that the same temporal layers as for H.264 SVC can be
extracted from the MPEG-4 Part 2 traffic. Although the bitstream syntax of the
latter does not support this extraction, it is in principle feasible for an
intelligent media gateway or decoder to drop the B frames belonging to the
respective temporal layers according to the H.264 SVC dyadic layer principle.


### 4.1. Temporal Layer Basics

TheG16-B3 GoP structure implies the repetition of the frame type pattern I0B2B1B2P0B2B1B2P0B2B1B2P0B2B1B2, whereby the subscripts denote the temporal layers (0, 1, 2) to which a frame
belongs. The temporal base layer (0) is therefore I0000P0000P0000P0000, with zeroes replacing the dropped B frames of temporal enhancement layers 1 and
2. The first temporal enhancement layer is 00B1000B1000B1000B10 and the second enhancement layer is 0B20B20B20B20B20B20B20B2.In case of our CIF sequences at a frame rate of 30
frames per second (fps), the temporal base layer represents a stream with a
frame rate of 7.5 fps, the combination (aggregation) of the base layer and the
first enhancement layer increases the frame rate to 15 fps, and the reception
of the second enhancement layer results in the full frame rate of 30 fps. We
note that the temporal base layer frames are required for decoding enhancement
layer 1 frames, and that enhancement layer 2 frames need both lower layers to
be decoded.Let us examine the video quality associated with
receiving certain temporal layers. Clearly, the average PSNR video quality of
the combination of all temporal layers, that is, of the aggregated traffic, is
equal to the average quality of the single-layer video stream. However, if we
would simply average the PSNR values (Q) of the base layer frames (QI0QP0QP0QP0), then this average would be unrealistically
high compared to the average of the corresponding single-layer (30 fps) stream,
since the subjective quality impression of human observers is much lower for
the frame rate of 7.5 fps. In order to include this perceptual quality
degradation in the PSNR measurement, we assume that the decoder duplicates a
received base layer frame until the next frame is received and decoded. The
result is the duplicated base layer frame sequence I00I00I00I00P04P04P04P04P08P08P08P08P012P012P012P012, where the upper index represents the duplicated frame number. This sequence has
a frame rate of 30 fps.The PSNR value of a duplicated framen located at frame number i is calculated based on the MSE between this duplicated frame n and the original frame i from the original (uncompressed) sequence.
This PSNR value reflects the subjective distortion that occurs when jerky
sequences consisting of duplicated frames are viewed by human observers. In
general, the perceived video quality of a sequence is high if the average PSNR
is high and the quality variation is low [5]. When there is low motion activity in the successive
frames, that is, when frames are alike (low MSE), then duplication of frames results in
barely noticeable jerkiness. The variation of the PSNR values is therefore also
small. On the other hand, when high motion activity is present, then successive
frames differ substantially and the MSE between successive frames is large, as well as
the quality variation. The computed overall PSNR average therefore sufficiently
incorporates the perceptual video quality reduction due to the reduced frame
rate (jerkiness).We apply the same principle to the computation of the
average quality when the temporal base and first enhancement layer are received
and decoded. This means that the following frame sequence is displayed:I00I00B12B12P04P04B16B16P08P08B110B110P012P012B114B114. The combination of all temporal layers results in displaying the sequence I00B21B12B23P04B25B16B27P08B29B110B211P012B213B114B215, which is the single-layer sequence.Before we analyze the temporal layer traffic
statistics, we describe the simple smoothing that we apply to the temporal base
and enhancement layers to decrease the traffic variability. LetXi denote the frame size (bytes) of frame i. Since there are large transmission gaps between frames of the base layer, we
can redistribute the frame sizes over these gaps by dividing the frame size by
four, and hence sending a quarter of each base layer frame during one frame
period: X0I0/4,X0I0/4,X0I0/4,X0I0/4,X4P0/4,X4P0/4,X4P0/4,X4P0/4,X8P0/4, X8P0/4,X8P0/4,X8P0/4,…. Equivalently, we say that we have smoothed the temporal base layer traffic over a=4 frames. Analogously, the first enhancement
layer traffic is smoothed over a=4 frames: X2B1/4,X2B1/4,X2B1/4,X2B1/4,X6B1/4, X6B1/4,X6B1/4,X6B1/4,X10B1/4,X10B1/4,X10B1/4,X10B1/4,X14B1/4, X14B1/4,X14B1/4,X14B1/4. The second layer is smoothed over a=2 frames since only one frame is missing in
between the B frames of this layer: X1B2/2,X1B2/2,X3B2/2, X3B2/2,X5B2/2,X5B2/2,X7B2/2,X7B2/2,X9B2/2,X9B2/2,X11B2/2, X11B2/2,X13B2/2,X13B2/2,X15B2/2,X15B2/2. This basic smoothing introduces extra decoding delays, but mitigates to some
extent the high rate variability as we demonstrate in the next section.


### 4.2. Results and Discussion

We treat each
temporal layer separately in the following analysis, except for the layer
quality where we assume the reception of all lower layers. The aggregation of
all layers is equivalent to the single-layer case, which is analyzed in detail
in [36]. The main reason
for treating each layer separately is that streaming protocols, such as the
Real Time Protocol [76, 77], typically packetize and stream each layer separately
to allow for different treatment of the layers in the network.In Table2,
we summarize traffic and quality statistics of the temporal base layer and the
two temporal enhancement layers included in the G16-B3 GoP structure.
The table includes frame size, bit rate, smoothed frame size, GoP size, and
video quality statistics. We estimate these statistics based on the five long
CIF sequences that we encode with H.264 SVC and MPEG-4 Part 2. In the first
column of Table 2, the encoding mode is specified by a code representing the
encoder (SV for H.264 SVC and Mp for MPEG-4 Part 2) and the
quantization scale. For each encoder, we present min/mean/max values (computed
across the five sequences) for two selected quantization scales that result in
approximately equal PSNR quality (max-to-min) ranges. For example, the base
layer quantization scale 28 for H.264 SVC results in the PSNR quality range
29.3–36.6 dB, and the quantization scale 4 for MPEG-4 Part 2 results in the
quality range 29.2–36.5 dB. We compare the various statistical quantities in
Table 2 based on matching quality ranges between encoders. Detailed results
for the full range of studied quantization scales, which gives the RD and VD curves
presented in this paper, are available in [73, 74].Table 2
Traffic statistics for the layers of temporal scalability encodings using H.264 SVC and MPEG-4 Part 2 for selected
quantization scales with GoP structure G16B3.
Frame SizeBit RateSmoothed (a)GoP SizeFrame QualityEnc.MeanCoVXPtMXMeanPeakCoVX(a)PtMX(a)CoVYPtMYMeanCoQVModeX¯SXX¯XmaxX¯X¯TXmaxTSX(a)X¯Xmax(a)X¯SYY¯YmaxY¯Q¯SQQ¯[kbyte][Mbps][Mbps][dB]Temporal base layer witha=4 for smoothingSV28Min0.6542.13415.9820.1573.2070.6233.9960.3652.49429.3070.256SV28Mean1.2692.54621.4730.3056.2000.9255.3690.6473.61433.9320.286SV28Max1.9302.87830.1930.4638.5041.1497.5490.8855.34036.6390.316Mp04Min0.9352.12017.5390.2245.6840.6114.3860.3452.37029.2300.256Mp04Mean1.6732.45821.6460.4018.2720.8665.4120.6113.47533.8340.282Mp04Max2.4682.69726.3440.59210.9791.0346.5870.8124.95836.4890.307SV42Min0.1372.60927.3630.0330.9570.9766.8430.3842.58026.3240.167SV42Mean0.2292.97531.5740.0551.6891.2067.8950.6554.14929.6780.218SV42Max0.3273.27639.9330.0782.2301.3909.9850.9066.58732.0230.268Mp20Min0.2672.35420.6360.0641.5290.7975.1600.3592.25326.3440.169Mp20Mean0.3922.46322.3450.0942.0940.8755.5870.4933.25629.6920.201Mp20Max0.5152.59623.8580.1242.8320.9675.9660.6124.28132.3450.230Temporal enhancement layer 1 witha=4 for smoothingSV28Min0.1112.56936.2560.0271.4490.9489.0620.8526.60933.0740.196SV28Mean0.1733.08750.3520.0421.9791.27012.5871.2009.65137.6160.215SV28Max0.2883.78776.3170.0692.6551.68419.0791.65417.03840.3930.241Mp04Min0.4302.04319.2670.1033.1470.5414.8160.4903.34533.3510.195Mp04Mean0.7382.48128.5680.1774.5540.8747.1410.8245.62637.6450.208Mp04Max1.1512.85042.0220.2765.8581.13210.5041.1119.43340.1930.222SV42Min0.0272.58836.2940.0060.2540.9619.0720.8315.85628.2750.121SV42Mean0.0322.86650.0690.0080.3771.13812.5161.0578.65431.3950.171SV42Max0.0473.22670.3480.0110.4791.36117.5861.32915.19333.6460.229Mp20Min0.1412.21118.4370.0340.7190.6874.6090.5943.54528.4730.126Mp20Mean0.1762.31927.2400.0421.1040.7676.8090.6454.69431.6260.153Mp20Max0.2122.61951.8620.0511.7520.98212.9640.7056.59134.3590.177Temporal enhancement layer 2 witha=2 for smoothingSV28Min0.1171.78528.6380.0280.9901.04614.3190.8607.01038.0620.040SV28Mean0.1682.44541.4460.0401.5651.56920.7221.22310.89441.2540.110SV28Max0.3223.33364.0040.0772.3762.24832.0021.64518.11144.0130.148Mp04Min0.7771.25011.1960.1863.0230.5305.5980.4713.71139.1640.027Mp04Mean1.3311.66416.8600.3194.8680.9268.4290.8015.56041.6040.078Mp04Max2.1612.03823.8370.5195.8081.25511.9181.0769.37743.8140.113SV42Min0.0271.69318.4120.0070.1200.9669.2050.7766.49930.1910.053SV42Mean0.0351.95633.2110.0080.2751.18516.6040.9498.19632.7090.126SV42Max0.0552.25946.0770.0130.3661.43223.0371.08612.31534.9840.198Mp20Min0.2641.37711.0690.0630.7990.6695.5340.5653.27230.5500.055Mp20Mean0.3321.45414.5150.0801.1150.7447.2570.6024.26933.2570.105Mp20Max0.4031.63325.8700.0971.6390.91312.9340.6305.27736.0000.129First, we can confirm the improved RD efficiency of
the H.264 SVC temporal layers as compared to the MPEG-4 Part 2 layers based on
the smaller mean frame sizes ranges (for corresponding quality ranges) or,
equivalently, the lower mean bit rate ranges for H.264 SVC. Secondly, the mean
bit rates are significantly lower for the H.264 SVC temporal enhancement layers
as compared to the base layer rates. This is also the case for the MPEG-4 Part
2 enhancement layer rates as compared to the base layer, but to a lesser
extent. The reason is that the base layer consists of large I and P frames (for
both encoders). The assignment of cascading quantizers to the H.264 SVC B
frames is responsible for the enhancement layer differences between the
encoders. As opposed to MPEG-4 Part 2, H.264 SVC introduces cascading
quantization scales that assign larger quantization parameters (lower quality
and equivalently lower bit rate) to B frames belonging to higher temporal
layers. This concept is based on the insight that the temporal base layer
requires higher quality than the next temporal layer, since all other
predictions depend on it. The quality (and bit rate) of each subsequent
temporal layer can be gradually reduced, since fewer layers depend on it. The
quality fluctuation that is introduced within a GoP is not subjectively
noticeable by human observers, as studied in the standard committee. Hence,
H.264 SVC temporal enhancement layers have significantly lower bit rates than
the base layer as compared to MPEG-4 Part 2.The quality analysis, and in particular the CoQV, demonstrates that the quality of the base layer is more variable than the
quality when the first enhancement layer is additionally received by the
decoder. When all layers are received, the CoQV is the lowest. We observe this quality
variability decrease for H.264 SVC and MPEG-4 Part 2.Next, we discuss the frame and GoP size coefficients
of variation CoV and peak-to-mean ratios PtM of the temporal layers. Table2 illustrates
that the CoV and PtM of the unsmoothed frame size traffic, that is,
traffic including zeroes (transmission gaps) for missing frames, are high for
all temporal layers and all encoders. The zero frame sizes are the main reason
behind this high variability. The H.264 SVC values are typically considerably higher
than the MPEG-4 Part 2 values, for example, the H.264 SVC CoV values for the first enhancement layer are as
high as 3.79 while the MPEG-4 Part 2 CoV reaches 2.85. With basic smoothing, the
maximum CoV and PtM values decrease, for example, the maximum CoV of the first enhancement layer of H.264 SVC
decreases to 1.68, while MPEG-4 Part 2's maximum CoV drops to 1.13. The GoP size CoV and PtM values exhibit similar trends, however, the
differences between H.264 SVC and MPEG-4 Part 2 values are smaller, because the CoV and PtM of the GoP size are equal to the CoV and PtM of layers smoothed over the entire GoP (a=16). Nevertheless, a fairly significant increase
of H.264 SVC layer variability remains over MPEG-4 Part 2 with the mean CoVs of the H.264 SVC temporal enhancement layers
being typically 1.5 times larger than the mean CoVs of the MPEG-4 Part 2 layers.In Table3,
we provide an overview of the maximum CoV and PtM values for each temporal layer. The table
includes the maximum of the maximum values, such as maxCoVmax, and the maximum of the mean values, such as maxPtM¯. In every instance, the overall maximum is over all studied quantization scales
(not only the selected quantization scales included in Table 2), while the
inner maximum or mean is over all sequences for a given quantization scale.Table 3
Maximum (over quantization scales) of maximum (over video sequences), and maximum of mean CoV and PtM values of H.264 SVC and MPEG-4 Part 2 temporal
base and enhancement layers (unsmoothed and smoothed (a)).
EncodermaxmaxmaxmaxmaxmaxmaxmaxCoVmaxPtMmaxCoV¯PtM¯CoVmax(a)PtMmax(a)CoV(a)¯PtM(a)¯Temporal base layer witha=4 for smoothingH.264 SVC3.2839.932.9934.181.399.981.228.55MPEG-42.8129.882.6125.211.117.470.976.3Temporal enhancement layer 1 witha=4 for smoothingH.264 SVC3.7979.703.0955.411.6819.921.2713.85MPEG-42.8555.562.4932.611.1313.890.898.15Temporal enhancement layer 2 witha=2 for smoothingH.264 SVC3.3664.002.4442.982.2732.001.5721.49MPEG-42.0426.841.6618.281.2513.420.939.14Table3 clearly demonstrates the higher CoV and PtM values of the H.264 SVC layer traffic as
compared to the MPEG-4 Part 2 traffic. We observe that the first H.264 SVC
enhancement layer has the highest CoV and PtM values among all unsmoothed layers. When
smoothing is applied, the values of the second enhancement layer are highest,
mainly because this layer is smoothed over two frames (a=2) as compared to four frames (a=4) for the other layers. Nevertheless, the
advantage of traffic smoothing to reduce traffic variability is clear when
comparing smoothed to unsmoothed values. After smoothing is applied, the H.264
SVC layers—especially enhancement layer 1 and even more so enhancement layer
2—still exhibit higher variability than MPEG-4 Part 2 layers, making network
transport of H.264 SVC temporal layers more challenging. The main reason for
the increased variability of H.264 SVC temporal layer traffic is attributable
to the improved compression tools (e.g., motion compensated prediction) that
manage to exploit redundancies more efficiently, and therefore are more
amenable to frame content variations.In Figure4, VD curves are depicted for each
temporal layer and the aggregated traffic (single-layer) of the Silence of
the Lambs and Star Wars IV sequences encoded with H.264 SVC and
MPEG-4 Part 2 (G16-B3 GoP structure). We provide VD curves for
unsmoothed and smoothed layer traffic. The VD curves for each temporal layer
represent CoV values as a function of the average PSNR
quality, obtained after decoding the current temporal layer and all lower
layers, as we explained in Section 4.1. The average quality
range increases from the temporal base layer VD curve with a quality range up
to approximately 39 dB, to about 46 dB when the decoder additionally receives
the first temporal enhancement layer, and to roughly 52 dB when the decoder
receives all temporal layers. The figure also includes the VD curve of the
aggregated traffic with values that lie between the individual unsmoothed and
smoothed temporal layer VD curves. When comparing VD curves for the Silence
of the Lambs sequence in Figures 4(a) and 4(c), respectively, for H.264 SVC
and MPEG-4 Part 2, the higher traffic variability (CoV) of H.264 SVC is pronounced. The same applies
to the Star Wars IV sequence VD curves in Figures 4(b) and 4(d).
Additionally, we depict the VD curves for five temporal layers of the G16-B15 GoP structure in Figure 5. We observe even higher CoV values for the unsmoothed layers as compared
to G16-B3.VD curves of three temporal layers (G16-B3 GoP
structure), unsmoothed, smoothed (sm), and aggregated (aggr), for the Silence
of the Lambs and Star Wars IV sequences.
(a)
Silence of the Lambs (SVC)(b)
Star Wars IV (SVC)(c)
Silence of the Lambs (MPEG-4)(d)
Star Wars IV (MPEG-4)VD curves of five temporal layers (G16-B15 GoP
structure), unsmoothed, smoothed (sm), and aggregated (aggr), for the Silence
of the Lambs and Star Wars IV sequences.
(a)
Silence of the Lambs (SVC)(b)
Star Wars IV (SVC)(c)
Silence of the Lambs (MPEG-4)(d)
Star Wars IV (MPEG-4)


## 4.1. Temporal Layer Basics

TheG16-B3 GoP structure implies the repetition of the frame type pattern I0B2B1B2P0B2B1B2P0B2B1B2P0B2B1B2, whereby the subscripts denote the temporal layers (0, 1, 2) to which a frame
belongs. The temporal base layer (0) is therefore I0000P0000P0000P0000, with zeroes replacing the dropped B frames of temporal enhancement layers 1 and
2. The first temporal enhancement layer is 00B1000B1000B1000B10 and the second enhancement layer is 0B20B20B20B20B20B20B20B2.In case of our CIF sequences at a frame rate of 30
frames per second (fps), the temporal base layer represents a stream with a
frame rate of 7.5 fps, the combination (aggregation) of the base layer and the
first enhancement layer increases the frame rate to 15 fps, and the reception
of the second enhancement layer results in the full frame rate of 30 fps. We
note that the temporal base layer frames are required for decoding enhancement
layer 1 frames, and that enhancement layer 2 frames need both lower layers to
be decoded.Let us examine the video quality associated with
receiving certain temporal layers. Clearly, the average PSNR video quality of
the combination of all temporal layers, that is, of the aggregated traffic, is
equal to the average quality of the single-layer video stream. However, if we
would simply average the PSNR values (Q) of the base layer frames (QI0QP0QP0QP0), then this average would be unrealistically
high compared to the average of the corresponding single-layer (30 fps) stream,
since the subjective quality impression of human observers is much lower for
the frame rate of 7.5 fps. In order to include this perceptual quality
degradation in the PSNR measurement, we assume that the decoder duplicates a
received base layer frame until the next frame is received and decoded. The
result is the duplicated base layer frame sequence I00I00I00I00P04P04P04P04P08P08P08P08P012P012P012P012, where the upper index represents the duplicated frame number. This sequence has
a frame rate of 30 fps.The PSNR value of a duplicated framen located at frame number i is calculated based on the MSE between this duplicated frame n and the original frame i from the original (uncompressed) sequence.
This PSNR value reflects the subjective distortion that occurs when jerky
sequences consisting of duplicated frames are viewed by human observers. In
general, the perceived video quality of a sequence is high if the average PSNR
is high and the quality variation is low [5]. When there is low motion activity in the successive
frames, that is, when frames are alike (low MSE), then duplication of frames results in
barely noticeable jerkiness. The variation of the PSNR values is therefore also
small. On the other hand, when high motion activity is present, then successive
frames differ substantially and the MSE between successive frames is large, as well as
the quality variation. The computed overall PSNR average therefore sufficiently
incorporates the perceptual video quality reduction due to the reduced frame
rate (jerkiness).We apply the same principle to the computation of the
average quality when the temporal base and first enhancement layer are received
and decoded. This means that the following frame sequence is displayed:I00I00B12B12P04P04B16B16P08P08B110B110P012P012B114B114. The combination of all temporal layers results in displaying the sequence I00B21B12B23P04B25B16B27P08B29B110B211P012B213B114B215, which is the single-layer sequence.Before we analyze the temporal layer traffic
statistics, we describe the simple smoothing that we apply to the temporal base
and enhancement layers to decrease the traffic variability. LetXi denote the frame size (bytes) of frame i. Since there are large transmission gaps between frames of the base layer, we
can redistribute the frame sizes over these gaps by dividing the frame size by
four, and hence sending a quarter of each base layer frame during one frame
period: X0I0/4,X0I0/4,X0I0/4,X0I0/4,X4P0/4,X4P0/4,X4P0/4,X4P0/4,X8P0/4, X8P0/4,X8P0/4,X8P0/4,…. Equivalently, we say that we have smoothed the temporal base layer traffic over a=4 frames. Analogously, the first enhancement
layer traffic is smoothed over a=4 frames: X2B1/4,X2B1/4,X2B1/4,X2B1/4,X6B1/4, X6B1/4,X6B1/4,X6B1/4,X10B1/4,X10B1/4,X10B1/4,X10B1/4,X14B1/4, X14B1/4,X14B1/4,X14B1/4. The second layer is smoothed over a=2 frames since only one frame is missing in
between the B frames of this layer: X1B2/2,X1B2/2,X3B2/2, X3B2/2,X5B2/2,X5B2/2,X7B2/2,X7B2/2,X9B2/2,X9B2/2,X11B2/2, X11B2/2,X13B2/2,X13B2/2,X15B2/2,X15B2/2. This basic smoothing introduces extra decoding delays, but mitigates to some
extent the high rate variability as we demonstrate in the next section.


## 4.2. Results and Discussion

We treat each
temporal layer separately in the following analysis, except for the layer
quality where we assume the reception of all lower layers. The aggregation of
all layers is equivalent to the single-layer case, which is analyzed in detail
in [36]. The main reason
for treating each layer separately is that streaming protocols, such as the
Real Time Protocol [76, 77], typically packetize and stream each layer separately
to allow for different treatment of the layers in the network.In Table2,
we summarize traffic and quality statistics of the temporal base layer and the
two temporal enhancement layers included in the G16-B3 GoP structure.
The table includes frame size, bit rate, smoothed frame size, GoP size, and
video quality statistics. We estimate these statistics based on the five long
CIF sequences that we encode with H.264 SVC and MPEG-4 Part 2. In the first
column of Table 2, the encoding mode is specified by a code representing the
encoder (SV for H.264 SVC and Mp for MPEG-4 Part 2) and the
quantization scale. For each encoder, we present min/mean/max values (computed
across the five sequences) for two selected quantization scales that result in
approximately equal PSNR quality (max-to-min) ranges. For example, the base
layer quantization scale 28 for H.264 SVC results in the PSNR quality range
29.3–36.6 dB, and the quantization scale 4 for MPEG-4 Part 2 results in the
quality range 29.2–36.5 dB. We compare the various statistical quantities in
Table 2 based on matching quality ranges between encoders. Detailed results
for the full range of studied quantization scales, which gives the RD and VD curves
presented in this paper, are available in [73, 74].Table 2
Traffic statistics for the layers of temporal scalability encodings using H.264 SVC and MPEG-4 Part 2 for selected
quantization scales with GoP structure G16B3.
Frame SizeBit RateSmoothed (a)GoP SizeFrame QualityEnc.MeanCoVXPtMXMeanPeakCoVX(a)PtMX(a)CoVYPtMYMeanCoQVModeX¯SXX¯XmaxX¯X¯TXmaxTSX(a)X¯Xmax(a)X¯SYY¯YmaxY¯Q¯SQQ¯[kbyte][Mbps][Mbps][dB]Temporal base layer witha=4 for smoothingSV28Min0.6542.13415.9820.1573.2070.6233.9960.3652.49429.3070.256SV28Mean1.2692.54621.4730.3056.2000.9255.3690.6473.61433.9320.286SV28Max1.9302.87830.1930.4638.5041.1497.5490.8855.34036.6390.316Mp04Min0.9352.12017.5390.2245.6840.6114.3860.3452.37029.2300.256Mp04Mean1.6732.45821.6460.4018.2720.8665.4120.6113.47533.8340.282Mp04Max2.4682.69726.3440.59210.9791.0346.5870.8124.95836.4890.307SV42Min0.1372.60927.3630.0330.9570.9766.8430.3842.58026.3240.167SV42Mean0.2292.97531.5740.0551.6891.2067.8950.6554.14929.6780.218SV42Max0.3273.27639.9330.0782.2301.3909.9850.9066.58732.0230.268Mp20Min0.2672.35420.6360.0641.5290.7975.1600.3592.25326.3440.169Mp20Mean0.3922.46322.3450.0942.0940.8755.5870.4933.25629.6920.201Mp20Max0.5152.59623.8580.1242.8320.9675.9660.6124.28132.3450.230Temporal enhancement layer 1 witha=4 for smoothingSV28Min0.1112.56936.2560.0271.4490.9489.0620.8526.60933.0740.196SV28Mean0.1733.08750.3520.0421.9791.27012.5871.2009.65137.6160.215SV28Max0.2883.78776.3170.0692.6551.68419.0791.65417.03840.3930.241Mp04Min0.4302.04319.2670.1033.1470.5414.8160.4903.34533.3510.195Mp04Mean0.7382.48128.5680.1774.5540.8747.1410.8245.62637.6450.208Mp04Max1.1512.85042.0220.2765.8581.13210.5041.1119.43340.1930.222SV42Min0.0272.58836.2940.0060.2540.9619.0720.8315.85628.2750.121SV42Mean0.0322.86650.0690.0080.3771.13812.5161.0578.65431.3950.171SV42Max0.0473.22670.3480.0110.4791.36117.5861.32915.19333.6460.229Mp20Min0.1412.21118.4370.0340.7190.6874.6090.5943.54528.4730.126Mp20Mean0.1762.31927.2400.0421.1040.7676.8090.6454.69431.6260.153Mp20Max0.2122.61951.8620.0511.7520.98212.9640.7056.59134.3590.177Temporal enhancement layer 2 witha=2 for smoothingSV28Min0.1171.78528.6380.0280.9901.04614.3190.8607.01038.0620.040SV28Mean0.1682.44541.4460.0401.5651.56920.7221.22310.89441.2540.110SV28Max0.3223.33364.0040.0772.3762.24832.0021.64518.11144.0130.148Mp04Min0.7771.25011.1960.1863.0230.5305.5980.4713.71139.1640.027Mp04Mean1.3311.66416.8600.3194.8680.9268.4290.8015.56041.6040.078Mp04Max2.1612.03823.8370.5195.8081.25511.9181.0769.37743.8140.113SV42Min0.0271.69318.4120.0070.1200.9669.2050.7766.49930.1910.053SV42Mean0.0351.95633.2110.0080.2751.18516.6040.9498.19632.7090.126SV42Max0.0552.25946.0770.0130.3661.43223.0371.08612.31534.9840.198Mp20Min0.2641.37711.0690.0630.7990.6695.5340.5653.27230.5500.055Mp20Mean0.3321.45414.5150.0801.1150.7447.2570.6024.26933.2570.105Mp20Max0.4031.63325.8700.0971.6390.91312.9340.6305.27736.0000.129First, we can confirm the improved RD efficiency of
the H.264 SVC temporal layers as compared to the MPEG-4 Part 2 layers based on
the smaller mean frame sizes ranges (for corresponding quality ranges) or,
equivalently, the lower mean bit rate ranges for H.264 SVC. Secondly, the mean
bit rates are significantly lower for the H.264 SVC temporal enhancement layers
as compared to the base layer rates. This is also the case for the MPEG-4 Part
2 enhancement layer rates as compared to the base layer, but to a lesser
extent. The reason is that the base layer consists of large I and P frames (for
both encoders). The assignment of cascading quantizers to the H.264 SVC B
frames is responsible for the enhancement layer differences between the
encoders. As opposed to MPEG-4 Part 2, H.264 SVC introduces cascading
quantization scales that assign larger quantization parameters (lower quality
and equivalently lower bit rate) to B frames belonging to higher temporal
layers. This concept is based on the insight that the temporal base layer
requires higher quality than the next temporal layer, since all other
predictions depend on it. The quality (and bit rate) of each subsequent
temporal layer can be gradually reduced, since fewer layers depend on it. The
quality fluctuation that is introduced within a GoP is not subjectively
noticeable by human observers, as studied in the standard committee. Hence,
H.264 SVC temporal enhancement layers have significantly lower bit rates than
the base layer as compared to MPEG-4 Part 2.The quality analysis, and in particular the CoQV, demonstrates that the quality of the base layer is more variable than the
quality when the first enhancement layer is additionally received by the
decoder. When all layers are received, the CoQV is the lowest. We observe this quality
variability decrease for H.264 SVC and MPEG-4 Part 2.Next, we discuss the frame and GoP size coefficients
of variation CoV and peak-to-mean ratios PtM of the temporal layers. Table2 illustrates
that the CoV and PtM of the unsmoothed frame size traffic, that is,
traffic including zeroes (transmission gaps) for missing frames, are high for
all temporal layers and all encoders. The zero frame sizes are the main reason
behind this high variability. The H.264 SVC values are typically considerably higher
than the MPEG-4 Part 2 values, for example, the H.264 SVC CoV values for the first enhancement layer are as
high as 3.79 while the MPEG-4 Part 2 CoV reaches 2.85. With basic smoothing, the
maximum CoV and PtM values decrease, for example, the maximum CoV of the first enhancement layer of H.264 SVC
decreases to 1.68, while MPEG-4 Part 2's maximum CoV drops to 1.13. The GoP size CoV and PtM values exhibit similar trends, however, the
differences between H.264 SVC and MPEG-4 Part 2 values are smaller, because the CoV and PtM of the GoP size are equal to the CoV and PtM of layers smoothed over the entire GoP (a=16). Nevertheless, a fairly significant increase
of H.264 SVC layer variability remains over MPEG-4 Part 2 with the mean CoVs of the H.264 SVC temporal enhancement layers
being typically 1.5 times larger than the mean CoVs of the MPEG-4 Part 2 layers.In Table3,
we provide an overview of the maximum CoV and PtM values for each temporal layer. The table
includes the maximum of the maximum values, such as maxCoVmax, and the maximum of the mean values, such as maxPtM¯. In every instance, the overall maximum is over all studied quantization scales
(not only the selected quantization scales included in Table 2), while the
inner maximum or mean is over all sequences for a given quantization scale.Table 3
Maximum (over quantization scales) of maximum (over video sequences), and maximum of mean CoV and PtM values of H.264 SVC and MPEG-4 Part 2 temporal
base and enhancement layers (unsmoothed and smoothed (a)).
EncodermaxmaxmaxmaxmaxmaxmaxmaxCoVmaxPtMmaxCoV¯PtM¯CoVmax(a)PtMmax(a)CoV(a)¯PtM(a)¯Temporal base layer witha=4 for smoothingH.264 SVC3.2839.932.9934.181.399.981.228.55MPEG-42.8129.882.6125.211.117.470.976.3Temporal enhancement layer 1 witha=4 for smoothingH.264 SVC3.7979.703.0955.411.6819.921.2713.85MPEG-42.8555.562.4932.611.1313.890.898.15Temporal enhancement layer 2 witha=2 for smoothingH.264 SVC3.3664.002.4442.982.2732.001.5721.49MPEG-42.0426.841.6618.281.2513.420.939.14Table3 clearly demonstrates the higher CoV and PtM values of the H.264 SVC layer traffic as
compared to the MPEG-4 Part 2 traffic. We observe that the first H.264 SVC
enhancement layer has the highest CoV and PtM values among all unsmoothed layers. When
smoothing is applied, the values of the second enhancement layer are highest,
mainly because this layer is smoothed over two frames (a=2) as compared to four frames (a=4) for the other layers. Nevertheless, the
advantage of traffic smoothing to reduce traffic variability is clear when
comparing smoothed to unsmoothed values. After smoothing is applied, the H.264
SVC layers—especially enhancement layer 1 and even more so enhancement layer
2—still exhibit higher variability than MPEG-4 Part 2 layers, making network
transport of H.264 SVC temporal layers more challenging. The main reason for
the increased variability of H.264 SVC temporal layer traffic is attributable
to the improved compression tools (e.g., motion compensated prediction) that
manage to exploit redundancies more efficiently, and therefore are more
amenable to frame content variations.In Figure4, VD curves are depicted for each
temporal layer and the aggregated traffic (single-layer) of the Silence of
the Lambs and Star Wars IV sequences encoded with H.264 SVC and
MPEG-4 Part 2 (G16-B3 GoP structure). We provide VD curves for
unsmoothed and smoothed layer traffic. The VD curves for each temporal layer
represent CoV values as a function of the average PSNR
quality, obtained after decoding the current temporal layer and all lower
layers, as we explained in Section 4.1. The average quality
range increases from the temporal base layer VD curve with a quality range up
to approximately 39 dB, to about 46 dB when the decoder additionally receives
the first temporal enhancement layer, and to roughly 52 dB when the decoder
receives all temporal layers. The figure also includes the VD curve of the
aggregated traffic with values that lie between the individual unsmoothed and
smoothed temporal layer VD curves. When comparing VD curves for the Silence
of the Lambs sequence in Figures 4(a) and 4(c), respectively, for H.264 SVC
and MPEG-4 Part 2, the higher traffic variability (CoV) of H.264 SVC is pronounced. The same applies
to the Star Wars IV sequence VD curves in Figures 4(b) and 4(d).
Additionally, we depict the VD curves for five temporal layers of the G16-B15 GoP structure in Figure 5. We observe even higher CoV values for the unsmoothed layers as compared
to G16-B3.VD curves of three temporal layers (G16-B3 GoP
structure), unsmoothed, smoothed (sm), and aggregated (aggr), for the Silence
of the Lambs and Star Wars IV sequences.
(a)
Silence of the Lambs (SVC)(b)
Star Wars IV (SVC)(c)
Silence of the Lambs (MPEG-4)(d)
Star Wars IV (MPEG-4)VD curves of five temporal layers (G16-B15 GoP
structure), unsmoothed, smoothed (sm), and aggregated (aggr), for the Silence
of the Lambs and Star Wars IV sequences.
(a)
Silence of the Lambs (SVC)(b)
Star Wars IV (SVC)(c)
Silence of the Lambs (MPEG-4)(d)
Star Wars IV (MPEG-4)


## 5. Spatial Scalability Traffic Analysis

In this
section, we focus on the spatial scalability layers of H.264 SVC and MPEG-4
Part 2, employing GoP structureG16-B3. All five CIF sequences are
downsampled to QCIF (176×144) resolution, which forms the spatial base
layer of the encodings. The CIF layer forms the spatial enhancement layer. The
statistical analysis treats each spatial layer separately, similar to the
temporal layer analysis. We do not consider the temporal scalable layers that
are present in each spatial layer, since they are the subject of the combined
scalability analysis in Section 8. We compare the spatial layer traffic
generated by both encoders, and we compare with single-layer QCIF and CIF
traffic. The latter is warranted by the lower rate-distortion efficiency of
spatial scalable encoding based on interlayer prediction, as compared to
single-layer encoding, even though the H.264 SVC encoding tools represent an
improvement over MPEG-4 Part 2.


### 5.1. Spatial Layer Basics

Since we do not
consider temporal layer issues in this spatial layer analysis, the statistical
processing of each spatial layer and the aggregated traffic follow the
single-layer analysis approach. However, the average quality (PSNR) assigned to
the QCIF layer does not represent the subjective quality perception, as compared
to the CIF layer, if the lower resolution effect is not taken into account.
Therefore, we upsample the decoded spatial QCIF base layer to CIF resolution
and compute the average quality based on the MSE between the upsampled QCIF and the original
(uncompressed) CIF sequence. The decoded CIF sequence is directly compared to
the original sequence. This approach is warranted for receivers with CIF
resolution displays, requiring upsampling of QCIF video streams to fit the
display size. We realize that the applied upsampling technique plays a role in
the subjective quality of the upsampled QCIF sequence. However for our
practical traffic study, this is of a lesser concern. We also clarify that the
quality that we associate with the spatial enhancement layer is identical to
the quality of the aggregated traffic (base and enhancement layers), since the
enhancement layer is only decodable if the spatial base layer has been
received.


### 5.2. Results and Discussion

In Table4, we
provide example H.264 SVC and MPEG-4 Part 2 traffic statistics (min/mean/max
values across sequences as in Section 4) of the spatial base layer, spatial
enhancement layer, the aggregated traffic, and single-layer QCIF and CIF
traffic for comparison with the spatial layers. In the first column of the
table, we specify the encoding mode by an encoder code (SVS for
spatially scalable H.264 SVC and Mp4S for spatially scalable MPEG-4 Part
2) and the quantization scale.Table 4
Traffic statistics for base (QCIF), and enhancement (CIF) layers, and aggregated traffic of spatial scalability encodings
using H.264 SVC and MPEG-4 Part 2 for selected quantization scales.
Frame sizeBit rateGoP sizeFrame qualityEnc.MeanCoVXPtMXMeanPeakCoVYPtMYMeanCoQVModeX¯SXX¯XmaxX¯X¯TXmaxTSYY¯YmaxY¯Q¯SQQ¯[kbyte][Mbps][Mbps][dB]Spatial base layer (QCIF)SVS28Min0.3461.49511.1950.0831.2930.4882.88327.9160.111SVS28Mean0.6321.75715.6030.1522.1630.6646.34431.9300.146SVS28Max0.8902.03325.2070.2142.8860.90111.23536.4930.161Mp4S04Min0.6390.9517.6100.1532.4440.5573.56027.9400.111Mp4S04Mean1.2321.18611.5660.2963.0090.7306.32931.9060.145Mp4S04Max1.7261.49216.8460.4143.6161.00611.75736.4340.161SVS42Min0.0841.55815.1140.0200.4720.4392.87625.4830.093SVS42Mean0.1351.75620.8310.0320.6450.5356.24128.5160.132SVS42Max0.1862.20126.9930.0450.8350.6309.83731.6850.165Mp4S20Min0.1681.0289.5270.0400.5980.4862.77625.4860.104Mp4S20Mean0.2451.18312.8760.0590.7400.5425.06428.7410.139Mp4S20Max0.3231.47114.8030.0781.0130.5777.35932.2890.185Single-layer for comparison with spatial base layer
(QCIF)SV28Mean0.6301.75815.5970.1512.1570.6646.36031.9320.146Mp04Mean1.2301.18811.5430.2952.9940.7316.34231.9090.145SV42Mean0.1341.75820.9310.0320.6440.5316.19828.5320.132Mp20Mean0.2441.19312.9380.0580.7400.5495.08428.7360.139Spatial enhancement layer (CIF)SVS28Min0.4271.61214.1710.1032.3160.5392.80537.0680.047SVS28Mean0.9622.04322.9810.2314.4680.7586.58139.6680.091SVS28Max1.4142.60944.3290.3395.5431.17514.15341.8550.111Mp4S04Min1.5750.6236.5780.3784.2160.4472.89438.9060.030Mp4S04Mean3.5780.98710.7190.8597.6290.7275.13740.8560.079Mp4S04Max5.3721.42219.9851.2899.4771.04810.10942.7170.105SVS38Min0.1311.86121.4100.0320.8300.4732.89431.3650.062SVS38Mean0.2622.16028.1910.0631.6060.6786.05633.5270.101SVS38Max0.3742.49147.0820.0902.0470.98112.23835.7790.129Mp4S16Min0.3570.7717.9760.0861.4640.4633.21730.7410.063Mp4S16Mean0.6670.93112.5940.1601.7870.6415.05633.2470.095Mp4S16Max0.9541.14818.2050.2292.2620.8079.41335.9760.124Aggregated (base + enhancement) spatial traffic
(CIF)SVS28Min0.7731.54512.8880.1853.3740.5102.77337.0680.047SVS28Mean1.5941.90019.4090.3836.5080.7006.36839.6680.091SVS28Max2.3042.32035.6780.5537.7711.03712.78841.8550.111Mp4S04Min2.2140.6726.9190.5316.6600.4533.00138.9060.030Mp4S04Mean4.8100.98910.7871.15410.4620.7105.34440.8560.079Mp4S04Max6.9831.38019.0691.67612.0191.02110.58542.7170.105SVS38Min0.2521.76119.5580.0611.4610.4602.73531.3650.062SVS38Mean0.4681.99824.5830.1122.5500.6256.27633.5270.101SVS38Max0.6622.35938.7730.1593.1070.86411.89735.7790.129Mp4S16Min0.5450.8589.2860.1311.9350.4672.87030.7410.063Mp4S16Mean0.9530.96412.3340.2292.6270.6055.15433.2470.095Mp4S16Max1.3431.08416.3300.3223.4890.7449.25635.9760.124Single-layer for comparison with aggregated traffic
(CIF)SV28Mean1.6161.92219.9870.3886.7800.6806.28940.9790.082Mp04Mean3.7231.07611.7510.8949.1820.6815.77941.4850.064SV42Mean0.2992.14928.0390.0721.8700.6366.84632.5650.099Mp20Mean0.9220.94410.6870.2212.3390.4854.12733.3770.094We first analyze the spatial base layer traffic,
comparing the mean frame sizes and mean bit rates of the H.264 SVC spatial base
layer with the MPEG-4 Part 2 base layer for approximately the same quality
ranges. We confirm the improved RD efficiency of H.264 SVC. The average
qualities are overall quite low, since we used spatial upsampling to compute
CIF resolution qualities, as explained in Section5.1. The coefficient of quality variation CoQV is in the range of 0.11–0.19 for both
encoders. For all spatial layers, Table 5 provides maximum-of-maximum
and maximum-of-mean values for the CoV and PtM across all quantization scales and sequences.
From the spatial base layer values, we observe overall significantly larger CoV and PtM values for H.264 SVC as compared to MPEG-4
Part 2, making the network transport of the H.264 SVC spatial base layer
challenging.Table 5
Maximum (across
quantization scales) of maximum (across five video sequences), and maximum of
mean CoV and PtM values of H.264 SVC and MPEG-4 Part 2 spatial base and enhancement layers, and aggregated traffic.
EncodermaxmaxmaxmaxCoVmaxPtMmaxCoV¯PtM¯Spatial base layer (QCIF)H.264 SVC2.2629.951.8820.85MPEG-41.5720.271.3315.00Spatial enhancement layer 1 (CIF)H.264 SVC2.6349.052.1728.19MPEG-41.4321.551.0312.77Aggregated (base + enhancement) traffic
(CIF)H.264 SVC2.3639.952.0124.58MPEG-41.3821.141.0613.08Single-layer (CIF)H.264 SVC2.6345.142.1528.04MPEG-41.4119.211.1513.56In Table4, we additionally summarize statistics of
single-layer QCIF encodings for comparison with the spatial base layer
statistics. Inspection of the values reveals that they are almost perfectly
identical, which confirms that the spatial base layer is encoded independently
from the spatial enhancement layers, and identical to single-layer encoding.
The reason is that the interlayer tools predict the spatial enhancement layer
employing the base layer and the latter is not predicted from the enhancement
layer information. Therefore, the spatial base layer statistics follow
single-layer trends that are extensively studied in [36].Examples of H.264 SVC and MPEG-4 Part 2 traffic
statistics of the spatial enhancement layer are summarized in Table4. We also
provide single-layer CIF statistics for comparison. For H.264 SVC, the average
enhancement layer bit rate is more than twice the bit rate of the base layer
for the highest qualities and converges to about the same bit rate for the
lowest qualities, see [73]. For MPEG-4 Part 2, the enhancement layer bit rate is
always significantly larger than the base layer rate. This is explained by the
enhanced coding efficiency of H.264 SVC's interlayer prediction tools.The enhancement layer average qualities extend to high
qualities since the complete CIF resolution is decodable by receivers. The COQV values are about 0.04–0.13, which is lower
than the base layer quality variability. Table5 provides maximum-of-maximum
and maximum-of-mean CoV and PtM enhancement layer values, which are typically
twice as large or larger for H.264 SVC than for MPEG-4 Part 2. Furthermore, the
H.264 SVC spatial enhancement layer has larger CoV and PtM values than the SVC base layer, while MPEG-4
Part 2 enhancement values are comparable to or lower than the base layer
values. Secondly, the CoV and PtM enhancement layer values are only slightly
larger than or comparable to single-layer CIF values in Table 5, for both H.264
SVC and MPEG-4 Part 2.Next, we discuss the aggregated traffic statistics
provided in Table4, and compare with the enhancement layer and single-layer
values. The mean frame sizes and bit rates are equal to the sum of the corresponding
base and enhancement layer values. The quality statistics are identical to
those of the enhancement layer, as discussed in Section 5.1. From Table 5, we again observe significantly larger maximum CoV and PtM values for the H.264 SVC aggregated traffic as
compared to MPEG-4 Part 2. Compared to the SVC enhancement layer, the CoV and PtM values of the aggregated traffic are generally
somewhat lower. Comparing the aggregated traffic statistics to the single-layer
values reveals that the variabilities of the aggregate traffic are somewhat
lower than the variabilities of the single-layer traffic.In Figure6, we depict VD
curves of the spatial layers (QCIF and CIF) and the aggregated traffic,
alongside the single-layer VD curves, for the Silence of the Lambs and Star
Wars IV sequences encoded with H.264 SVC and MPEG-4 Part 2. We observe that
the base layer and corresponding QCIF single-layer VD curves are identical for
all sequences and encoders, as expected. Comparing Figures 6(a) and 6(c) for Silence
of the Lambs encoded with H.264 SVC and MPEG-4 Part 2, clearly reveals the
higher variability of the H.264 SVC base layer traffic. This is also observable
in Figures 6(b) and 6(d) for the Star Wars IV sequence. The enhancement
layer VD curves for H.264 SVC are above the MPEG-4 Part 2 curves in all cases.
The VD curves of the aggregated traffic are the combined result of the base and
enhancement layer variabilities, and as such, they are generally positioned
between these two VD curves. An interesting distinction between H.264/SVC and
MPEG-4 Part 2 is that the MPEG-4 layer 0 QCIF streams have higher traffic
variabilities than the corresponding MPEG-4 layer 1 CIF streams. With H.264
SVC, this relationships is reversed, that is, the layer 1 CIF H.264 SVC streams
have higher variability than the corresponding H.264 SVC layer 0 QCIF streams,
further underscoring the high-traffic variability of the spatial enhancement
layer of H.264 SVC.VD curves of two spatial layers (0=QCIF, 1=CIF), the
aggregated traffic (aggr), and single-layer traffic (single), for the Silence
of the Lambs and Star Wars IV sequences (G16-B3 GoP
structure).
(a)
Silence of the Lambs (SVC)(b)
Star Wars IV (SVC)(c)
Silence of the Lambs (MPEG-4)(d)
Star Wars IV (MPEG-4)


## 5.1. Spatial Layer Basics

Since we do not
consider temporal layer issues in this spatial layer analysis, the statistical
processing of each spatial layer and the aggregated traffic follow the
single-layer analysis approach. However, the average quality (PSNR) assigned to
the QCIF layer does not represent the subjective quality perception, as compared
to the CIF layer, if the lower resolution effect is not taken into account.
Therefore, we upsample the decoded spatial QCIF base layer to CIF resolution
and compute the average quality based on the MSE between the upsampled QCIF and the original
(uncompressed) CIF sequence. The decoded CIF sequence is directly compared to
the original sequence. This approach is warranted for receivers with CIF
resolution displays, requiring upsampling of QCIF video streams to fit the
display size. We realize that the applied upsampling technique plays a role in
the subjective quality of the upsampled QCIF sequence. However for our
practical traffic study, this is of a lesser concern. We also clarify that the
quality that we associate with the spatial enhancement layer is identical to
the quality of the aggregated traffic (base and enhancement layers), since the
enhancement layer is only decodable if the spatial base layer has been
received.


## 5.2. Results and Discussion

In Table4, we
provide example H.264 SVC and MPEG-4 Part 2 traffic statistics (min/mean/max
values across sequences as in Section 4) of the spatial base layer, spatial
enhancement layer, the aggregated traffic, and single-layer QCIF and CIF
traffic for comparison with the spatial layers. In the first column of the
table, we specify the encoding mode by an encoder code (SVS for
spatially scalable H.264 SVC and Mp4S for spatially scalable MPEG-4 Part
2) and the quantization scale.Table 4
Traffic statistics for base (QCIF), and enhancement (CIF) layers, and aggregated traffic of spatial scalability encodings
using H.264 SVC and MPEG-4 Part 2 for selected quantization scales.
Frame sizeBit rateGoP sizeFrame qualityEnc.MeanCoVXPtMXMeanPeakCoVYPtMYMeanCoQVModeX¯SXX¯XmaxX¯X¯TXmaxTSYY¯YmaxY¯Q¯SQQ¯[kbyte][Mbps][Mbps][dB]Spatial base layer (QCIF)SVS28Min0.3461.49511.1950.0831.2930.4882.88327.9160.111SVS28Mean0.6321.75715.6030.1522.1630.6646.34431.9300.146SVS28Max0.8902.03325.2070.2142.8860.90111.23536.4930.161Mp4S04Min0.6390.9517.6100.1532.4440.5573.56027.9400.111Mp4S04Mean1.2321.18611.5660.2963.0090.7306.32931.9060.145Mp4S04Max1.7261.49216.8460.4143.6161.00611.75736.4340.161SVS42Min0.0841.55815.1140.0200.4720.4392.87625.4830.093SVS42Mean0.1351.75620.8310.0320.6450.5356.24128.5160.132SVS42Max0.1862.20126.9930.0450.8350.6309.83731.6850.165Mp4S20Min0.1681.0289.5270.0400.5980.4862.77625.4860.104Mp4S20Mean0.2451.18312.8760.0590.7400.5425.06428.7410.139Mp4S20Max0.3231.47114.8030.0781.0130.5777.35932.2890.185Single-layer for comparison with spatial base layer
(QCIF)SV28Mean0.6301.75815.5970.1512.1570.6646.36031.9320.146Mp04Mean1.2301.18811.5430.2952.9940.7316.34231.9090.145SV42Mean0.1341.75820.9310.0320.6440.5316.19828.5320.132Mp20Mean0.2441.19312.9380.0580.7400.5495.08428.7360.139Spatial enhancement layer (CIF)SVS28Min0.4271.61214.1710.1032.3160.5392.80537.0680.047SVS28Mean0.9622.04322.9810.2314.4680.7586.58139.6680.091SVS28Max1.4142.60944.3290.3395.5431.17514.15341.8550.111Mp4S04Min1.5750.6236.5780.3784.2160.4472.89438.9060.030Mp4S04Mean3.5780.98710.7190.8597.6290.7275.13740.8560.079Mp4S04Max5.3721.42219.9851.2899.4771.04810.10942.7170.105SVS38Min0.1311.86121.4100.0320.8300.4732.89431.3650.062SVS38Mean0.2622.16028.1910.0631.6060.6786.05633.5270.101SVS38Max0.3742.49147.0820.0902.0470.98112.23835.7790.129Mp4S16Min0.3570.7717.9760.0861.4640.4633.21730.7410.063Mp4S16Mean0.6670.93112.5940.1601.7870.6415.05633.2470.095Mp4S16Max0.9541.14818.2050.2292.2620.8079.41335.9760.124Aggregated (base + enhancement) spatial traffic
(CIF)SVS28Min0.7731.54512.8880.1853.3740.5102.77337.0680.047SVS28Mean1.5941.90019.4090.3836.5080.7006.36839.6680.091SVS28Max2.3042.32035.6780.5537.7711.03712.78841.8550.111Mp4S04Min2.2140.6726.9190.5316.6600.4533.00138.9060.030Mp4S04Mean4.8100.98910.7871.15410.4620.7105.34440.8560.079Mp4S04Max6.9831.38019.0691.67612.0191.02110.58542.7170.105SVS38Min0.2521.76119.5580.0611.4610.4602.73531.3650.062SVS38Mean0.4681.99824.5830.1122.5500.6256.27633.5270.101SVS38Max0.6622.35938.7730.1593.1070.86411.89735.7790.129Mp4S16Min0.5450.8589.2860.1311.9350.4672.87030.7410.063Mp4S16Mean0.9530.96412.3340.2292.6270.6055.15433.2470.095Mp4S16Max1.3431.08416.3300.3223.4890.7449.25635.9760.124Single-layer for comparison with aggregated traffic
(CIF)SV28Mean1.6161.92219.9870.3886.7800.6806.28940.9790.082Mp04Mean3.7231.07611.7510.8949.1820.6815.77941.4850.064SV42Mean0.2992.14928.0390.0721.8700.6366.84632.5650.099Mp20Mean0.9220.94410.6870.2212.3390.4854.12733.3770.094We first analyze the spatial base layer traffic,
comparing the mean frame sizes and mean bit rates of the H.264 SVC spatial base
layer with the MPEG-4 Part 2 base layer for approximately the same quality
ranges. We confirm the improved RD efficiency of H.264 SVC. The average
qualities are overall quite low, since we used spatial upsampling to compute
CIF resolution qualities, as explained in Section5.1. The coefficient of quality variation CoQV is in the range of 0.11–0.19 for both
encoders. For all spatial layers, Table 5 provides maximum-of-maximum
and maximum-of-mean values for the CoV and PtM across all quantization scales and sequences.
From the spatial base layer values, we observe overall significantly larger CoV and PtM values for H.264 SVC as compared to MPEG-4
Part 2, making the network transport of the H.264 SVC spatial base layer
challenging.Table 5
Maximum (across
quantization scales) of maximum (across five video sequences), and maximum of
mean CoV and PtM values of H.264 SVC and MPEG-4 Part 2 spatial base and enhancement layers, and aggregated traffic.
EncodermaxmaxmaxmaxCoVmaxPtMmaxCoV¯PtM¯Spatial base layer (QCIF)H.264 SVC2.2629.951.8820.85MPEG-41.5720.271.3315.00Spatial enhancement layer 1 (CIF)H.264 SVC2.6349.052.1728.19MPEG-41.4321.551.0312.77Aggregated (base + enhancement) traffic
(CIF)H.264 SVC2.3639.952.0124.58MPEG-41.3821.141.0613.08Single-layer (CIF)H.264 SVC2.6345.142.1528.04MPEG-41.4119.211.1513.56In Table4, we additionally summarize statistics of
single-layer QCIF encodings for comparison with the spatial base layer
statistics. Inspection of the values reveals that they are almost perfectly
identical, which confirms that the spatial base layer is encoded independently
from the spatial enhancement layers, and identical to single-layer encoding.
The reason is that the interlayer tools predict the spatial enhancement layer
employing the base layer and the latter is not predicted from the enhancement
layer information. Therefore, the spatial base layer statistics follow
single-layer trends that are extensively studied in [36].Examples of H.264 SVC and MPEG-4 Part 2 traffic
statistics of the spatial enhancement layer are summarized in Table4. We also
provide single-layer CIF statistics for comparison. For H.264 SVC, the average
enhancement layer bit rate is more than twice the bit rate of the base layer
for the highest qualities and converges to about the same bit rate for the
lowest qualities, see [73]. For MPEG-4 Part 2, the enhancement layer bit rate is
always significantly larger than the base layer rate. This is explained by the
enhanced coding efficiency of H.264 SVC's interlayer prediction tools.The enhancement layer average qualities extend to high
qualities since the complete CIF resolution is decodable by receivers. The COQV values are about 0.04–0.13, which is lower
than the base layer quality variability. Table5 provides maximum-of-maximum
and maximum-of-mean CoV and PtM enhancement layer values, which are typically
twice as large or larger for H.264 SVC than for MPEG-4 Part 2. Furthermore, the
H.264 SVC spatial enhancement layer has larger CoV and PtM values than the SVC base layer, while MPEG-4
Part 2 enhancement values are comparable to or lower than the base layer
values. Secondly, the CoV and PtM enhancement layer values are only slightly
larger than or comparable to single-layer CIF values in Table 5, for both H.264
SVC and MPEG-4 Part 2.Next, we discuss the aggregated traffic statistics
provided in Table4, and compare with the enhancement layer and single-layer
values. The mean frame sizes and bit rates are equal to the sum of the corresponding
base and enhancement layer values. The quality statistics are identical to
those of the enhancement layer, as discussed in Section 5.1. From Table 5, we again observe significantly larger maximum CoV and PtM values for the H.264 SVC aggregated traffic as
compared to MPEG-4 Part 2. Compared to the SVC enhancement layer, the CoV and PtM values of the aggregated traffic are generally
somewhat lower. Comparing the aggregated traffic statistics to the single-layer
values reveals that the variabilities of the aggregate traffic are somewhat
lower than the variabilities of the single-layer traffic.In Figure6, we depict VD
curves of the spatial layers (QCIF and CIF) and the aggregated traffic,
alongside the single-layer VD curves, for the Silence of the Lambs and Star
Wars IV sequences encoded with H.264 SVC and MPEG-4 Part 2. We observe that
the base layer and corresponding QCIF single-layer VD curves are identical for
all sequences and encoders, as expected. Comparing Figures 6(a) and 6(c) for Silence
of the Lambs encoded with H.264 SVC and MPEG-4 Part 2, clearly reveals the
higher variability of the H.264 SVC base layer traffic. This is also observable
in Figures 6(b) and 6(d) for the Star Wars IV sequence. The enhancement
layer VD curves for H.264 SVC are above the MPEG-4 Part 2 curves in all cases.
The VD curves of the aggregated traffic are the combined result of the base and
enhancement layer variabilities, and as such, they are generally positioned
between these two VD curves. An interesting distinction between H.264/SVC and
MPEG-4 Part 2 is that the MPEG-4 layer 0 QCIF streams have higher traffic
variabilities than the corresponding MPEG-4 layer 1 CIF streams. With H.264
SVC, this relationships is reversed, that is, the layer 1 CIF H.264 SVC streams
have higher variability than the corresponding H.264 SVC layer 0 QCIF streams,
further underscoring the high-traffic variability of the spatial enhancement
layer of H.264 SVC.VD curves of two spatial layers (0=QCIF, 1=CIF), the
aggregated traffic (aggr), and single-layer traffic (single), for the Silence
of the Lambs and Star Wars IV sequences (G16-B3 GoP
structure).
(a)
Silence of the Lambs (SVC)(b)
Star Wars IV (SVC)(c)
Silence of the Lambs (MPEG-4)(d)
Star Wars IV (MPEG-4)


## 6. Fine Granular Scalability Traffic Analysis

We compare
H.264 SVC fine granularity scalability (SVC FGS) with MPEG-4 Part 2 FGS (MPEG-4
FGS) traffic based on GoP structureG16-B3. We analyze the base and
enhancement layers separately and do not consider the temporal layers in this
section, since they are the subject of our combined FGS-temporal analysis in
Section 8.


### 6.1. FGS Layer Basics

For MPEG-4 FGS,
many possible FGS structures can be used such as basic FGS, FGS temporal
(FGST), combined FGS-FGST, and multilayer FGST, which are detailed in [71]. In this study, we use the
basic FGS structure, depicted in Figure 3(c), with one FGS enhancement layer
frame for every base layer frame. We employ the H.264 FGS prediction loop
illustrated in Figure 3(b), which is closed with respect to both the highest
and lowest quality points.The subsequent FGS analysis is based on the CIF video
sequencesSilence of the Lambs, Star Wars IV, NBC 12 News,
and Sony Demo. We configured both encoders with one FGS enhancement
layer and specified the base layer quantization scale. We study the traffic
characteristics of the FGS base layer, the untruncated and the truncated
enhancement layer, as well as the aggregated (base + enhancement) traffic.


### 6.2. Results and Discussion

We analyze the
statistics of base, enhancement, and aggregated traffic separately, in
correspondence with the various possibilities of reception at the decoder. For
selected base layer quantization scales, we present values of SVC FGS and
MPEG-4 FGS traffic statistics for overlapping quality ranges in Table6. We
provide minimum, mean, and maximum (across the five video sequences) values of
the traffic statistics. In the first column of the table, the encoder
quantization scales are specified for MPEG-4 FGS (Mp4f) and SVC FGS (SVF). In Table 7,
we present the maximum values across quantization scales and sequences. We
observe from Table 6 a significant compression efficiency improvement in the
base layer due to the improved tools in SVC FGS. These improved compression
tools result in very high traffic variabilities for the SVC FGS base layer with
maximum CoV and PtM values up to 2.5 and 39.9, as compared to up
to 1.5 and 22.14 for MPEG-4 FGS, as observed in Table 7. The maximum of means
values are similarly higher for SVC FGS. From these values, we conclude that
significant traffic variability is introduced in the SVC FGS base layer as
compared to MPEG-4 FGS. When comparing with single-layer H.264 SVC, see
[74], we find that the
base layer of SVC FGS (Table 6) is nearly identical since the prediction
structure of both utilizes a closed loop.Table 6
Frame size, GoP size, bit rate, and quality statistics of FGS encodings using SVC FGS and MPEG-4 FGS for selected base
layer quantization scales.
Frame sizeBit rateGoP sizeFrame qualityEnc.MeanCoVXPtMXMeanPeakCoVYPtMYMeanCoQVmodeX¯SXX¯XmaxX¯X¯TXmaxTSYY¯YmaxY¯Q¯SQQ¯[kbyte][Mbps][Mbps][dB]FGS base layerSVF24Min1.3841.45111.8850.3324.2270.4392.61738.1940.041SVF24Mean2.2131.78415.7420.5317.8310.6635.41642.6750.086SVF24Max3.5952.16325.3510.86311.2211.0029.49745.6270.111Mp4f04Min1.8810.7577.4090.4515.6980.4983.05338.9780.033Mp4f04Mean3.7411.11611.9150.8988.9760.7056.01441.3030.080Mp4f04Max5.8841.47018.9921.41211.1701.05511.44243.3320.103SVF38Min0.2711.78718.8070.0651.3800.4852.75930.3530.066SVF38Mean0.4212.11825.5710.1012.4130.6356.84334.2760.092SVF38Max0.6442.48842.0570.1553.3890.93612.36837.2660.116Mp4f16Min0.5681.12511.6160.1361.9940.4932.72231.5650.061Mp4f16Mean0.8811.25713.9750.2112.8210.5935.50634.1440.087Mp4f16Max1.2841.43117.2290.3083.5810.7208.63337.0450.115FGS enhancement layer (untruncated)SVF24Min2.0981.0404.9360.5043.9580.3382.19340.9490.049SVF24Mean3.2301.3206.9050.7754.8470.4223.07945.8290.072SVF24Max4.6411.57210.0071.1145.6350.5784.55748.4520.088Mp4f02Min2.4040.3281.7080.5771.7380.1991.52744.0640.014Mp4f02Mean4.1290.4962.7400.9912.3640.3972.30846.1570.058Mp4f02Max6.3900.6834.1051.5342.7200.6053.59647.6330.081SVF28Min1.4461.4536.1870.3473.1560.3642.02938.0860.049SVF28Mean2.0891.5918.8040.5014.0620.4403.17943.4810.077SVF28Max3.3811.78813.0130.8115.0200.6154.90346.2790.096Mp4f28Min12.5730.2741.9203.0177.5300.2661.86542.0520.093Mp4f28Mean20.3330.3982.5264.88011.2760.3922.41744.2480.114Mp4f28Max29.8790.5743.9347.17114.1350.5693.74646.2860.134FGS aggregated (base + untruncated enhancement) trafficSVF24Min3.4011.1387.9020.8166.8490.3982.37840.9490.049SVF24Mean5.1281.3989.6321.23111.3250.5183.94045.8600.079SVF24Max8.2361.65913.8831.97715.6190.7246.05548.8010.098Mp4f02Min6.6950.3793.5081.6079.9120.3002.11944.0640.014Mp4f02Mean12.9290.6455.8833.10315.8360.5273.85746.1570.058Mp4f02Max21.1360.9228.5495.07321.8610.8066.82247.6330.081SVF28Min2.3441.5079.3010.5635.2330.4082.25238.0860.049SVF28Mean3.3291.60111.4700.7998.8110.5034.02243.4330.083SVF28Max5.6441.77716.3901.35412.6820.6886.44546.5650.102Mp4f28Min13.1490.2682.0453.1567.9750.2591.84742.0520.093Mp4f28Mean21.0080.3902.5915.04212.0810.3832.36344.2480.114Mp4f28Max30.6010.5603.9067.34415.0170.5543.62546.2860.134Table 7
Maximum-of-maximum, and maximum-of-mean CoV and PtM values of SVC FGS and MPEG-4 FGS base and
enhancement layers, and aggregated traffic.
EncodermaxmaxmaxmaxCoVmaxPtMmaxCoV¯PtM¯FGS base layerH.264 SVC2.539.92.1325.9MPEG-41.522.141.314.8FGS enhancement layer (untruncated)H.264 SVC2.1120.281.8711.9MPEG-40.64.00.492.74Aggregated FGS traffic (base + untruncated
enhancement)H.264 SVC1.9725.51.7615.5MPEG-40.928.540.645.88Table6 also gives selected examples to compare the
untruncated FGS enhancement layers of both encoders. From Table 7, CoV and PtM have maxima up to 2.11 and 20.28,
respectively, for SVC FGS, compared to up to 0.6 and 4.0 for MPEG-4 FGS. The
SVC FGS enhancement layer has been subject to improved compression tools,
resulting in increased variability at the frame level. Analogously, for the
aggregated traffic with untruncated enhancement layer (Table 6), we have a CoV of 1.97 and a PtM of 25.5 for SVC FGS, as compared to 0.92 and
8.54 for MPEG-4 FGS.Next, we examine the RD graphs of the SVC FGS and
MPEG-4 FGS layers. Figure7 depicts the base, untruncated enhancement, and
aggregated traffic (base + untruncated enhancement) RD graphs for SVC FGS and
MPEG-4 FGS encodings of the Silence of the Lambs sequence. The FGS base layer RD graphs are typical
(quality increases monotonically as a function of the bit rate) and demonstrate
the improved RD efficiency of SVC FGS in the base layer. The untruncated
enhancement layer for MPEG-4 FGS contains refinement information allowing
high-quality reconstruction of the frames, resulting in the near-flat RD curve.
The aggregated traffic RD graphs are the summation of the base and untruncated
enhancement layer rates (per quality value).Figure 7
RD curves for
SVC FGS and MPEG-4 FGS base and untruncated enhancement (enh) layers, and
aggregated traffic (aggr) of Silence of the Lambs sequence (G16-B3).To compare MPEG-4 FGS and SVC FGS with various
truncations of the enhancement layer, we use average base layer PSNR qualities
that are approximately equal. ForStar Wars IV, we select quantization
scales 34 and 8, respectively, for SVC FGS and MPEG-4 FGS, corresponding to an
average base layer PSNR of approximately 34 dB. We further choose quantization
scales 38 and 16 for SVC FGS and MPEG-4 FGS corresponding to a PSNR of
approximately 37 dB for the Silence of the Lambs sequence. We truncate
the enhancement layer progressively with 10% increments of the enhancement
layer bit rate.The RD graphs obtained for the aggregated (base + truncated enhancement) traffic for both
sequences are depicted in Figure8(a). The steep rise of the SVC FGS enhancement
layer RD curve for every 10% increment in bit rate is in clear contrast to
MPEG-4 FGS, which has a much lower RD performance with more gradual increments.
This lower RD performance is explained by ignoring the enhancement layer in the
prediction loop of MPEG-4 FGS. This also clearly demonstrates the substantial
coding improvements made to the enhancement layer of SVC FGS, without
significantly increasing the computational complexity (a major concern for
portable devices). We also observe from Figure 8 that the Star Wars IV sequence has a better RD performance, which is consistent with earlier results.
We note that the truncation of the MPEG-4 FGS enhancement layer resulted in
outliers that are included in Figure 8 as disconnected tick marks.RD and VD curves of MPEG-4 FGS and SVC FGS enhancement layers
truncated progressively with 10% increments, forSilence o/t Lambs and Star
Wars IV sequences.
(a)
RD graph (base + truncated enhancement)(b)
VD graph (base + truncated enhancement)(c)
RD graph (truncated enhancement)(d)
VD graph (truncated enhancement)The VD curves illustrate the significant contrast in
variability between SVC FGS and MPEG-4 FGS. These VD curve points correspond to
the RD curve points and represent the variability of the progressively
truncated enhancement layer. For SVC FGS, we observe a marginal decrease in
variability for increasing bit rate. The plots also include the smoothed
traffic (a=4, denoted by sm) VD curves, which show that the high variability of the SVC FGS
stream can be significantly reduced by smoothing. However, the unsmoothed
MPEG-4 FGS curves lie well below the smoothed SVC FGS stream curves, pointing
to the inherently high variability introduced by the SVC FGS encoder. (The Star
Wars IV VD curves for MPEG-4 FGS are above the Silence of the Lambs VD curves in Figure 8(b) due to the higher base layer CoV of Star Wars IV for the considered quantization scale 8.)Although we consider a basic truncation strategy,
which truncates each enhancement layer's progressive refinement (PR) slice by
the same percentage, the traffic variability is still high. This is because the
truncation of each PR slice results in widely variable truncated PR slice sizes
(bytes). The SVC FGS traffic variability is consistently high across the range
of percentages of enhancement layer added to the base layer; an important
characteristic to take into account in the design of transport protocols as the
enhancement layer is typically sent over a more error prone path with respect
to the base layer.


## 6.1. FGS Layer Basics

For MPEG-4 FGS,
many possible FGS structures can be used such as basic FGS, FGS temporal
(FGST), combined FGS-FGST, and multilayer FGST, which are detailed in [71]. In this study, we use the
basic FGS structure, depicted in Figure 3(c), with one FGS enhancement layer
frame for every base layer frame. We employ the H.264 FGS prediction loop
illustrated in Figure 3(b), which is closed with respect to both the highest
and lowest quality points.The subsequent FGS analysis is based on the CIF video
sequencesSilence of the Lambs, Star Wars IV, NBC 12 News,
and Sony Demo. We configured both encoders with one FGS enhancement
layer and specified the base layer quantization scale. We study the traffic
characteristics of the FGS base layer, the untruncated and the truncated
enhancement layer, as well as the aggregated (base + enhancement) traffic.


## 6.2. Results and Discussion

We analyze the
statistics of base, enhancement, and aggregated traffic separately, in
correspondence with the various possibilities of reception at the decoder. For
selected base layer quantization scales, we present values of SVC FGS and
MPEG-4 FGS traffic statistics for overlapping quality ranges in Table6. We
provide minimum, mean, and maximum (across the five video sequences) values of
the traffic statistics. In the first column of the table, the encoder
quantization scales are specified for MPEG-4 FGS (Mp4f) and SVC FGS (SVF). In Table 7,
we present the maximum values across quantization scales and sequences. We
observe from Table 6 a significant compression efficiency improvement in the
base layer due to the improved tools in SVC FGS. These improved compression
tools result in very high traffic variabilities for the SVC FGS base layer with
maximum CoV and PtM values up to 2.5 and 39.9, as compared to up
to 1.5 and 22.14 for MPEG-4 FGS, as observed in Table 7. The maximum of means
values are similarly higher for SVC FGS. From these values, we conclude that
significant traffic variability is introduced in the SVC FGS base layer as
compared to MPEG-4 FGS. When comparing with single-layer H.264 SVC, see
[74], we find that the
base layer of SVC FGS (Table 6) is nearly identical since the prediction
structure of both utilizes a closed loop.Table 6
Frame size, GoP size, bit rate, and quality statistics of FGS encodings using SVC FGS and MPEG-4 FGS for selected base
layer quantization scales.
Frame sizeBit rateGoP sizeFrame qualityEnc.MeanCoVXPtMXMeanPeakCoVYPtMYMeanCoQVmodeX¯SXX¯XmaxX¯X¯TXmaxTSYY¯YmaxY¯Q¯SQQ¯[kbyte][Mbps][Mbps][dB]FGS base layerSVF24Min1.3841.45111.8850.3324.2270.4392.61738.1940.041SVF24Mean2.2131.78415.7420.5317.8310.6635.41642.6750.086SVF24Max3.5952.16325.3510.86311.2211.0029.49745.6270.111Mp4f04Min1.8810.7577.4090.4515.6980.4983.05338.9780.033Mp4f04Mean3.7411.11611.9150.8988.9760.7056.01441.3030.080Mp4f04Max5.8841.47018.9921.41211.1701.05511.44243.3320.103SVF38Min0.2711.78718.8070.0651.3800.4852.75930.3530.066SVF38Mean0.4212.11825.5710.1012.4130.6356.84334.2760.092SVF38Max0.6442.48842.0570.1553.3890.93612.36837.2660.116Mp4f16Min0.5681.12511.6160.1361.9940.4932.72231.5650.061Mp4f16Mean0.8811.25713.9750.2112.8210.5935.50634.1440.087Mp4f16Max1.2841.43117.2290.3083.5810.7208.63337.0450.115FGS enhancement layer (untruncated)SVF24Min2.0981.0404.9360.5043.9580.3382.19340.9490.049SVF24Mean3.2301.3206.9050.7754.8470.4223.07945.8290.072SVF24Max4.6411.57210.0071.1145.6350.5784.55748.4520.088Mp4f02Min2.4040.3281.7080.5771.7380.1991.52744.0640.014Mp4f02Mean4.1290.4962.7400.9912.3640.3972.30846.1570.058Mp4f02Max6.3900.6834.1051.5342.7200.6053.59647.6330.081SVF28Min1.4461.4536.1870.3473.1560.3642.02938.0860.049SVF28Mean2.0891.5918.8040.5014.0620.4403.17943.4810.077SVF28Max3.3811.78813.0130.8115.0200.6154.90346.2790.096Mp4f28Min12.5730.2741.9203.0177.5300.2661.86542.0520.093Mp4f28Mean20.3330.3982.5264.88011.2760.3922.41744.2480.114Mp4f28Max29.8790.5743.9347.17114.1350.5693.74646.2860.134FGS aggregated (base + untruncated enhancement) trafficSVF24Min3.4011.1387.9020.8166.8490.3982.37840.9490.049SVF24Mean5.1281.3989.6321.23111.3250.5183.94045.8600.079SVF24Max8.2361.65913.8831.97715.6190.7246.05548.8010.098Mp4f02Min6.6950.3793.5081.6079.9120.3002.11944.0640.014Mp4f02Mean12.9290.6455.8833.10315.8360.5273.85746.1570.058Mp4f02Max21.1360.9228.5495.07321.8610.8066.82247.6330.081SVF28Min2.3441.5079.3010.5635.2330.4082.25238.0860.049SVF28Mean3.3291.60111.4700.7998.8110.5034.02243.4330.083SVF28Max5.6441.77716.3901.35412.6820.6886.44546.5650.102Mp4f28Min13.1490.2682.0453.1567.9750.2591.84742.0520.093Mp4f28Mean21.0080.3902.5915.04212.0810.3832.36344.2480.114Mp4f28Max30.6010.5603.9067.34415.0170.5543.62546.2860.134Table 7
Maximum-of-maximum, and maximum-of-mean CoV and PtM values of SVC FGS and MPEG-4 FGS base and
enhancement layers, and aggregated traffic.
EncodermaxmaxmaxmaxCoVmaxPtMmaxCoV¯PtM¯FGS base layerH.264 SVC2.539.92.1325.9MPEG-41.522.141.314.8FGS enhancement layer (untruncated)H.264 SVC2.1120.281.8711.9MPEG-40.64.00.492.74Aggregated FGS traffic (base + untruncated
enhancement)H.264 SVC1.9725.51.7615.5MPEG-40.928.540.645.88Table6 also gives selected examples to compare the
untruncated FGS enhancement layers of both encoders. From Table 7, CoV and PtM have maxima up to 2.11 and 20.28,
respectively, for SVC FGS, compared to up to 0.6 and 4.0 for MPEG-4 FGS. The
SVC FGS enhancement layer has been subject to improved compression tools,
resulting in increased variability at the frame level. Analogously, for the
aggregated traffic with untruncated enhancement layer (Table 6), we have a CoV of 1.97 and a PtM of 25.5 for SVC FGS, as compared to 0.92 and
8.54 for MPEG-4 FGS.Next, we examine the RD graphs of the SVC FGS and
MPEG-4 FGS layers. Figure7 depicts the base, untruncated enhancement, and
aggregated traffic (base + untruncated enhancement) RD graphs for SVC FGS and
MPEG-4 FGS encodings of the Silence of the Lambs sequence. The FGS base layer RD graphs are typical
(quality increases monotonically as a function of the bit rate) and demonstrate
the improved RD efficiency of SVC FGS in the base layer. The untruncated
enhancement layer for MPEG-4 FGS contains refinement information allowing
high-quality reconstruction of the frames, resulting in the near-flat RD curve.
The aggregated traffic RD graphs are the summation of the base and untruncated
enhancement layer rates (per quality value).Figure 7
RD curves for
SVC FGS and MPEG-4 FGS base and untruncated enhancement (enh) layers, and
aggregated traffic (aggr) of Silence of the Lambs sequence (G16-B3).To compare MPEG-4 FGS and SVC FGS with various
truncations of the enhancement layer, we use average base layer PSNR qualities
that are approximately equal. ForStar Wars IV, we select quantization
scales 34 and 8, respectively, for SVC FGS and MPEG-4 FGS, corresponding to an
average base layer PSNR of approximately 34 dB. We further choose quantization
scales 38 and 16 for SVC FGS and MPEG-4 FGS corresponding to a PSNR of
approximately 37 dB for the Silence of the Lambs sequence. We truncate
the enhancement layer progressively with 10% increments of the enhancement
layer bit rate.The RD graphs obtained for the aggregated (base + truncated enhancement) traffic for both
sequences are depicted in Figure8(a). The steep rise of the SVC FGS enhancement
layer RD curve for every 10% increment in bit rate is in clear contrast to
MPEG-4 FGS, which has a much lower RD performance with more gradual increments.
This lower RD performance is explained by ignoring the enhancement layer in the
prediction loop of MPEG-4 FGS. This also clearly demonstrates the substantial
coding improvements made to the enhancement layer of SVC FGS, without
significantly increasing the computational complexity (a major concern for
portable devices). We also observe from Figure 8 that the Star Wars IV sequence has a better RD performance, which is consistent with earlier results.
We note that the truncation of the MPEG-4 FGS enhancement layer resulted in
outliers that are included in Figure 8 as disconnected tick marks.RD and VD curves of MPEG-4 FGS and SVC FGS enhancement layers
truncated progressively with 10% increments, forSilence o/t Lambs and Star
Wars IV sequences.
(a)
RD graph (base + truncated enhancement)(b)
VD graph (base + truncated enhancement)(c)
RD graph (truncated enhancement)(d)
VD graph (truncated enhancement)The VD curves illustrate the significant contrast in
variability between SVC FGS and MPEG-4 FGS. These VD curve points correspond to
the RD curve points and represent the variability of the progressively
truncated enhancement layer. For SVC FGS, we observe a marginal decrease in
variability for increasing bit rate. The plots also include the smoothed
traffic (a=4, denoted by sm) VD curves, which show that the high variability of the SVC FGS
stream can be significantly reduced by smoothing. However, the unsmoothed
MPEG-4 FGS curves lie well below the smoothed SVC FGS stream curves, pointing
to the inherently high variability introduced by the SVC FGS encoder. (The Star
Wars IV VD curves for MPEG-4 FGS are above the Silence of the Lambs VD curves in Figure 8(b) due to the higher base layer CoV of Star Wars IV for the considered quantization scale 8.)Although we consider a basic truncation strategy,
which truncates each enhancement layer's progressive refinement (PR) slice by
the same percentage, the traffic variability is still high. This is because the
truncation of each PR slice results in widely variable truncated PR slice sizes
(bytes). The SVC FGS traffic variability is consistently high across the range
of percentages of enhancement layer added to the base layer; an important
characteristic to take into account in the design of transport protocols as the
enhancement layer is typically sent over a more error prone path with respect
to the base layer.


## 7. Medium Grain Scalability Traffic Analysis

In this section, we focus on the medium grain scalability (MGS) mode of H.264 SVC,
employing GoP structureG16-B0, which signifies 15 P frames in between I
frames and no B frames. The resulting MGS base layer with CIF resolution
conforms to the restricted Baseline profile of H.264/AVC. The MGS enhancement
layer adds information that improves the quality of each video frame type up to
the maximum quality encoded in the enhancement layer. Similar to the previous
sections, the statistical analysis treats each layer separately and also
aggregates the traffic in both layers. We compare the layer traffic generated
by H.264 SVC MGS, however, we are not able to compare with equivalent traffic
of MPEG-4 Part 2 since this standard does not include a similar quality scalability
mode.


### 7.1. MGS Layer Basics

The statistical
processing of the base layer, MGS enhancement layer, and the aggregated traffic
follow the single-layer analysis approach. As for spatial scalability and FGS,
the quality that we associate with the MGS enhancement layer is identical to
the quality of the aggregated traffic (base and enhancement layers).The MGS enhancement layer studied in this analysis
supports one quality enhancement with a quantization parameter decrease of 6
(increased quality). We leave the study of multiple MGS quality extraction
points (up to 16) within this enhancement layer for future research as well as
the statistical analysis of theG16-B3 GoP structure.


### 7.2. Results and Discussion

Table8 enumerates example H.264 SVC traffic statistics (min/mean/max) values across
sequences of the base layer, MGS enhancement layer, and the aggregated traffic.
In the first column of the table, we specify the encoding mode by the encoder
code SVM followed by the quantization scale.Table 8
Traffic statistics for base, and enhancement layers, and aggregated traffic of medium grain scalability encodings
using H.264 SVC for selected quantization scales.
Frame sizeBit rateGoP sizeFrame qualityEnc.MeanCoVXPtMXMeanPeakCoVYPtMYMeanCoQVmodeX¯SXX¯XmaxX¯X¯TXmaxTSYY¯YmaxY¯Q¯SQQ¯[kbyte][Mbps][Mbps][dB]Base layerSVM28Min0.7680.9547.9580.1842.5190.5912.98137.4720.039SVM28Mean1.7251.38315.5510.4145.3470.7907.39539.6820.083SVM28Max2.8431.95029.8730.6826.6461.19514.52341.5190.109SVM34Min0.3631.24811.8080.0871.5420.5673.17733.3240.054SVM34Mean0.7461.61320.2480.1793.1840.7818.48535.8160.099SVM34Max1.1842.08935.6310.2844.1041.16316.56237.9040.132SVM38Min0.2231.39014.3890.0531.0730.5453.30330.6070.062SVM38Mean0.4301.69522.2340.1032.0830.7428.80833.2160.098SVM38Max0.6752.05136.1180.1622.6231.07316.60035.4110.147SVM42Min0.1411.40316.7000.0340.7710.5223.23628.2290.069SVM42Mean0.2571.70422.8810.0621.3180.6858.71630.7830.091SVM42Max0.3982.10033.6300.0961.5960.93615.42133.0660.112MGS enhancement layerSVM28Min3.6530.3281.9750.8772.6160.3191.85441.6480.020SVM28Mean6.1350.4932.9011.4723.8390.4862.73143.5890.063SVM28Max9.8950.6994.4532.3754.6910.6924.23945.0960.091SVM34Min2.1180.3972.1880.5081.5280.3872.11537.7150.034SVM34Mean3.7740.5793.5310.9062.8090.5703.39539.5660.077SVM34Max6.5130.8335.6621.5633.4200.8235.51741.2110.110SVM38Min1.4090.4162.3830.3381.0400.4042.34834.4380.043SVM38Mean2.6300.6183.9540.6312.1730.6073.77136.5190.075SVM38Max4.5560.9096.5581.0932.6440.8986.30438.3530.125SVM42Min0.9110.4262.5950.2190.7330.4122.48831.5570.051SVM42Mean1.7350.6484.4090.4161.5950.6344.14033.7720.069SVM42Max2.9360.9607.2820.7052.0420.9466.85835.5230.087Aggregated (base + enhancement) trafficSVM28Min4.4210.3963.3781.0614.7980.3411.98841.6480.020SVM28Mean7.8600.5635.1311.8868.6550.5023.23243.5890.063SVM28Max12.3680.7898.3082.96810.8830.7285.30345.0960.091SVM34Min2.4820.4423.8980.5962.7030.3752.08237.7150.034SVM34Mean4.5200.6225.8691.0855.6340.5613.54439.5660.077SVM34Max7.5140.8829.7851.8037.0310.8245.79841.2110.110SVM38Min1.6320.4524.0720.3922.0280.3822.26734.4380.043SVM38Mean3.0600.6496.3430.7344.0870.5883.84836.5190.075SVM38Max5.0970.93410.4591.2235.0710.8786.18238.3530.125SVM42Min1.0520.4584.3260.2521.5040.3852.39831.5570.051SVM42Mean1.9920.6706.7410.4782.8340.6074.11033.7720.069SVM42Max3.2440.96210.7150.7793.4780.9076.48635.5230.087Comparing the mean bit rates between the base layer
and corresponding MGS enhancement layer (same quantization scale), it is
evident that the enhancement layer adds a large increase in bit rate to the
base layer, and this for the entire range of studied quantization scales and
sequences. The spanned decrease in quantization scale of 6, which halves the
quantization step size, is encoded less efficiently by the MGS tools, resulting
in the much larger required bit rates.The CoV values of the MGS enhancement layer are
considerably lower than the CoV values of the base layer (G16-B0). From
Table 9, the maximum of the maximum CoV and PtM values are, respectively, 2.10 and 36.12 for
the base layer while for the enhancement layer both values are 0.98 and 10.72.
The differences are also this large for the maximum of the means of CoV and PtM. The aggregated traffic has maximum values that are comparable to or slightly
larger than the values of the enhancement layer, hence the CoV and PtM values of the base layer are greatly reduced
if transported in conjunction with the enhancement layer.Table 9
Maximum- (across quantization scales) of-maximum (across five video sequences), and
maximum-of-mean CoV and PtM values of H.264 SVC MGS base and enhancement layers, and aggregated traffic.
EncodermaxmaxmaxmaxCoVmaxPtMmaxCoV¯PtM¯MGS base layerH.264 SVC2.1036.121.7022.88MGS enhancement layerH.264 SVC0.987.680.664.84Aggregated (base + enhancement) trafficH.264 SVC0.9610.720.676.92The statistics on the GoP level have similar trends,
however the difference between the CoV values of the base; and enhancement layers is
less pronounced while still significant differences exist between PtM values.


## 7.1. MGS Layer Basics

The statistical
processing of the base layer, MGS enhancement layer, and the aggregated traffic
follow the single-layer analysis approach. As for spatial scalability and FGS,
the quality that we associate with the MGS enhancement layer is identical to
the quality of the aggregated traffic (base and enhancement layers).The MGS enhancement layer studied in this analysis
supports one quality enhancement with a quantization parameter decrease of 6
(increased quality). We leave the study of multiple MGS quality extraction
points (up to 16) within this enhancement layer for future research as well as
the statistical analysis of theG16-B3 GoP structure.


## 7.2. Results and Discussion

Table8 enumerates example H.264 SVC traffic statistics (min/mean/max) values across
sequences of the base layer, MGS enhancement layer, and the aggregated traffic.
In the first column of the table, we specify the encoding mode by the encoder
code SVM followed by the quantization scale.Table 8
Traffic statistics for base, and enhancement layers, and aggregated traffic of medium grain scalability encodings
using H.264 SVC for selected quantization scales.
Frame sizeBit rateGoP sizeFrame qualityEnc.MeanCoVXPtMXMeanPeakCoVYPtMYMeanCoQVmodeX¯SXX¯XmaxX¯X¯TXmaxTSYY¯YmaxY¯Q¯SQQ¯[kbyte][Mbps][Mbps][dB]Base layerSVM28Min0.7680.9547.9580.1842.5190.5912.98137.4720.039SVM28Mean1.7251.38315.5510.4145.3470.7907.39539.6820.083SVM28Max2.8431.95029.8730.6826.6461.19514.52341.5190.109SVM34Min0.3631.24811.8080.0871.5420.5673.17733.3240.054SVM34Mean0.7461.61320.2480.1793.1840.7818.48535.8160.099SVM34Max1.1842.08935.6310.2844.1041.16316.56237.9040.132SVM38Min0.2231.39014.3890.0531.0730.5453.30330.6070.062SVM38Mean0.4301.69522.2340.1032.0830.7428.80833.2160.098SVM38Max0.6752.05136.1180.1622.6231.07316.60035.4110.147SVM42Min0.1411.40316.7000.0340.7710.5223.23628.2290.069SVM42Mean0.2571.70422.8810.0621.3180.6858.71630.7830.091SVM42Max0.3982.10033.6300.0961.5960.93615.42133.0660.112MGS enhancement layerSVM28Min3.6530.3281.9750.8772.6160.3191.85441.6480.020SVM28Mean6.1350.4932.9011.4723.8390.4862.73143.5890.063SVM28Max9.8950.6994.4532.3754.6910.6924.23945.0960.091SVM34Min2.1180.3972.1880.5081.5280.3872.11537.7150.034SVM34Mean3.7740.5793.5310.9062.8090.5703.39539.5660.077SVM34Max6.5130.8335.6621.5633.4200.8235.51741.2110.110SVM38Min1.4090.4162.3830.3381.0400.4042.34834.4380.043SVM38Mean2.6300.6183.9540.6312.1730.6073.77136.5190.075SVM38Max4.5560.9096.5581.0932.6440.8986.30438.3530.125SVM42Min0.9110.4262.5950.2190.7330.4122.48831.5570.051SVM42Mean1.7350.6484.4090.4161.5950.6344.14033.7720.069SVM42Max2.9360.9607.2820.7052.0420.9466.85835.5230.087Aggregated (base + enhancement) trafficSVM28Min4.4210.3963.3781.0614.7980.3411.98841.6480.020SVM28Mean7.8600.5635.1311.8868.6550.5023.23243.5890.063SVM28Max12.3680.7898.3082.96810.8830.7285.30345.0960.091SVM34Min2.4820.4423.8980.5962.7030.3752.08237.7150.034SVM34Mean4.5200.6225.8691.0855.6340.5613.54439.5660.077SVM34Max7.5140.8829.7851.8037.0310.8245.79841.2110.110SVM38Min1.6320.4524.0720.3922.0280.3822.26734.4380.043SVM38Mean3.0600.6496.3430.7344.0870.5883.84836.5190.075SVM38Max5.0970.93410.4591.2235.0710.8786.18238.3530.125SVM42Min1.0520.4584.3260.2521.5040.3852.39831.5570.051SVM42Mean1.9920.6706.7410.4782.8340.6074.11033.7720.069SVM42Max3.2440.96210.7150.7793.4780.9076.48635.5230.087Comparing the mean bit rates between the base layer
and corresponding MGS enhancement layer (same quantization scale), it is
evident that the enhancement layer adds a large increase in bit rate to the
base layer, and this for the entire range of studied quantization scales and
sequences. The spanned decrease in quantization scale of 6, which halves the
quantization step size, is encoded less efficiently by the MGS tools, resulting
in the much larger required bit rates.The CoV values of the MGS enhancement layer are
considerably lower than the CoV values of the base layer (G16-B0). From
Table 9, the maximum of the maximum CoV and PtM values are, respectively, 2.10 and 36.12 for
the base layer while for the enhancement layer both values are 0.98 and 10.72.
The differences are also this large for the maximum of the means of CoV and PtM. The aggregated traffic has maximum values that are comparable to or slightly
larger than the values of the enhancement layer, hence the CoV and PtM values of the base layer are greatly reduced
if transported in conjunction with the enhancement layer.Table 9
Maximum- (across quantization scales) of-maximum (across five video sequences), and
maximum-of-mean CoV and PtM values of H.264 SVC MGS base and enhancement layers, and aggregated traffic.
EncodermaxmaxmaxmaxCoVmaxPtMmaxCoV¯PtM¯MGS base layerH.264 SVC2.1036.121.7022.88MGS enhancement layerH.264 SVC0.987.680.664.84Aggregated (base + enhancement) trafficH.264 SVC0.9610.720.676.92The statistics on the GoP level have similar trends,
however the difference between the CoV values of the base; and enhancement layers is
less pronounced while still significant differences exist between PtM values.


## 8. Combined Scalability Traffic Analysis

The H.264 SVC
encoder supports combined scalability that allows to extract temporal, spatial,
and SNR layers from one bitstream. The result of this flexibility from a video
traffic analysis viewpoint is that analyzing all possible temporal-spatial-SNR
encoding combinations of layers is prohibitive. Therefore, we focus on two case
studies: spatiotemporal and FGS-temporal scalability. We compare the base and
enhancement layers to the traffic characteristics obtained and discussed in the
preceding sections that analyzed each scalability mode in isolation.First, we explore the combined spatiotemporal
scalability case, which is based on the spatial scalable encodings used in
Section5, that is, we employ GoP structure G16-B3 supporting three
temporal layers in each spatial QCIF and CIF layer. Secondly, we analyze
combined FGS-temporal scalability based on the encodings used in Section 6,
supporting three temporal layers in the FGS base and enhancement layers.


### 8.1. Combined Spatiotemporal Scalability

Figures9 and 10 depict the VD curves of the
temporal layers in each spatial layer and in the aggregated traffic (base +
enhancement) for the Silence of the Lambs and Star Wars IV sequences. Each complete spatial layer has been individually analyzed in
Section 5. In the following, we focus on the temporal layers embedded in each
spatial layer and compare with the corresponding single-layers.VD curves of all temporal layers (0,1,2) embedded in each spatial layer and in the
aggregated (base + enhancement) traffic stream, for the Silence of the Lambs sequence encoded with H.264 SVC (G16-B3).
(a)
Spatial base (QCIF)(b)
Spatial enhancement (CIF)(c)
Aggregated traffic (CIF)VD curves of all temporal layers (0,1,2)
embedded in each spatial layer and in the aggregated (base + enhancement)
traffic stream, for the Star Wars IV sequence encoded with H.264 SVC (G16-B3).
(a)
Spatial base (QCIF)(b)
Spatial enhancement (CIF)(c)
Aggregated traffic (CIF)First, we recall from Section5 that the spatial base
layer (QCIF) statistics are identical to the single-layer (QCIF) statistics,
because these layers are identically encoded. Therefore, the temporal layer
statistics of the spatial base layer are also identical to the statistics of
the temporal layers embedded in the single-layer QCIF stream. Secondly, we
compare the VD curves of the temporal layers embedded in the aggregated spatial
stream in Figures 9(c) and 10(c) to the VD curves of the layers of the
temporal-scalability only CIF encodings in Figures 4(a) and 4(b). Visual
inspection reveals that these temporal layer VD curves have comparable values,
however, with somewhat lower CoV values in the low-quality range of Figures
4(a) and 4(b). Thirdly, the spatial enhancement layer's temporal layers in
Figures 9(b) and 10(b) cannot be directly compared with any prior results.
However, visual inspection reveals that the VD curves in Figures 9(b) and 10(b)
are very similar to the temporal layer VD curves of the aggregated spatial
traffic in Figures 9(c) and 10(c). These VD curves have the same shapes, but
the VD curves of the spatial enhancement layer have a slight vertical offset
(somewhat higher CoV) than the VD curves of the aggregate streams.
This indicates that the variability of the aggregated traffic is dominated by
the spatial enhancement layer.From (i) the similarity of the temporal layer VD curves
of the spatial base and aggregate streams with the corresponding VD curves of
the temporal-scalability only encodings, and (ii) the similarity of the temporal layers embedded
in the spatial enhancement layer with the temporal layers in the aggregated
spatial stream, we conclude that it suffices to separately analyze the layers
of temporal-scalability only encodings at the individual spatial resolution
(QCIF and CIF) to obtain good estimates of the traffic variabilities of the
layers in the combined spatiotemporal encoding.


### 8.2. Combined FGS-Temporal Scalability

The SVC FGS encoder supports FGS-temporal
scalability, which adds progressive refinement (PR) information to each
temporal layer embedded in the base layer. This PR information is provided by
the FGS enhancement layer. In this section, the three temporal layers included
in the base and enhancement layer are separately examined. Figures11 and 12
depict the temporal layers for base, untruncated enhancement, and aggregated
(base + untruncated enhancement) traffic for the Silence of the Lambs and Star Wars IV sequences.VD curves of all temporal layers (0,1,2) embedded in each FGS layer and in the aggregated
(base + untruncated enhancement) traffic stream, for the Silence of the
Lambs sequence encoded with SVC FGS (G16-B3).
(a)
FGS base layer(b)
FGS enhancement layer (untruncated)(c)
Aggregated FGS traffic (base + untruncated enhancement)VD curves of all temporal layers (0,1,2)
embedded in each FGS layer and in the aggregated (base + untruncated
enhancement) traffic stream, for the Star Wars IV sequence encoded with
SVC FGS (G16-B3).
(a)
FGS base layer(b)
FGS enhancement layer (untruncated)(c)
FGS combination layer (base + untruncated enhancement)We compare the VD curves of the temporal layers
embedded in the FGS base layers to the VD curves of the temporal-scalability
only encodings in Figures4(a) and 4(b). First, we observe that the temporal
layers embedded in the FGS base layers in Figures 11(a) and 12(a) have
comparable variability to the layers of the temporal-scalability only encodings
in Figures 4(a) and 4(b). Direct comparison of the VD curves in Figures 11(a)
and 12(a) with the VD curves in Figures 4(a) and 4(b) is difficult, because the
qualities associated with the temporal layers are computed differently (a
constant low PSNR value is used for missing frames in Figures 11 and 12 versus
the PSNR between duplicated, and original frame is used in Figures 4(a) and
4(b)). Nevertheless, the maximum CoV values and the CoV values at the low- and high-quality ends of
corresponding curves are very close. Given this similarity between the VD
curves of the temporal layers embedded in the FGS base layer and the VD curves
of the layers in the temporal-scalability only streams in Figures 4(a) and 4(b),
we conclude that it suffices to study the traffic statistics of the layers of
temporal-scalability only encodings to obtain reasonable estimates of the traffic
variabilities of the temporal layers embedded in the FGS base layer. On the
other hand, the FGS enhancement layer traffic, the aggregated FGS traffic, and
their embedded temporal layers cannot be meaningfully compared to any
previously obtained results. However, the unprecedented high variabilities of
these streams are indicative of the high variability the network path
encounters when different layers are transmitted independently.


## 8.1. Combined Spatiotemporal Scalability

Figures9 and 10 depict the VD curves of the
temporal layers in each spatial layer and in the aggregated traffic (base +
enhancement) for the Silence of the Lambs and Star Wars IV sequences. Each complete spatial layer has been individually analyzed in
Section 5. In the following, we focus on the temporal layers embedded in each
spatial layer and compare with the corresponding single-layers.VD curves of all temporal layers (0,1,2) embedded in each spatial layer and in the
aggregated (base + enhancement) traffic stream, for the Silence of the Lambs sequence encoded with H.264 SVC (G16-B3).
(a)
Spatial base (QCIF)(b)
Spatial enhancement (CIF)(c)
Aggregated traffic (CIF)VD curves of all temporal layers (0,1,2)
embedded in each spatial layer and in the aggregated (base + enhancement)
traffic stream, for the Star Wars IV sequence encoded with H.264 SVC (G16-B3).
(a)
Spatial base (QCIF)(b)
Spatial enhancement (CIF)(c)
Aggregated traffic (CIF)First, we recall from Section5 that the spatial base
layer (QCIF) statistics are identical to the single-layer (QCIF) statistics,
because these layers are identically encoded. Therefore, the temporal layer
statistics of the spatial base layer are also identical to the statistics of
the temporal layers embedded in the single-layer QCIF stream. Secondly, we
compare the VD curves of the temporal layers embedded in the aggregated spatial
stream in Figures 9(c) and 10(c) to the VD curves of the layers of the
temporal-scalability only CIF encodings in Figures 4(a) and 4(b). Visual
inspection reveals that these temporal layer VD curves have comparable values,
however, with somewhat lower CoV values in the low-quality range of Figures
4(a) and 4(b). Thirdly, the spatial enhancement layer's temporal layers in
Figures 9(b) and 10(b) cannot be directly compared with any prior results.
However, visual inspection reveals that the VD curves in Figures 9(b) and 10(b)
are very similar to the temporal layer VD curves of the aggregated spatial
traffic in Figures 9(c) and 10(c). These VD curves have the same shapes, but
the VD curves of the spatial enhancement layer have a slight vertical offset
(somewhat higher CoV) than the VD curves of the aggregate streams.
This indicates that the variability of the aggregated traffic is dominated by
the spatial enhancement layer.From (i) the similarity of the temporal layer VD curves
of the spatial base and aggregate streams with the corresponding VD curves of
the temporal-scalability only encodings, and (ii) the similarity of the temporal layers embedded
in the spatial enhancement layer with the temporal layers in the aggregated
spatial stream, we conclude that it suffices to separately analyze the layers
of temporal-scalability only encodings at the individual spatial resolution
(QCIF and CIF) to obtain good estimates of the traffic variabilities of the
layers in the combined spatiotemporal encoding.


## 8.2. Combined FGS-Temporal Scalability

The SVC FGS encoder supports FGS-temporal
scalability, which adds progressive refinement (PR) information to each
temporal layer embedded in the base layer. This PR information is provided by
the FGS enhancement layer. In this section, the three temporal layers included
in the base and enhancement layer are separately examined. Figures11 and 12
depict the temporal layers for base, untruncated enhancement, and aggregated
(base + untruncated enhancement) traffic for the Silence of the Lambs and Star Wars IV sequences.VD curves of all temporal layers (0,1,2) embedded in each FGS layer and in the aggregated
(base + untruncated enhancement) traffic stream, for the Silence of the
Lambs sequence encoded with SVC FGS (G16-B3).
(a)
FGS base layer(b)
FGS enhancement layer (untruncated)(c)
Aggregated FGS traffic (base + untruncated enhancement)VD curves of all temporal layers (0,1,2)
embedded in each FGS layer and in the aggregated (base + untruncated
enhancement) traffic stream, for the Star Wars IV sequence encoded with
SVC FGS (G16-B3).
(a)
FGS base layer(b)
FGS enhancement layer (untruncated)(c)
FGS combination layer (base + untruncated enhancement)We compare the VD curves of the temporal layers
embedded in the FGS base layers to the VD curves of the temporal-scalability
only encodings in Figures4(a) and 4(b). First, we observe that the temporal
layers embedded in the FGS base layers in Figures 11(a) and 12(a) have
comparable variability to the layers of the temporal-scalability only encodings
in Figures 4(a) and 4(b). Direct comparison of the VD curves in Figures 11(a)
and 12(a) with the VD curves in Figures 4(a) and 4(b) is difficult, because the
qualities associated with the temporal layers are computed differently (a
constant low PSNR value is used for missing frames in Figures 11 and 12 versus
the PSNR between duplicated, and original frame is used in Figures 4(a) and
4(b)). Nevertheless, the maximum CoV values and the CoV values at the low- and high-quality ends of
corresponding curves are very close. Given this similarity between the VD
curves of the temporal layers embedded in the FGS base layer and the VD curves
of the layers in the temporal-scalability only streams in Figures 4(a) and 4(b),
we conclude that it suffices to study the traffic statistics of the layers of
temporal-scalability only encodings to obtain reasonable estimates of the traffic
variabilities of the temporal layers embedded in the FGS base layer. On the
other hand, the FGS enhancement layer traffic, the aggregated FGS traffic, and
their embedded temporal layers cannot be meaningfully compared to any
previously obtained results. However, the unprecedented high variabilities of
these streams are indicative of the high variability the network path
encounters when different layers are transmitted independently.


## 9. Conclusion

We examined the video traffic characteristics of the temporal, spatial, and FGS scalability
modes of the scalable video coding (SVC) extension of the H.264/AVC standard
and compared with equivalent MPEG-4 Part 2 scalable video traffic. We also
analyzed SVC's combined spatiotemporal and combined FGS-temporal scalability.
Our traffic study focused on the joint characterization of the average bit rate
and the bit rate variability as a function of the video quality. We employed
long CIF resolution video sequences with a wide variety of texture and motion
features. We summarize our findings for each scalability mode as follows.(i) For the temporal scalability mode of SVC with
three temporal layers, which we examined separately, we have found that the
maximum coefficient of variation CoV of the frame sizes over all sequences and all
unsmoothed SVC temporal layers is above 3.3, with the CoV of temporal layer 1 being as high as 3.8. For
MPEG-4 Part 2, the maximum CoV stays below 2.9. Across temporal layers, we
have found that temporal layer 1 has the highest variability. When basic
smoothing is applied to SVC layers, we have found that the maximum CoV falls to 1.4 and 1.7 for the base layer and
temporal layer 1, respectively, while the CoV of temporal layer 2 falls to 2.27. For MPEG-4
Part 2, the smoothed CoV does not exceed 1.25. These figures point to
the significant increase in bit rate variability of temporal scalable SVC over
MPEG-4 Part 2. From the bit rate and quality analysis, we find that the mean
bit rates for the SVC temporal enhancement layers are significantly lower than
for the base layer due to the presence of large I and P frames and the
cascading quantizer assignment for SVC B frames. We also confirm that the
coefficient of quality variation decreases as each layer is cumulatively added,
thus increasing the subjective quality at the receiver.(ii) The spatial scalability traffic analysis first
focused on the separate analysis of the QCIF base layer, the CIF enhancement
layer, and the aggregated CIF stream, without considering the temporal
scalability present in each spatial layer. We have found that SVC's spatial
enhancement layer (CoV up to 2.6) has larger traffic variability than
its base layer (CoV up to 2.3) contrary to MPEG-4 Part 2
enhancement layer's traffic variability (CoV up to 1.4) which is lower than or comparable
to its base layer (CoV up to 1.6). We have also found that the
spatial base layer statistics are perfectly identical to the single-layer QCIF
statistics, confirming that the spatial base layer is encoded independently of
the enhancement layer. The traffic variabilities of both SVC and MPEG-4 Part 2
for the enhancement layer (CIF) are comparable to or slightly higher than for
single-layer CIF. For the aggregated traffic (CIF), we have found significantly
higher traffic variability for SVC as compared to MPEG-4 Part 2. Comparing with
the CIF enhancement layer, the CoV of the aggregated traffic is generally lower
than that of the enhancement layer.(iii) We analyzed FGS by treating base layer,
enhancement layer, and aggregated traffic separately. There has been a
significant effort in the SVC extension to improve the RD efficiency over
MPEG-4 FGS, the success of which can be clearly seen in up to 50% improvement
made in many cases. We have studied the simple truncation of the enhancement
layer of both encoders in progressive steps of 10% of the total enhancement
layer and have found that the variability of SVC for each point can be over 2.5
times that of MPEG-4 FGS, which has CoV values less than or equal to 1. Smoothing the
truncated bitstream lowers the SVC CoV to the range 1–1.5, while for MPEG-4 FGS,
smoothing reduces the traffic variability to the range 0.4–0.6. Compared with
single-layer encodings, we have found that the base
layer statistics are quite similar, given that both use a closed loop
prediction structure. We have observed that the untruncated enhancement layer
of MPEG-4 FGS contains almost the full refinement information for the entire
bit rate range (for all quantizers), resulting in an almost flat RD curve; in
contrast, SVC provides significant quality increases for increases in the
untruncated enhancement layer bit rate.(iv) We examined combined spatiotemporal
scalability by analyzing the temporal layers embedded in each spatial layer and
compared with the layers in temporal-scalability only encodings. We have
observed comparable values except in the low-quality range where somewhat lower
traffic variability is exhibited by the temporal-scalability only encodings. We
have also observed that the variability of the aggregated traffic is mainly
determined by the spatial enhancement layer. From the fact that the VD curves
of the temporal layers embedded in each spatial layer are similar to the VD
curves of the corresponding temporal-scalability only encodings, and that the
spatial enhancement layer is similar to that of the aggregated spatial traffic,
we conclude that it suffices to analyze the video traffic of each resolution
separately to obtain a good estimate of the traffic variabilities of all
embedded layers. We also examined combined FGS-temporal scalability of SVC.
Given the similarity of the temporal VD curves in the FGS base layer to the
temporal layer curves embedded in the single layer, a reasonable estimate of
the traffic variabilities of all layers embedded in the FGS base layer can be
obtained from the single-layer equivalent.Overall, these results clearly point to unprecedented levels of compression efficiency as well
as traffic variability for SVC coding, a factor which should be taken into
consideration for the design of efficient network transport protocols and
mechanisms for H.264 SVC scalable-encoded video.



---

*Source: 164027-2008-11-04.xml*