<figure><img src="image_1.png" /></figure>

HAL
open science

# Compression principles and applications to Digital Speech, Audio and Video

Charbel Tannous

<figure><img src="image_2.png" /></figure>

## To cite this version:

Charbel Tannous. Compression principles and applications to Digital Speech, Audio and Video. Master. Algorithmes de Compression, UBO Brest, France. 2018, pp.25. hal-04716837

HAL Id: hal-04716837

https://hal.science/hal-04716837v1

Submitted on 1 Oct 2024

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Compression principles and applications to Digital Speech, Audio and Video

C. Tannous

Université de Brest, Lab-STICC, CNRS-UMR 6285, F-29200 Brest, FRANCE

(Dated: October 1, 2024)

Compression of Digital Speech, Audio and Video is reviewed with emphasis on basic principles and available or potential applications. Description of various standards pertaining to Digital Speech, Audio and Video such as CELP, MPEG audio and video in the storage media and broadcast cases are reviewed.

PACS numbers: *43.71.-k, 43.71.+m, *43.72.-p, 43.72.+q, 89.70.+c, 42.30.Va

Keywords: Speech perception, Speech processing, MP3 compression, Video coding

Contents

I. Introduction

1

A. Quantization and Dynamic Range

2

B. Coding and compression

4

C. Entropic compression

7

II. Digital Speech

9

A. Pulse Code Modulation

11

B. Linear Predictive Coding

12

C. Code Excited Linear Prediction

13

D. Vocoders

15

III. Digital Audio

15

A. MPEG 1, 2, 4

15

B. ATRAC3

16

C. DTS

17

D. Audio Broadcast

18

IV. Digital Video

19

A. MPEG 1, 2, 4

19

B. H264

21

C. Video Broadcast

22

A. Historical codes

24

1. Morse code

24

2. Baudot code

24

References

25

I. INTRODUCTION

Considering streams of text, speech, audio, still images and video as a continuous time-dependent signal, analog to digital conversion (ADC) entails time sampling it following Nyquist [1] rule and quantizing its amplitude into given steps and providing data protection with error detection coding, error correction coding, synchronization, authentication coding etc...

Thus ADC and coding operations increase tremendously data volume and rate leading to a much larger bandwidth requiring compression.
A. Quantization and Dynamic Range

In the simplest case, signal amplitude is quantized uniformly with a constant step according to Pulse Code Modulation (PCM) [1]. When large variations with time are observed in the signal or when PCM noise is important, it is better to use non-uniform quantization where a larger quantization step is used when the signal changes slowly and a smaller step when the signal changes rapidly.

While Nyquist sampling allows getting a faithful digital version of the signal free of aliasing distortion, quantization determines its dynamic range (DR) that can be described precisely. Consider a signal $x(t)$ with $x_{max}$, $x_{min}$ the largest and smallest values respectively over the time variation of $x(t)$.

DR is by definition the ratio: $DR = \log_{10} \left(\frac{x_{max}}{x_{min}}\right)^2$ in Bel units [1] whereas in Decibels (dB) it is expressed as: $DR = 10 \log_{10} \left(\frac{x_{max}}{x_{min}}\right)^2$.

Consequently for a 1 bit quantized signal $x_{min} = 1, x_{max} = 2$, $DR = 10 \cdot \log_{10}[(2)^2] \approx 6$ dB. For an n-bit quantized signal, $x_{max} = 2^n x_{min}$ resulting into a $DR = 6n$ dB. n the number of quantization bits or bit-depth determines digitized signal quality whereas $f_s$ determines its resemblance to the original analog signal. Note that this formula does not account for quantization noise that might reduce tremendously the dynamic range as we describe further below [1]. The Audio CD (Compact Disk) is an example of 16 bit quantized music with $DR=6x16=96$ dB which is excellent for Audio. However, when quantization noise is considered, the $DR$ becomes a lot smaller, making people shy away from the CD and return to analog magnetic cassettes and Vinyl disks.

The same fate struck MP3 (originating from MPEG1 Audio Layer III), the compressed form of Audio CD with a compression factor of 1:12. After its large scale deployment and its world wide popular success, many felt that MP3 had several shortcomings due to ADC stressing a fact that no digital music format can be better than the analog Vinyl record or the analog magnetic cassette.

The SACD (Super Audio Compact Disk) was developed in order to combat quantization noise and find a digital format as close as possible to its analog source in order to satisfy a good musical ear.

SACD uses oversampling [1] to combat quantization noise [1] while mimicking as close as possible the original analog signal. This leads to an Effective Number of quantization Bits (ENOB) that is higher than the original one leading to a higher dynamic range. In addition to oversampling, a noise shaping operation [1] is performed by a $\Sigma - \Delta$ filter [2] to achieve efficiently a higher dynamic range otherwise the oversampling would become extremely high.

Oversampling means by Nyquist theorem that sampling frequency $f_s \ge 2f_0$ where $f_0$ is the signal cutoff frequency. Undersampling occurs when $f_s < 2f_0$ producing aliasing due to overlap between signal band centered at $f = 0$ and its replica at $f = f_s$ originating from sampling [2].

Consequently $\Delta$ the quantization step, is reduced to $\Delta'$ by the over-sampling ratio (OSR) K, however this is not enough or might be costly as explained further below.

Additionally, a special filtering $\Sigma - \Delta$ operation called noise shaping allows efficient reduction of noise via attenuation of the noise spectral characteristics amplitude inside the Nyquist band $[-f_s/2 : f_s/2]$.

In order to derive the effect of oversampling on $\Delta$, assume that noise spectra has amplitude given by $\sigma_s^2/f_s$ leading to Nyquist band surface given by $\sigma_s^2$ the error standard deviation (since $\sigma_s^2 = f_s \times (\sigma_s^2/f_s)$).

Making the additional assumption that noise amplitude is unaffected by oversampling, this means the equality of ratios $\sigma_s^2/f_s = \sigma_s'^2/f_s' (f_s' being the oversampling frequency) leading to $\frac{\sigma_s'^2}{\sigma_s^2} = \frac{f_s'}{f_s} = K$.

Recalling that the error standard deviation is related to the quantization step $\Delta$ as $\sigma = \frac{\Delta^2}{12}$, the following is obtained:

$$ \frac{\sigma_s'^2}{\sigma_s^2} = \frac{f_s'}{f_s} = K = \frac{\Delta'^2}{\Delta^2} \qquad (1) $$

Since $\Delta = 1/2^n$, $\Delta' = 1/2^{n'}$ with n and n' the respective number of quantization bits. $n' > n$ since $\Delta' < \Delta$. Quantization gain in terms of number of bits is given by $\Delta n = n' - n = \log_2(K)/2$.
This is very inefficient since this relation is logarithmic implying a very large OSR value K is required in order to increase $\Delta n$.

$\Sigma - \Delta$ filtering allows to increase $\Delta n$ much faster than oversampling. It is based on introducing a delay in the signal and a feedback operation (cf. Fig. 1).

<figure><img src="image_4.png" /><figcaption>**Fig.1:** Effect of signal unit delaying (left) and feedback (right) in order to perform $\Sigma - \Delta$ filtering. At left: the transfer function $D = z^{-1}$ the unit delay z-transform, At right: a feedback loop yields: $Y = (X + Y)z^{-1}$ and $H = \frac{Y}{X} = \frac{z^{-1}}{1-z^{-1}}$. Adapted from Orfanidis [3].</figcaption></figure>

The unit delay z-transform $D = z^{-1}$ is inserted inside a feedback loop as displayed in Fig. 1. Feedback leads to $Y = (X + Y)z^{-1}$ and $H = \frac{Y}{X} = \frac{z^{-1}}{1-z^{-1}}$ where X, Y, H are the respective z-transforms of input, output and transfer function.

This is first-order $\Sigma - \Delta$ filtering. Iterating the operation p times yields $Y = (\frac{z^{-1}}{1-z^{-1}})^p X$, the p-order $\Sigma - \Delta$ transform given by the transfer function $H^p = (\frac{z^{-1}}{1-z^{-1}})^p$.

Filtering entails using the modulus-squared $\Sigma - \Delta$ transfer function $|H^p|^2$ in order to evaluate the transformed standard deviation [3] given by:

$$ \sigma_s^2 = \frac{\sigma_s'^2}{f_s'} \int_{-f_s/2}^{f_s/2} df \left| \left( \frac{z^{-1}}{1-z^{-1}} \right)^p \right|^2 = \frac{\sigma_s'^2}{f_s'} \int_{-f_s/2}^{f_s/2} df \left| \frac{z^{-1}}{1-z^{-1}} \right|^{2p} \quad (2) $$

The standard z-transform formula [3]: $z = \exp(j2\pi f/f_s)$ leads to:

$$ \left| \frac{z^{-1}}{1-z^{-1}} \right|^{2p} = 2|\sin(\pi f/f_s)|^{2p} \approx 2|\pi f/f_s|^{2p} \quad (3) $$

Performing integration of eq. 2 leads to:

$$ \sigma_s^2 = \sigma_s'^2 \frac{\pi^{2p}}{(2p+1) K^{2p+1}} \quad (4) $$

Thus non-uniform quantization, after over-sampling with OSR K and p-order $\Sigma - \Delta$ filtering [3] (cf. Fig. 1), leads to the rule:

$$ \Delta n = \left( p + \frac{1}{2} \right) \log_2 K - \frac{1}{2} \log_2 \left( \frac{\pi^{2p}}{2p+1} \right) \quad (5) $$

This gives an effective number of bits ENOB=n + $\Delta n = n'$.

Note that without $\Sigma - \Delta$ filtering, i.e. $p = 0$ we get: $\Delta n = \frac{1}{2} \log_2 K$ leading to extremely high oversampling ratio (cf. Table 1).

Consequently, SACD properties are usually briefly described as 120 dB, 100 kHz bandwidth, with 1 bit quantization and 64 x 44.1 kHz sampling rate that contrasts with the Audio CD 16 bit quantization and 44.1 kHz sampling rate.

Note that the number 44100= (2x3x5x7)² and the Audio CD is read at a rate of 75 sectors/s, each sector containing 98 audio frames, each frame containing 6 audio samples with each sample made of 16 bits. Thus we get: 6x98x75=44100.

In fact, 44.1 kHz rate originated from the PCM adaptor [4].

High-quality PCM audio requires a significantly larger bandwidth than a regular Stereo FM analog audio signal. For example, a raw 16-bit PCM signal produces a bitrate about 16x44100 b/s requiring an analog bandwidth ~ 705.6
kHz (compared to about 30-40 kHz of bandwidth required for an analog stereo audio signal). When digital coding is taken into account, that bitrate might be doubled along with the bandwidth to reach 1.4 to 1.5 MHz. A standard analog audio recorder could not meet that requirement during the 1980 years. One practical solution consisted of using a video tape recorder capable of recording signals with this bandwidth, to store audio information.

However a conversion of digital audio into video was required. Such an audio recording system therefore includes two devices, namely the PCM adaptor, which converts audio into video, and the video tape recorder. A PCM adaptor takes an analog audio (stereo) signal as an input converting it into bits to be coded and modulated, resulting into a BW (Black-White) video signal.

The latter can be stored on any ordinary analog video tape recorder, with each line containing 3 audio samples. Thus the choice of CD sampling frequency, originates from the number of video lines, frame rate and bits per line.

Additionally, 44.1 kHz sampling frequency resulted from a need of video compatibility with 25 frames/s BW (European countries) and 30 frames/s BW (North-American countries) formats used for audio storage. This results for NTSC: 245x3x60=44100 and for PAL: 294x3x50=44100 where 245 (resp. 294) are the number of video lines/frame, 3 is the number of audio samples/line and 60 (resp. 50) is the number of fields (half-frame)/s.

In fact, a more precise description of SACD is given below:

1. The 6 dB rule should not be applied to an oversampled 1 bit quantized signal, since would get a wrong *DR* as explained below.

2. Oversampling at 64 times Nyquist rate i.e. 64 x 44.1 kHz gives a bitrate of 64 x 44.1 kHz x 1 bit=2.8 Mb/s.

3. Using non-uniform quantization with OSR $K = 64$ and $\Sigma - \Delta$ order $p = 3 - 4$ (see Table 1), an equivalent ENOB of 20 bits is obtained yielding (6 dB/bit) x 20 bits =120 dB dynamic range.

4. Equivalent bandwidth/bit, is given by: $(2.8 \text{ Mb/s})/(20 \text{ bit} \times 2) \approx 70 \text{ kHz}$. Hence 100 kHz is an over estimation.

<table><caption>**Table 1:** Application of formula [3] $\Delta n = (p + \frac{1}{2}) \log_2 K - \frac{1}{2} \log_2 (\frac{\pi^{2p}}{2p+1})$ to ENOB evaluation in the SACD case, ENOB=1+Δn. Note that if we require an ENOB=16 with 1 bit, Δn = 15 would require a very large K ~ 10⁹ to achieve same Δn without Σ - Δ filtering. Another example is for obtaining an ENOB=20 from 1 bit, a Σ - Δ order p=3-4 is sufficient while K = 64.</caption><thead><tr><th>K (OSR)</th><th>p (Σ - Δ)</th><th>Δn</th><th>ENOB</th></tr></thead><tbody><tr><td>64</td><td>0</td><td>3.0</td><td>4.0</td></tr><tr><td>64</td><td>1</td><td>8.40</td><td>9.40</td></tr><tr><td>64</td><td>2</td><td>13.51</td><td>14.51</td></tr><tr><td>64</td><td>3</td><td>18.53</td><td>19.53</td></tr><tr><td>64</td><td>4</td><td>23.52</td><td>24.52</td></tr><tr><td>64</td><td>5</td><td>28.48</td><td>29.48</td></tr></tbody></table>

**Note:** Quantization is either scalar where a single value of the signal at time $kT_s$ is processed (k an integer and $T_s$ the sampling time) or vectorial (called VQ) where several sampled value of the signal at times $kT_s + n_V T_s, n_V = 1, 2... n_{VQ}$ a set of successive sampled values to be processed simultaneously at instant $kT_s$. $n_{VQ}$ determines the vector dimension. Note that vector construction can also be made symmetrically with respect to the sampling instant (i.e. with samples before and after $kT_s$).

Lack of attractiveness of VQ originates from the heavy signal processing requiring particular mathematical operations to tackle it, leading to a large processing time (in coding and decoding) that discourages its use. Nevertheless, VQ is used in many audio formats as described further below.

## B. Coding and compression

In the digital world of processing, communication and storage, source coding plays an important role since a digital representation of specific elements or patterns contained in the analog signal, called source symbols (alphanumeric
characters in a text, phonemes in speech, moving images in a video...) require specific operations in order to preserve
data integrity.

There are several types of coding depending on applications such as processing, communication, or storage. Source coding deals with a signal whereas Channel coding deals with signal transmission across it.
Some source coding families are:

* Line coding (or Modulation coding) is used to achieve at least three goals:

1. Automatic clock recovery in order to avoid an additional signal carrying time coding of the signal.

2. Reducing harmonic distortions by symmetrizing the binary (0,1) signal turning it closer to a purely harmonic signal (-1,1)

3. Avoiding Inter-Symbol Interference by placing zeroes between consecutive ones, e.g. replacing (11) by (10000001).

* Synchronization coding

* Compression coding

* Authentication coding

In communications, channel coding families are required such as:

* Error detection coding: A signal is distorted during transmission, thus a code is set up to protect its integrity like parity coding, MD5 [5]...

* Error correction coding: When a signal error is detected, specific correction must be applied for signal recovery

* Multi-channel coding: Stereo (2 channels), AC3 (6 channels named also 5.1 with 5 satellite channels and one low-frequency channel), MPEG2 AAC (1-48 channels), MPEG4-AAC (1-48 channels)... audio streams are jointly processed in order to reduce redundancies and mutual interference...

* Synchronization coding

* Compression coding

Coding operations transform source symbols into coded symbols (codewords) increasing substantially data volume to represent data stream, requiring compression.

There are many compression methods, some suitable for text and others for phonetic symbols in speech, musical symbols in audio, and graphical items in still images, and movies (or video).

Compression algorithms exploit redundancies or limitations present in entities such as text, speech, audio, image, video...

Redundancies or limitations occur in the following cases:

* Spatial: An image might have a fixed background or contain repeating patterns

* Temporal: Frames might evolve slowly such that it becomes possible to apply motion estimation algorithms to predict their evolution

* Frequential: When one performs a Fourier Transform of an image or a sound, one might discover that some frequential patterns are repeated akin to notes in music...

* Time-Frequency combination: It is possible to analyze images in both time and frequency with Wavelets searching for repeating patterns at different scales in order to obtain a potential 1:500 compression factor while preserving image quality.

* Entropic: Repetition might be present in a statistical fashion allowing to allocate less data to a highly repetitive pattern and more data to a rarely repetitive pattern

* Perceptual: Human auditive and visual system are sensitive to sound, light, colors with various degrees of sensitivity allowing to use these limitations in digital compression algorithms.
Compression algorithms are divided in two major classes, lossy and lossless.

Lossless algorithms use the fact that certain data symbols on the input data are more frequent than others. Variable length codes are assigned to the input data with shorter codes being assigned to more frequent symbols or symbol sequences and longer codes to rare symbols. Typically, the compression ratio is 1:4 to 1:5.

Lossy algorithms are usable where it is permissible to loose quality in transmission and storage while retaining the general contents of data such as sound transmission (MP3, WAV...) or storage as well as in images (JPEG, PNG) and video (MPEG, H264).

Permissible loss is mostly compensated by human eye, ear and brain physiological capabilities. The resulting compression ratio range is from 1:10 to about 1:200.

Shannon Information Theory allows to link a symbol occurrence to its coding representation thus it makes a rigorous basis for compression analysis. For example, given a message made up of N symbols, in total, containing n different symbols, information content measured in bits of the message is NHS where HS is Shannon Entropy (SE) given by:

$$H_S = - \sum_{i=1}^{n} p_i \log_2(p_i) \qquad (6)$$

where pᵢ is symbol i occurrence probability. What is regarded a symbol depends on the signal, it might be an ASCII code, 16 or 32 bit words, words in a text ...

Symbol length is given by the number of characters, thus mean length is given by:

$$\langle l \rangle = \sum_{i=1}^{n} p_i l_i \qquad (7)$$

Suppose some quantity of interest is distributed exponentially [6] as given by the probability density function (PDF):

$$p(x) = \frac{1}{b} \exp[-(x-a)/b], \quad x \ge a, \quad f(x) = 0 \quad \text{elsewhere} \qquad (8)$$

with mean ⟨x⟩ = a + b and variance σ² = ⟨x²⟩ - ⟨x⟩² = b².

Let us assume that the mean value of the distribution is one tenth the dynamic range i.e. a + b = 25.6, thus a = 0, b = 25.6.

Evaluating SE on the basis, each value in the interval [0-255] is considered a symbol, with n = 256, a = 0, b = 25.6, pᵢ = exp[-i/25.6]/25.6, we get a mean SE of 6.03 bits per measured value which is 25% less than 8 bits we need to represent data as a sequence of bytes.

Even if the dynamic range is increased by a factor of four, without increasing the Analog to Digital Conversion (ADC), using a ten bit ADC, the mean SE would have been the same and hence possible compression gain even higher (about 40%).

When various probability distributions are compared, the exponential distribution appears to be particularly hard to compress. If we consider, for instance, a discrete distribution following a Gaussian behavior with a PDF:

$$p(x) = \frac{1}{\sigma\sqrt{2\pi}} \exp\left(-\frac{[x - \mu]^2}{2\sigma^2}\right), \quad x \in ] -\infty, +\infty[ \qquad (9)$$

μ is PDF mean and σ is standard deviation. Using the same root mean square as the exponential PDF ⟨x²⟩ = 2b², we obtain an SE of only 3.98 bits supposing Gaussian PDF parameters: σ² = 2b² - μ² with μ < b√2 chosen such that the normalization condition:

$$\sum_{i=1}^{n} p_i = \sum_{i=1}^{n} \frac{1}{\sigma\sqrt{2\pi}} \exp\left(-\frac{[i - \mu]^2}{2\sigma^2}\right) = 1, \quad n = 256 \qquad (10)$$

is satisfied, requiring μ - 5σ > 0 and μ + 5σ < n.

In conclusion, SE depends strongly on PDF.

Codes are characterized by:

* Comma free codes: A comma is a separator between codewords like a time interval separating the dit and dat in Morse code (cf. Appendix).
*   Prefix free codes: This is a sufficient condition to ensure that a code is uniquely decodable is that no code word is a prefix of another.

*   Kraft inequality [5] for unique decodability: Consider a binary source code and let the length of codewords be $l_1, l_2, \dots, l_m$ (i.e. source has $m$ different symbols). Then unique decodability is given by condition:

$$ \sum_{k=1}^{m} 2^{-l_k} \le 1 \qquad (11) $$

There are several categories of compression algorithms:

*   Statistical occurrence: Shorter codes for more frequent symbols and longer ones with less frequent symbols as done by lossless Huffman coding.

*   Concatenating symbols: Instead of coding single symbols, assembling several symbols to be coded as in lossless Arithmetic coding might be more efficient as done by Arithmetic coding.

*   Waveform: Fitting a time variation of a signal with an analytic function like Fourier Series amounts to make a time to frequency transformation since Fourier theorem states that any periodic function can be expressed as an infinite sum of sines and cosines. Practically, one identifies a finite set of important frequencies contained in the signal to perform an efficient lossy compression.

*   Wavelet transform [5]: It analyzes a signal both in time and in frequency as follows: Considering a signal $x(t)$, a time interval is selected, and wavelet analysis identifies frequencies that constitute $x(t)$ in that interval as if one were zooming in time instead of space. This is equivalent to identifying patterns at different scales.

*   Linear Prediction: Assuming a sample $x(n)$ to vary linearly over a time interval $\delta t$ a linear prediction $\hat{x}(n)$ can be made over $\delta t$. The residual $|x(n) - \hat{x}(n)|$ has a smaller variation than $x(n)$ thus needing a smaller number of quantization bits leading to lossy compression.

*   Variable bitrate: Instead of coding with a fixed bitrate, it is possible to vary the bitrate according to signal nature. For instance, when time variation of frames is slow, a small bitrate is chosen, however when fast action runs in frames, a larger bitrate is adopted.

*   Perceptual: Human eyes and ears have finite spectral bandwidth, response time, adaptive capabilities... Then it is possible to exploit these limitations obtained from psycho-acoustic and psycho-visual measurements in order to use them in digital lossy compression algorithms. Compression might reach 1:2 as proven by a perceptual entropy $H_J$ derived, in the audio case, by Johnston [7] from Shannon entropy $H_S$ with the result $H_J \sim H_S/2$.

## C. Entropic compression

There are three types of lossless algorithms that perform entropic compression based on combating symbol redundancy from a statistical point of view: Run Length Encoding (RLE), Huffman and Arithmetic coding.

1.  Run Length Encoding (RLE):

    Run length is the number of successive repetitions of a character (symbol). When a symbol $S$ occurs $n$ consecutive times in the input stream, it is possible to replace its $n$ occurrences by $n.S$.

2.  Huffman coding:

    Huffman coding [8] assigns a codeword to each symbol, with the codeword being as short as 1 bit, or considerably longer than the input symbols, strictly depending on their probabilities. The optimal number of bits to be used for each symbol is $\log_2(1/p)$, where $p$ is the probability of a given character.

The problem with this scheme lies in the fact that Huffman codes have to be an integral number of bits long. For example, if the probability of a character is 1/3, the optimum number of bits to code that character is around 1.6 from SE. Then Huffman coding scheme has to assign 1 or 2 bits to the code leading to an inefficiently compressed message.
This non-optimal coding becomes a noticeable problem when the probability of a character becomes very high. If a statistical method can be developed that can assign a 90% probability to a given character, the optimal code size would be 0.15 bits. Huffman coding system would probably assign a 1 bit code to the symbol, which is 6 times larger than necessary.

As an example, consider a musical piece made of six notes A, B, C, D, E et F. with respective probabilities: 25%, 25%, 20%, 15%, 10% and 5%.

Shannon entropy is: $H_S = -\sum_{i=1}^{6} p_i \log_2(p_i) = 2.423219$ exactly between 2 and 3 since $4 < 6 < 8$. Note that $\log_2(6) = 2.58$.

Huffman coding is displayed in Fig. 2.

<figure><img src="image_5.png" /><figcaption>Fig.2: Huffman tree and resulting coding: A: 01, B: 10, C: 11, D: 001, E: 0000, F: 0001. Note that Huffman coding is prefix free since no code among A,B,C,D,E,F is a prefix to another.</figcaption></figure>

Applying Kraft inequality to Huffman coded source gives the following length of 6 codewords as: $\ell_A = 2, \ell_B = 2, \ell_C = 2, \ell_D = 3, \ell_E = 4, \ell_A = 4$. Then unique decodability is given by Kraft condition:

$$ \sum_{k=1}^{6} 2^{-\ell_k} = 3 \times 2^{-2} + 2^{-3} + 2 \times 2^{-4} = 1 \qquad (12) $$

Coding efficiency: $\eta = H/\langle l \rangle = 2.42/2.43 = 98\%$, compression rate =3/2.4=1.2 since 3 is the most probable number of bits.

### 3. Arithmetic coding:

Arithmetic coding completely bypasses the idea of replacing an input symbol with a specific code. Instead, it concatenates several symbols and replaces them with a single floating point number. The longer (and more complex) the message, the more digits are needed in the output number.

The output from an arithmetic coding process is a single number in the interval [0:1]. This single number can be uniquely decoded to create the exact stream of symbols that went into its construction. In order to construct the output number, the symbols being encoded should have a set probabilities assigned to them. For example, if the message "polynomial" is to be encoded, the associated symbol probability distribution is [9]:

Once the character probabilities are known, the individual symbols need to be assigned a range along a "probability line", which belongs to interval [0:1]. It doesn't matter which characters are assigned which segment of the range, as long as it is done in the same manner by both the encoder and the decoder. The nine character symbol set use here would look like this:

Each character is assigned a portion of the [0:1] interval that corresponds to its probability of appearance. Arithmetic coding proceeds as follows:

* There is no one to one correspondence between source symbols and codewords.

* An entire sequence of source symbols is assigned a single codeword.
<table><caption>**Table 2:** Example text "polynomial" with its characters and their occurrences. Adapted from Ref. [9]</caption><thead><tr><td>Character</td><td>Probability</td></tr></thead><tbody><tr><td>a</td><td>1/10</td></tr><tr><td>i</td><td>1/10</td></tr><tr><td>l</td><td>2/10</td></tr><tr><td>m</td><td>1/10</td></tr><tr><td>n</td><td>1/10</td></tr><tr><td>o</td><td>2/10</td></tr><tr><td>p</td><td>1/10</td></tr><tr><td>y</td><td>1/10</td></tr></tbody></table>

<table><caption>**Table 3:** Arithmetic coding result for the "polynomial" message. Adapted from Ref. [9]</caption><thead><tr><td>Character</td><td>Interval width</td><td>Range</td></tr></thead><tbody><tr><td>p</td><td>1.0</td><td>0.8 - 0.9</td></tr><tr><td>o</td><td>0.1</td><td>0.6 - 0.8</td></tr><tr><td>l</td><td>0.02</td><td>0.2 - 0.4</td></tr><tr><td>y</td><td>0.004</td><td>0.9 - 1.0</td></tr><tr><td>n</td><td>0.0004</td><td>0.5 - 0.6</td></tr><tr><td>o</td><td>0.00004</td><td>0.6 - 0.8</td></tr><tr><td>m</td><td>0.000008</td><td>0.4 - 0.5</td></tr><tr><td>i</td><td>0.0000008</td><td>0.1 - 0.2</td></tr><tr><td>a</td><td>0.00000008</td><td>0.0 - 0.1</td></tr><tr><td>l</td><td>0.000000008</td><td>0.2 - 0.4</td></tr></tbody></table>

* Each source symbol is represented by an interval in [0,1]. As the number of symbols increases, the interval size reduces in accordance with the probability of occurrence of symbols.

* Inaccurate probability models can lead to non-optimal results thus it is better to use an adaptive, context dependent probability model. Context dependent probabilities are based on a predefined neighborhood around the symbol being coded.

## II. DIGITAL SPEECH

In order to compress speech, it would be useful to describe first its important properties. Speech signals are non-stationary and may be considered as quasi-stationary over short time segments, typically 5-20 ms.

A speech signal can be classified as voiced, unvoiced or mixed. Voiced sounds are generated when the vocal cords vibrate in such a way that air flow from the lungs is interrupted periodically, creating pulses to excite the vocal tract. When vocal cords are stationary, turbulence created by air flow through a vocal tract constriction generates unvoiced sounds.

Voiced speech corresponds to vowels such as /a/, /i/, /o/, /u/..., whereas unvoiced speech corresponds to consonants such as /th/ (e.g. through), /sh/ (e.g. shut)... Mixed speech corresponds to e.g. /z/...

Voiced speech is produced by exciting the vocal tract with quasi-periodic air pulses generated by larynx vibrating vocal chords producing vowels. Time dependent voiced sound is characterized by strong periodicity, with a fundamental frequency $f_0$ referred to as pitch (cf. Fig. 4).

Unvoiced sounds, on the other hand, do not display periodicity but random behavior. The spectrum (cf. Fig. 4) displays a significant amount of high-frequency components proper to fast changing signals.

Voiced speech is quasi-periodic in time and harmonically structured in frequency while unvoiced speech is random and broadband (cf. Fig. 4). In general, energy of voiced time segments is larger than that of unvoiced segments.

Short-time power spectrum of voiced speech has a fine structure with several peaks (cf. Fig. 4) signaling quasi-periodicity of speech originating from larynx vocal chords vibrations.

Speech spectral envelope (cf. Fig. 4) displays a set of peaks called formants, the vocal tract resonant modes. They are due to interaction of source and vocal tract consisting of the pharynx and mouth. An average vocal tract produces three to five formants below 5 kHz. Higher formants are also important in wideband and unvoiced speech.
Unvoiced speech is produced by forcing air through a constriction in the vocal tract. Nasal sounds (e.g. /n/) are due to acoustic coupling between nasal and vocal tracts. Plosive sounds (e.g. /p/) are produced by releasing abruptly air pressure excess caused by a vocal tract constriction.

Unvoiced sounds (such as consonants) occur as air is passed through some constriction in the mouth, or expressed as a sudden burst. The position where the constriction is formed depends on sound e.g. phoneme nature.

Phonemes depend on language, for instance French phonemes numbered 37 are less numerous than the 45 English or 42 Russian phonemes implying a more structurally complex speech than French qualified as straightforward, spanning only one octave of frequency, lacking modulation and lexical stress.

<figure><img src="image_6.png" /><figcaption>**Fig.3:** Speech production relating each element of human physiology to its precise contribution through a White noise source and time varying filter to model the input of physiological elements such as trachea, pharyngeal, oral, and nasal cavities, nostrils and mouth (Tongue, lips and palate). Adapted from Spanias [10] and Chu [11].</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>**Fig.4:** Voiced (top) and Unvoiced (bottom) speech with their time dependent behavior at left and short-time power spectra at right. In voiced speech (top right) Pitch corresponds to a basic frequency $f_0$ followed by its higher harmonics $2f_0$, $3f_0$...Adapted from Spanias [10]</figcaption></figure>

There are two varieties of speech: Narrow Band (NB) speech limited to 4 kHz and Wideband Speech (WB) limited to 7 kHz. NB speech is for ordinary telephony whereas WB is for tele-conferencing and high-quality telephony.

Voice coding consists of the following schemes (cf. Tables 6,7,8) depending on bandwidth and bitrate with specific algorithms (based on signal features that relate to time, frequency or perceptual properties):

* Waveform coding characterized by high bitrate with an algorithm based mainly on signal variation from the time-frequency point of view without considering its perceptual properties.
* Vocoders characterized by low bitrate with an algorithm based mainly on signal perceptual properties.

* Hybrid coders characterized by low to high bitrate with an algorithm based on both time-frequency and perceptual properties of the signal

Quality in Speech and analog telephony is given in Table 4.

<table><caption>**Table 4:** Mean Opinion Score (MOS) of a speech signal transmitted over an analog telephone line. Generally a speech signal is required to have 70 dB dynamic range for excellent audibility.</caption><thead><tr><th>MOS</th><th>Quality</th><th>Dynamic Range</th></tr></thead><tbody><tr><td>5</td><td>Excellent</td><td>50-60 dB and higher</td></tr><tr><td>4</td><td>Good</td><td>40-50 dB</td></tr><tr><td>3</td><td>Fair</td><td>30-40 dB</td></tr><tr><td>2</td><td>Poor</td><td>20-30 dB</td></tr><tr><td>1</td><td>Bad</td><td>10-20 dB</td></tr></tbody></table>

## A. Pulse Code Modulation

Pulse Code Modulation (PCM) is the simplest method for ADC voice with 8 kb/s sampling and 8 bit uniform quantization to yield the 64 kb/s DS0 [12] standard rate of communication. 8 kb/s sampling originates from Nyquist theorem based on 4 kHz bandwidth NB speech and 8 bit quantization means 8 bit x6 dB/bit=48 dB of dynamic range (cf. Table 4) with no account of quantization noise.

The advantage of uniform PCM is that quantization error power is independent of signal power, implying that higher power signals are quantized with the same resolution as lower power ones.

Compared (i.e. compressed-expanded) PCM concerns coders in which quantization levels are not uniformly distributed in order to combat quantization noise.

Such coders may be modeled using a compressive nonlinearity, followed by uniform PCM. The nonlinearity is called μ-law in North-America and A-law in Europe.

μ-law is a North-American companding method applicable to PCM voice telephone with the following μ-law compressor function [1]:

$$f(x) = x_{max} \frac{\log(1 + \mu|x|/x_{max})}{\log(1 + \mu)} \text{sgn}(x/x_{max}), \quad |x|/x_{max} \le 1 \qquad (13)$$

$x_{max}$ is the maximum input level with the standard value μ = 255, in the United States.

μ-law companding for PCM voice delivers a stable value of $(S/N)$, despite large crest factor $S_x$ variations among individual speakers. $S_x$ is defined as the ratio of signal peak to its root mean square value.

Most signals (particularly music and voice signals) do not have uniformly distributed amplitudes. They cover a wide range of amplitudes having mostly moderate values requiring non-uniform quantization or companding.

Recovering the signal entails applying Digital to Analog Conversion (DAC) and expanding, thus the name companding. The signal is supposed to be normalized thus $|x| \le 1$. Taking μ as the companding factor, the non-linear compressor function $f(x)$ is basically linear in $x$ for small signal amplitudes saturating as $x$ grows:

A-law:

$$f(x) = \begin{cases} \text{sgn}(x) \frac{A|x|/x_{max}}{\log(1+A)}, & 0 \le |x|/x_{max} \le 1/A \\ \text{sgn}(x) \frac{(1+\log(A|x|/x_{max}))}{\log(1+A)}, & 1/A \le |x|/x_{max} \le 1 \end{cases}$$

A is a factor similar to μ appearing in μ-law with a value A=87.6 used in Europe. For large A and μ, the compressor functions are similar in shape but for smaller values of A there is a break in A-law.

Generally companding allows to gain an equivalent 6 bit (i.e. 36 dB). Thus an 8 bit speech signal becomes equivalent to a 14 bit one.
In differential PCM (DPCM), each sample $x(n)$ is compared to a prediction $\hat{x}(n)$, and the prediction residual $|x(n) - \hat{x}(n)|$ having smaller variations than $x(n)$, needs fewer quantization bits.

In one standard based on DPCM, continuously varying-slope delta (cf. Fig. 5) modulation (DM) [1] is used, input speech signal is over sampled to either 16 or 32 kHz. When sampling frequency is greater than Nyquist rate, we expect that signal value is roughly equal to its previous value. This is used to predict the signal value and compress the bit rate (cf. Fig. 5).

Delta modulation provides approximately the same quality of voice as PCM at one-half the bandwidth with error rates about $10^{-2}$ to $10^{-3}$. Values of the over sampled signal are predicted and the difference signal is quantized at one bit per sample, with an adaptively varying fashion. This standard performs poorly in quiet environments but surprisingly well in extremely noisy environments.

A setback occurs when there is an overload originating from error estimation failure as illustrated in Fig. 5 with the signal moving further away from the DM approximation.

<figure><img src="image_8.png" /><figcaption>**Fig. 5:** Delta modulation (DM) approximation is obtained from oversampling yielding a predicted signal $\hat{x}_q(t)$ close to the original one $x(t)$. The error $\epsilon(t) = x(t) - \hat{x}_q(t)$ is approximately $\pm \delta$ where $\delta$ is the quantized value of $\epsilon(t)$. Overload is illustrated when the error estimation fails. Adapted from Carlson [1]</figcaption></figure>

DPCM systems with adaptive prediction and quantization are referred to as adaptive differential PCM systems (ADPCM). G.726 is a commonly used ADPCM standard which can operate at 16, 24, 32, or 40 kb/s (2-5 bits/sample). G.726 ADPCM is frequently used at 32 kb/s in telephony. A filter is used to predict signal values and the prediction residual is quantized using a semi-logarithmic companded PCM quantizer (cf. section II A) with 2-5 bits per sample. Adaptation means that quantization step size changes according to amplitude of previous samples.

AMR (Adaptive Multi-Rate) codec (coder-decoder) encodes narrowband (200-3400 Hz) signals for each 20 ms (millisecond) frame sampled at 8 kHz using variable bitrates ranging from 4.75 to 12.2 kb/s with quality speech starting at 7.4 kb/s. AMR has a minimum bitrate of 4.75 kb/s, a peak of 12.2 kb/s and a typical conversational rate of 4 kb/s. It has been used in GSM (Groupe Spécial Mobile) telephony (cf. Table 7).

## B. Linear Predictive Coding

Speech coding is the art of creating a minimally redundant representation of the speech signal that can be efficiently transmitted or stored in digital media, and decoding the signal with good perceptual quality. Speech is typically quantized using either 16-bit uniform or 8-bit companded quantization (equivalent to 14 bit) [13].

A speech coder converts a digitized speech signal into a coded representation, which is usually transmitted in frames [13]. A speech decoder receives coded frames and synthesizes reconstructed speech. Speech coders differ primarily in bitrate, complexity (measured in operations per second), delay (measured in milliseconds between recording and playback), and perceptual quality of the synthesized speech [13].

There are different types of speech coders displayed in Table 5. A selection of specific speech coding standards is given in Table 6. Waveform coders attempt to code the exact shape of the speech signal waveform, without considering the nature of human speech production and perception. These coders are high-bitrate coders (typically above 16 kb/s). Linear prediction coders (LPC), assume that a speech signal is produced by some linear time-invariant model.

The excitation function is a quasi-periodic signal constructed with discrete pulses (1-8 per pitch period), pseudo-random noise [13], or some combination of both. If the excitation is generated only at the receiver, based on a transmitted pitch period and voicing information, then the system is designated as an LPC vocoder. LPC vocoders provide speech with bitrates comprised roughly between 2.0 and 4.8 kb/s [13].

Coder quality given in MOS for several coders is displayed versus bitrate in Fig. 6 conforming with Table 4.
<table><caption>**Table 5:** Categories of Speech Coding Algorithms. LPC is Linear Prediction coding. AS is Analysis by Synthesis [13]. LPC-based Analysis-by-Synthesis (AS) coders (LPC-AS) choose an excitation function by testing a large set of candidate excitations and choosing the best. LPC-AS coders are used in most standards between 4.8 and 16 kb/s. Sub-band coders are frequency-domain coders that describe speech signal from spectral features pertaining to different frequency bands.</caption><thead><tr><th>Speech Coder Class</th><th>Bitrate (kb/s)</th><th>Complexity</th><th>Applications</th></tr></thead><tbody><tr><td>Waveform coders</td><td>16-64</td><td>Low</td><td>Terrestrial telephone</td></tr><tr><td>Sub-band coders</td><td>12-256</td><td>Medium</td><td>Teleconferencing, Audio</td></tr><tr><td>LPC-AS</td><td>4.8-16</td><td>High</td><td>Wireless Digital cellular</td></tr><tr><td>LPC vocoder</td><td>2.0-4.8</td><td>High</td><td>Satellite telephony, military</td></tr></tbody></table>

<table><caption>**Table 6:** Comparison of voice encoding methods. Waveform coders are: DM is Delta Modulation, ADM is Adaptive DM, DPCM is Differential PCM, ADPCM is Adaptive Differential PCM. Note the very low bitrate of LPC.</caption><thead><tr><th>Encoding method</th><th>Sampling rate, kHz</th><th>Bits per sample</th><th>Bitrate, kb/s</th></tr></thead><tbody><tr><td>DM</td><td>64-128</td><td>1</td><td>64-128</td></tr><tr><td>PCM</td><td>8</td><td>7-8</td><td>56-64</td></tr><tr><td>ADM</td><td>48-64</td><td>1</td><td>48-64</td></tr><tr><td>DPCM</td><td>8</td><td>4-6</td><td>32-48</td></tr><tr><td>ADPCM</td><td>8</td><td>3-4</td><td>24-32</td></tr><tr><td>LPC</td><td>0.04-0.1</td><td>80</td><td>3-8</td></tr></tbody></table>

## C. Code Excited Linear Prediction

Code Excited Linear Prediction (CELP) makes LP coefficients of the input signal to be first analyzed and then quantized to be used in an LP synthesis filter driven by the output produced by excitation codebooks [13].

Encoding is performed in two steps. Long-term prediction coefficients are calculated in the first step. In the second step, a perceptually weighted error between the input signal and the output of the LP synthesis filter is minimized [13].

LP coefficients are quantized by vector quantization and the excitation can be either Multiple Pulse Excitation (MPE) or Regular Pulse Excitation (RPE) [14].

Both model the excitation signal by multiple pulses, however, a difference exists in the degrees of freedom for pulse positions. MPE allows more freedom on the interpulse distance than RPE which has a fixed interpulse distance. Thus MPE achieves better coding quality than RPE [13] while RPE requires less computing effort.

CELP coder consists of three modules [14]: a CELP core encoder, a bitrate scalable tool, and a bandwidth extension

<figure><img src="image_9.png" /><figcaption>**Fig.6:** Comparison of speech coders quality with Mean Opinion Score (MOS) value as explained in Table 4. PCM16 is based on 16 bit quantization resulting into 128 kb/s bitrate corresponding to 2 DS0 whereas the A/μ law coder corresponds to a single DS0. LPC achieves the lowest bitrate at a poor quality price.</figcaption></figure>
tool. The CELP core encoder provides the basic encoding functions.

Quantized coefficients and pitch delay information as well as the wideband speech to be encoded are supplied from the core encoder to the bandwidth extension tool.

CELP core encoder [14] performs LP analysis and pitch analysis on the input speech signal. The obtained LP coefficients, the pitch lag (phase or delay), pitch and gains, and excitation signal are encoded as well as mode information.

Due to their high coding efficiency, variations of CELP coding techniques together with other algorithms have enabled speech waveform coders to reduce 32 kb/s G.726 ADPCM bitrate to about 4 kb/s while maintaining roughly the same speech quality.

The first halving of bitrate from 32 kb/s to 16 kb/s happened with the G.728 low-delay CELP (LD-CELP) coder. At the cost of increasing the buffering delay from one to five samples (0.625 ms at 8 kHz sampling rate), the G.728 coder halved the bitrate of 32 kb/s G.726 ADPCM while maintaining equivalent or better speech quality for all conditions tested.

The second halving of bitrate to 8 kb/s happened with the G.729 conjugate-structure algebraic CELP (CS-ACELP).

The third halving of bitrate to 4 kb/s happened with a 4 kb/s coder based on CELP achieving an equivalent speech quality as 32 kb/s G.726 ADPCM at least for clean speech conditions at the cost of further increasing the buffering delay to 30-35 ms.

<table><caption>**Table 7:** Speech coders with many varieties of CELP hybrid coders depending on the algorithm they use in comparison with PCM and ADPCM waveform coders. Varieties of CELP (Code Excited Linear Prediction) are Mixed Excitation Linear Prediction (MELP) LD for Low Delay, ACELP for Algebraic, CS for Conjugate Structure... Varieties are also indicated by their Standard Body affiliation such as GSM (Groupe Spécial Mobile), IS (Digital Cellular Communications), Inmarsat (Satellite Communications), FS (Secure Communications) or a company such as Qualcomm (QCELP). Algebraic CELP (ACELP) algorithm creates an excitation by choosing one vector from an adaptive codebook and another from a fixed codebook. HR is Half-Rate (HR) as used by GSM HR VSELP. VSELP (Vector-Sum Excited LPC algorithm) (Note SELP and not CELP) models the excitation vector as sum of one adaptive and two random code-vectors. Note that G.722 is a standard for wideband speech coding indicated by SBC (Speech Band Coding), a hybrid coder. MIPS is Million Instructions Per Second. Adapted from Proakis [15]</caption><thead><tr><td>Algorithm</td><td>Bitrate (kb/s)</td><td>MOS</td><td>Complexity (MIPS)</td><td>Frame Size (ms)</td></tr></thead><tbody><tr><td>PCM G.711</td><td>64</td><td>4.3</td><td>0.01</td><td>0</td></tr><tr><td>ADPCM G.726</td><td>32</td><td>4.1</td><td>2</td><td>0.125</td></tr><tr><td>SBC G.722</td><td>48/56/64</td><td>4.1</td><td>5</td><td>0.125</td></tr><tr><td>LD-CELP G.728</td><td>16</td><td>4.0</td><td>30</td><td>0.625</td></tr><tr><td>CS-ACELP G.729</td><td>8</td><td>4.0 (3.8)</td><td>20 (11)</td><td>10</td></tr><tr><td>G.723.1</td><td>6.3/5.3</td><td>4.0/3.7</td><td>11</td><td>10</td></tr><tr><td>GSM HR VSELP</td><td>6.3</td><td>3.4</td><td>14</td><td>20</td></tr><tr><td>IS-54 VSELP</td><td>8</td><td>3.5</td><td>14</td><td>20</td></tr><tr><td>IS-96 QCELP</td><td>1.2/2.4/4.8/9.6</td><td>3.3</td><td>15</td><td>20</td></tr><tr><td>Inmarsat-B APC</td><td>9.6/12.8</td><td>3.1/3.4</td><td>10</td><td>20</td></tr><tr><td>MELP</td><td>2.4</td><td>3.2</td><td>40</td><td>22.5</td></tr><tr><td>FS 1016-CELP</td><td>4.8</td><td>3.2</td><td>16</td><td>30</td></tr></tbody></table>

Another variety of coders is a multimode coder such as QCELP (developed by Qualcomm), a standard (cf. Table 8) that was adopted by the North American digital cellular standard based on code-division multiple access (CDMA) technology allowing access to 30 simultaneous calls. The coder selects one of four different data rates (1 kb/s, 2 kb/s, 4 kb/s and 8 kb/s) every 20 ms depending on the speech activity. Good quality speech is obtained at an average rate of 4 kb/s [13].

Another example is Standard G.729 (ACELP) a speech coder operating at 8 kb/s, based on algebraic code-excited LPC. G.723.1 operates at either 5.3 or 6.3 kb/s [13] (cf. Table 8).
<table><caption>**Table 8:** Wideband (WB) speech coders. AMR is Adaptive Multi-Rate used initially in NB speech like GSM and later extended to WB speech. ISDN is Integrated Services Digital Network that is equivalent to 128 kb/s i.e. 2 DS0 or twice the basic 64 kb/s rate. ACELP is Algebraic Code Excited Linear Prediction. MIPS is Million Instructions Per Second. WMOPS is Weighted Million Operations Per Second. VoIP is Voice over IP network. Note that Instruction Per Second applies to float number arithmetic operations whereas Operations Per Second applies to integer arithmetic operations. Adapted from Proakis [15]</caption><tr><td>Standard</td><td>G.722</td><td>G.722.1</td><td>AMR-WB (or G.722.2)</td></tr><tr><td>Bitrate(kb/s)</td><td>48, 56, 64</td><td>24, 32</td><td>23.85, 23.05, 19.85, 18.25, 15.85, 14.25, 12.65, 8.85, 6.6</td></tr><tr><td>Type</td><td>Sub-Band ADPCM</td><td>Transform Coding</td><td>ACELP</td></tr><tr><td>Frame size</td><td>0.125 ms</td><td>20 ms</td><td>20 ms</td></tr><tr><td>Quality</td><td></td><td>Poor performance @ 64 kb/s in some special conditions</td><td>Good performance at 12.65 kb/s rates and higher<br>15.85 ≥ G.722 @ 56 kb/s<br>23.05 ≥ G.722 @ 64 kb/s</td></tr><tr><td>Complexity</td><td>10 MIPS</td><td>&lt; 15 MIPS</td><td>38 WMOPS</td></tr><tr><td>Applications</td><td>ISDN</td><td>ISDN &amp; VoIP</td><td>3G wireless</td></tr><tr><td></td><td>Video Conferencing</td><td></td><td>Similar to G.722.1</td></tr></table>

## D. Vocoders

Narrowband voice coding has been used in cellular phone systems since the initial deployment of vocoders whose goal is to reduce bandwidth while simultaneously delivering acceptable call quality.

Voice measurement is based on selecting a set of listeners to assess the quality of spoken sentences after coding and decoding by some algorithm. Listeners are asked to evaluate recordings they heard by comparison to some reference standards i.e. analog voice recordings or 64 kb/s PCM samples.

A Mixed-Excitation Linear Prediction (MELP) vocoder operates at at 2.4 kb/s (cf. Table 7). It is based on the LPC model with additional features that include mixed excitation, aperiodic pulses, adaptive spectral enhancement, pulse dispersion filtering, and Fourier magnitude modeling [13].

# III. DIGITAL AUDIO

## A. MPEG 1, 2, 4

MPEG standard for Audio and Video is organized with MPEG1, MPEG2 and MPEG4 [16] as follows:

1. MPEG1: Compact Disk (CD), desktop computer monitor and Single Definition (SD) Terrestrial TV

2. MPEG2: DVD and High Definition (HD) Terrestrial TV

3. MPEG4: Blu-Ray, Hand-held devices such as e-tablets, mobile phones and Head-mounted devices such as virtual reality goggles and helmets

It targets storage media (CD, DVD and Blu-Ray), broadcast, digital networks and other devices such as e-books, e-tablets, Virtual Reality gears, wrist-watches and smartphones.

MP3 is an audio compression method that is a shortcut to MPEG1-Layer III standard. It was created by Karlheinz Brandenburg [14] in his PhD thesis to compress initially Audio CD. It should not be confused with MPEG3 standard that existed for a short time and was replaced later in parts by MPEG2 and MPEG4.

MP3 coding is based on human cochlea frequential sensitivity that may be described by a set of 24 overlapping critical frequency bands with different bandwidths acting like a filter bank to analyze any incoming sound. In MP3 these critical bands are mapped onto a frequency uniform scale as displayed in Fig. 7. The number of bands (32) is arbitrarily chosen since a power of 2 simplifies Fourier analysis.
Uniform Audio Filter Bands

<figure><img src="image_10.png" /></figure>

Non-uniform critical Bands

<caption>**Fig.7:** Mapping of the 32 non-uniform frequency critical bands onto a uniform scale. Human cochlea contains in fact only 24 bands. 32 being a power of 2 simplifies tremendously Fourier analysis. Adapted from Pan [17]</caption>

MPEG1/audio standard has 3 distinct layers for compression. Layer I forms the most basic algorithm while Layer II and Layer III are enhancements that use some elements found in Layer I. Each successive layer improves the compression performance but at the cost of greater encoder and decoder complexity [17].

• Layer I

Layer I algorithm codes audio in frames of 384 audio samples. It does so by grouping together 12 samples from each of the 32 sub-bands (thus 11x32=384), as shown in Fig. 7. Each group of 12 samples gets a bit allocation [17] and, if bit allocation is not zero, a scale factor. Bit allocation tells the decoder the number of bits used to represent each sample. In Layer I this allocation is 0 to 15 bits per sub-band. The scale factor resizes the samples to make full use of the quantizer range. Each scale factor is a 6 bit word. The decoder multiplies the decoded quantizer output with the scale factor to recover the quantized sub-band value. The dynamic range of the scale factors alone is over 120 dB. Compression rate is: 1:4

• Layer II

Layer II improves algorithms of Layer I. It codes audio data in larger groups and makes restrictions on possible bit allocations pertaining to middle and higher sub-bands. It also uses a more compact coding of bit allocation, scale factor values, and quantized samples. This allows layer II to provide better audio quality by saving bits such that more bits are available for sub-band values [17]. Layer II encoder forms frames of 1152 samples per audio channel. Layer I codes data in single groups of 12 samples for each sub-band whereas layer II codes data in 3 groups of 12 samples (3x384=1152) for each sub-band. Compression rate is: 1:6-8

• Layer III

Layer III algorithm is based on the same filter bank present in Layer I and Layer II, Layer III compensates for some of critical band deficiencies by processing audio data with a special block processing [17]. Compression rate is: 1:12.

When the actual (S/N) (signal to noise ratio) is significantly smaller than the theoretical upper bound, some digital audio recording systems chose a number of quantization bits *n* = 14 in order to obtain Hi-Fi quality with (S/N)=84 dB (not accounting for quantization noise). Picking a bandwidth-conserving alternative to increasing *n*, Audio performance may be improved through the use of companding, equivalent to non-uniform quantization as done in MP3 coding.

Advanced Audio Coding (AAC) is an extension of MPEG2 Layer III (non-uniform quantization) and exists also in MPEG4 where VQ is used. It can encode with greater quality and lower data rate than MP3. It also supports multichannel audio (1-48 channels) including low-frequency effects channels used in movies with a quality superior to Dolby AC3 used in DVD and Dolby Digital movies.

B. ATRAC3

Sony has developed ATRAC3 with the following features:

1. High Coding Efficiency.

Coding efficiency is about twice that of ATRAC (Mini-Disk), and realizes the same level of sound quality at
<table><caption>**Table 9:** MPEG characteristics and Quantization methods. Note that in MPEG-1/2 Layers I/II time-frequency structure is given by the number of critical bands since audio samples (32x12=384) are not grouped, whereas in MPEG-1/2 Layers III, audio samples are grouped into (384/2=192) pair or triplet (3x192= 576) sets. AAC is Advanced Audio Coding and AC3 is Dolby 5.1 audio channel standard for home Cinema. VQ is vector quantization and TwinVQ is Transform-domain weighted interleaved VQ developed by NTT (Nippon Telecom and Telephone) Japan. Audio LPC differs from Speech LPC in the sense it uses a modified discrete Fourier cosine transform involving VQ coefficients [18]. ATRAC is Adaptive Transform Acoustic Coding developed by Sony Japan for the Magneto-Optic Mini-Disc and Sony Digital Dynamic Sound (SDDS) format for Digital Cinema. Adapted from Ref. [7].</caption><thead><tr><th>Standard</th><th>Time-frequency structure</th><th>Quantization</th><th>Coding</th></tr></thead><tbody><tr><td>MPEG-1/2 Layers I/II</td><td>32</td><td>uniform</td><td>Block companding</td></tr><tr><td>MPEG-1/2 Layers III</td><td>576/192</td><td>non-uniform</td><td>Huffman coding [1]</td></tr><tr><td>MPEG2 AAC</td><td>1024/128</td><td>non-uniform</td><td>Huffman coding [1]</td></tr><tr><td>MPEG4 TwinVQ</td><td>1024/128 (960/120)</td><td>VQ</td><td>Audio LPC</td></tr><tr><td>AC3</td><td>256/128</td><td>VQ</td><td>Audio LPC</td></tr><tr><td>ATRAC</td><td>96 .. 512</td><td>uniform</td><td>Block companding</td></tr></tbody></table>

128 kb/s/Stereo.

2. Wide Range of Applications.

ATRAC3 is the most suitable for mobile players, recorders and USB keys. Note that any portable device with copyright protection mechanisms and a network connection paves the way for streaming services.

3. Encoding spectral coefficient data obtained from a signal analysis consisting of a band splitting filter and block processing in an efficient way based on signal characteristics.

4. Reducing further bitrate by adopting a joint stereo coding scheme taking advantage of correlation between both channels.

5. ATRAC3 is the basis of multichannel audio sound with 7.1 channels called SDDS (Sony Dynamic Digital Sound) for cinema theaters.

## C. DTS

DTS (Digital Theatre Sound) precedes SDDS in Cinema Theater applications [19]. It uses a different approach than AC3. Audio data are stored on a CD, separately from film and synchronized with picture via a time code. A compression ratio of only 1:4 is adopted to store the multichannel data information. For comparison, given film medium response and error rate, AC3 originally used a compression ratio of 1:12 like MP3.

The consumer version of the DTS system is specified as follows:

Number of Audio Channels= 1-10.1, Sampling frequency = 8-192 kHz, Quantization =16-24 (bit per sample)
Bitrate= 32- 6144 kb/s, Frame Size= 512 samples.

In the consumer version of the DTS system, frequency mapping uses 32 channels between 0 and 24 kHz unlike MPEG mapping that goes up to 26 kHz (cf. Fig. 7). Eight additional sub-bands and two additional sub-bands can be employed to cover frequencies ranging between 24 and 48 kHz and 48 to 96 kHz respectively.

For most applications DTS applies relatively small compression ratios operating in lossless mode. Very good performance is achieved for data rates around 1 Mb/s in the 5.1-channel case. DTS finds besides Cinema, applications such as storage of multichannel audio on CD and DVD. Its flexible bitstream format and fine tuning of audio quality for multichannel audio coding make DTS an ideal candidate for high quality, multichannel Digital Audio and Video Broadcast (DAB and DVB).
## D. Audio Broadcast

Digital Audio Broadcast (DAB) is the successor of Analog FM radio.

Multipath interference is a major problem that plagues broadcast and wireless communications originating from signal multiple reflections creating echoes producing destructive interference.

In presence of multipath interference, echo delay is often longer than symbol duration of the main signal. This results in a high level of interference. Echoes can be treated by increasing symbol duration leading to additional bandwidth required.

Broadcasting needs modulation techniques given in Table 10 along with their spectral efficiency.

<table><caption>**Table 10:** Digital modulation methods. ASK is Amplitude Shift Keying with 1 bit coding yielding 1 or 0 amplitude, whereas PSK is Phase Shift Keying yielding 0, $\pi$ phase corresponding to 0,1 bit value respectively. Adapted from Ref. [20].</caption><thead><tr><td>Modulation method</td><td>Spectral Efficiency</td></tr></thead><tbody><tr><td>ASK</td><td>1 bit/second/Hz</td></tr><tr><td>PSK</td><td>1 bit/second/Hz</td></tr><tr><td>QPSK</td><td>2 bits/second/Hz</td></tr><tr><td>8-PSK</td><td>3 bits/second/Hz</td></tr><tr><td>16-QAM</td><td>4 bits/second/Hz</td></tr><tr><td>32-QAM</td><td>5 bits/second/Hz</td></tr><tr><td>64-QAM</td><td>6 bits/second/Hz</td></tr><tr><td>256-QAM</td><td>8 bits/second/Hz</td></tr></tbody></table>

A special modulation technique called Coded Orthogonal Frequency Division Multiplex (COFDM) is particularly adapted for combating multipath interference in DAB and DVB as well (cf. Section IV C).

COFDM originates from OFDM, a multicarrier transmission technology that is currently being used in DAB and has proved its robustness against multipath interference with all adjacent carrier frequencies being orthogonally polarized.

Moreover, DAB is based on MPEG1 Layer 2 (compression rate 1:6) stereo sound, contrasting with MP3 that uses Layer 3 and a higher compression rate of 1:12.

DAB uses 1.536 MHz spectrum with a data rate of up to 1.5 Mb/s. This allows for up to six audio channels (akin to Dolby AC3) at 256 kb/s.

DAB yields higher spectral efficiency, measured in programmes/MHz/transmitter site than analog systems. In many countries, this has led to a higher number of stations available to listeners.

In order to illustrate spectral efficiency, consider analog FM that requires 200 kHz [1] or 0.2 MHz per station. Frequency re-use factor is about 15 for stereo transmissions meaning (in the case of stereo FM) that only 15 transmitter sites can use the same channel frequency without problems arising from co-channel interference.

Assuming a total availability of 102 FM channels with 0.2 MHz bandwidth (102 x 0.2=20.4 MHz), the 20.5 MHz FM band [87.5-108.0] MHz allows an average of 102/15 = 6.8 possible radio channels at each transmitter site. This yields a spectral efficiency of $1/15/(0.2 \text{ MHz}) = 0.30$ programmes/transmitter/MHz.

DAB with 192 kb/s codec requires $1.536 \text{ MHz} \times (192 \text{ kb/s})/(1136 \text{ kb/s}) = 0.26 \text{ MHz}$ per station or 260 kHz which is similar to stereo FM (with 200 kHz). The frequency re-use factor for local channels and multi-frequency broadcasting networks is typically 4 or 5, resulting in $1/4/(0.26 \text{ MHz}) = 0.96$ programmes/transmitter/MHz. This is 3.2 times more efficient than analog FM for local stations since $3.2 \times 0.30 = 0.96$.

For single frequency network transmission, such as country-wide broadcast, the channel re-use factor is only 1, resulting in $1/1/0.26 \text{ MHz} = 3.85$ programmes/transmitter/MHz, which is 12.83 times more efficient than FM for national and regional networks, since $12.83 \times 0.30 = 3.85$.

Multichannel audio technology of DAB can be improved quality wise with DTS, or increasing the number of channels by adopting MPEG2 AAC or MPEG4 AAC. Indeed in 2006, DAB+, an upgrade to DAB was adopted with High Efficiency AAC Version 2 (HE-AAC v2) audio codec (also known as eAAC+). DAB+, has also adopted the MPEG2 Surround audio format with better error correction coding.
IV. DIGITAL VIDEO

Video encoding software allows for a great number of configuration parameters to be set according to a desired outcome. Some of the most important encoding parameters are briefly described below.

* Resolution: The total number of pixels in every video frame. Higher resolution videos have more pixels and the ability to deliver more detail. They subsequently require more computing power to process and store.

* Frame rate: The amount of frames per second. The higher the frame rate, the smoother a video stream is.

* Compression: The level of compression applied to a video results in a bitrate. The bitrate can be constant or variable depending on encoding.

There are, of course, other crucial factors in video encoding such as rate control mode, frame time control, encoding profile, color space...

The ultimate goal of video encoding is to find the optimal balance between compression and quality. This balance is defined by the combination of bitrate, resolution and frame rate.

A. MPEG 1, 2, 4

In the video case, performing bitrate calculation assumes a typical frame rate along image size, resolution, aspect ratio, color and noise. Reducing bitrate of a video stream is based on exploiting redundancies in order to reduce data flow volume such as:

* Spatial: An image might have a fixed background (like in computer animations) or contain repeating patterns

* Temporal: Frames might evolve slowly such that it is possible to apply motion estimation algorithms to predict evolution

* Perceptual: Human visual system is more sensitive to BW than color implying it is possible e.g. to sub-sample color or to use filterbanks containing essential frequential components perceived by humans.

* Entropic: Statistical redundancy in frames can be allocated less data when the pattern is highly repetitive

<table><caption>**Table 11:** Comparison between different video formats. MJPEG or Motion JPEG consists of a stream of JPEG images which is a lot less sophisticated than MPEG4 and H.264.</caption><thead><tr><th>Standard</th><th>Compression ratio</th><th>Advantages</th><th>Disadvantages</th></tr></thead><tbody><tr><td>MJPEG</td><td>1:20</td><td>Low CPU usage<br>Clearer images than MPEG4 at lower frame rates<br>Not sensitive to motion complexity</td><td>Poor efficiency in comparison to MPEG4 and H.264<br>Poorer Quality with complex textures, lines, and curves</td></tr><tr><td>MPEG4</td><td>1:50</td><td>Good for streaming and TV broadcasting<br>Compatibility with many digital and mobile devices</td><td>Sensitive to motion complexity with lower compression in some cases<br>High CPU usage</td></tr><tr><td>H.264</td><td>1:100</td><td>Generally efficient<br>Highly efficient in slow-motion content</td><td>Highest CPU usage<br>Sensitive to motion complexity with lower compression in some cases</td></tr></tbody></table>

Estimation of video bitrate is based on several frame parameters such as:

* Image size:

* Digital Color sub-sampling management:

YC<sub>b</sub>C<sub>r</sub> - General representation of a video signal containing one luma (luminance) and two chroma (chrominance) components. For example, MPEG2 is based on 4:2:0 YC<sub>b</sub>C<sub>r</sub>.
<figure><img src="image_11.png" /><figcaption>**Fig.8:** Partitioning of a frame into macroblocks, sub-macroblocks and slices. (Left) Macroblocks made of 16 x 16-pixel are typically composed by a 16x16 block of luminance (Y) samples, and two 8x8 blocks of chrominance samples (with blue $C_b$ and red $C_r$ components. Sub-macroblocks: 8x8, 8x4, 4x8 and 4x4 allow flexible intra-frame prediction. (Right) Partitioning into slices allows even more flexible intra-frame prediction. Adapted from Wiegand *et al.* [21]</figcaption></figure>

<table><caption>**Table 12:** Common image formats with frame size, rate and uncompressed bitrates. †A vertical sub-sampling of chroma components by a factor of 2 is assumed. BT.601 is an ITU (International Telecommunication Union a European organization) video standard, with resolution and frame rate given by 720 x 576 x 30. SIF is Source Input Format with 352x240 pixels when frame rate is 30p and 352x288 pixels when frame rate is 25p. CIF is Common Intermediate Format with a resolution of 352x288. QCIF is Quarter Common Intermediate Format with a resolution of 176 x 144. Note that a 29.97 frame rate is used instead of 30 in order to avoid sound interference caused by choice of color sub-carrier.</caption><thead><tr><th>Format</th><th>Name</th><th>Frame Size</th><th>Frame Rate</th><th>Uncompressed rate</th><th>MByte/s</th></tr></thead><tbody><tr><td>QCIF</td><td></td><td>176 x 144</td><td>29.97</td><td></td><td>2949.21</td></tr><tr><td>SIF</td><td></td><td>352 x 240</td><td>29.97</td><td></td><td>9830.70</td></tr><tr><td>CIF</td><td></td><td>352 x 288</td><td>29.97</td><td></td><td>11796.84</td></tr><tr><td>BT.601†</td><td></td><td>720 x 576</td><td>25</td><td></td><td>40500.00</td></tr><tr><td>HDTV†</td><td></td><td>1920 x 1035</td><td>30</td><td></td><td>232875.00</td></tr><tr><td>UHD (4K)</td><td></td><td>3840 x 2160</td><td>30</td><td></td><td>931550.00</td></tr></tbody></table>

4:4:4 - A video format for high-end studio recordings, where Y, $C_b$, and $C_r$ are sampled equally.

4:2:2 - A video format commonly used for studio recordings, where there is one $C_b$ sample and one $C_r$ sample for every two Y samples (1 pixel in a 1 x 2 grid). 2:1 horizontal down-sampling with no vertical down-sampling. Thus same number of samples are allocated to chroma and luma.

4:2:0 - A video format used by DVD, where there is one $C_b$ sample and one $C_r$ sample for every four Y samples (1 pixel in a 2 x 2 grid). 2:1 horizontal down-sampling and 2:1 vertical down-sampling. $C_b$ and $C_r$ are sampled on every other line, with one set of chroma samples for each two luma samples on a line. This amounts to a sub-sampling of chroma by a factor of two compared to luma (and by a factor of four for a single $C_b$ or $C_r$ component).

4:1:1 - Component digital video format with one $C_b$ sample and one $C_r$ sample for every four Y samples. 4:1 horizontal down-sampling with no vertical down-sampling. Chroma is sampled on every line, but only for every four luma pixels (1 pixel in a 1 x 4 grid). This amounts to a chroma sub-sampling by a factor of two compared to luma (and by four for a single $C_b$ or $C_r$ component).

Analog bandwidth is given by $n_H n_V n_F = 221$ MHz with $n_H$, the horizontal number of spots per line, $n_V$ the number of lines per frame whereas $n_F$ is the number of frames/second. In Blu-Ray 4K format, bitrate is given by: $r = 4096 \times 2160 \times 25 \times 30$ (pixels/frame) (frames/second) (bits/pixel).

**Note:** 4K UHD-TV frame format is 3840 x 2160 that is slightly different from the Blu-Ray 4096 x 2160 format. According to Nyquist theorem [1], equivalent bandwidth is half-bitrate or $B = r/2$. Thus $B = 3.3$ GHz amounting to 15 times the analog bandwidth of 221 MHz. Each chroma component $C_r, C_b$ has same bandwidth value. This implies that $C_r, C_b$ possess individually 1.65 GHz bandwidth giving a total bandwidth of 6.6 GHz.
Let us apply these results to evaluate the volume of an UHD (4K) movie of duration 90 minutes with a typical compression rate of 1:200.

Total volume is given by $V = r \times \tau$ with r the bitrate and $\tau$ the total movie duration. Thus $V = 6.6 \times 10^9 \times 90 \times 60 / 200 = 178 \times 10^9$ bits or 22.27 GBytes (after division by 8 since 1 Byte=8 bits). A Blu-Ray disk such as BD25 (that stores about 25 GBytes) is sufficient to store 22.27 GBytes with its ancillary data.

<table><caption>**Table 13:** Comparison of Video compression standards [21]. MJPEG is Motion JPEG. Digital Cinema based on MJPEG-2K use pictures compressed with Wavelets [22] yielding higher compression. PSTN is Public Switched Telephone Network. DVB is Digital Video Broadcast. Macroblocks and sub-macroblocks are depicted in Fig. 8. HEVC is High Efficiency Video Coding version of H.264.</caption><thead><tr><th>Standard</th><th>Compression method</th><th>Application</th><th>Bitrate</th></tr></thead><tbody><tr><td>MJPEG</td><td>Still image compression</td><td>Digital Cinema</td><td>Variable</td></tr><tr><td>MJPEG-2K</td><td>Wavelet Image Compression</td><td>Digital Cinema</td><td>Variable</td></tr><tr><td>MPEG1</td><td>Macroblock</td><td>Early Digital Television and CD Video</td><td>1.5Mb/s</td></tr><tr><td>MPEG2</td><td>Macroblock</td><td>Digital Television and DVD Video</td><td>2-20 Mb/s</td></tr><tr><td>MPEG4</td><td>Object-based coding, Synthetic Content, Interactivity</td><td>Blu-Ray Video</td><td>Mb/s</td></tr><tr><td>H.261</td><td>Macroblock</td><td>Mobile devices</td><td>Many 10 to 100 kb/s</td></tr><tr><td>H.263</td><td>Macroblock</td><td>Video conferencing over ISDN</td><td>128 kb/s</td></tr><tr><td>H.264</td><td>Sub-macroblock</td><td>Video telephony over PSTN</td><td>33.6kb/s</td></tr><tr><td>H.264/ MPEG4-HEVC</td><td>Intra-frame prediction</td><td>Blu-Ray Video, DVB Mobile devices</td><td>Many 10 to 100 kb/s Mb/s</td></tr><tr><td>H.264/ MPEG4-HEVC</td><td>Slice, Intra-frame prediction</td><td>Blu-Ray Video, DVB Mobile devices</td><td>Many 10 to 100 kb/s</td></tr></tbody></table>

## B. H264

H.264 codec treats intra-frame spatial redundancies in order to reduce bitrates whereas previous MPEG codecs dealt only with temporal inter-frame redundancies. In MPEG, a frame is decomposed into macroblocks consisting of a number of luma and chroma pixels (cf. Table 12) and monitoring data change between neighboring macroblocks in order to perform motion estimation.

H.264 offers more flexibility than MPEG by decomposing a frame into sub-macroblocks (cf. Fig. 8) (8×8, 8×4, 4×8 and 4×4) in order to perform efficiently intra-frame prediction.

H.264 performs other compression operations leading to a bitrate that is half of MPEG2 with same overall quality. H.264 codec is a block-based compression system partitioning frames into macroblocks made of 16 x 16-pixel entities typically composed by a 16x16 block of luma (Y) samples, and two 8x8 blocks of chroma samples (with blue $C_b$ and red $C_r$ components). Macroblocks are used in H.264, MPEG1, MPEG2, MPEG4 [16], H.261 and H.263. There are three types of macroblocks [16] in MPEG2 and MPEG4:

* I-Macroblock: Intra-macroblocks use intra-frame prediction relying on neighboring samples inside a given frame.

* P-Macroblock: Predictive-macroblocks use inter-frame prediction with motion estimation from previous frames.

* B-Macroblock: Bidirectional-macroblocks use both forward and backward prediction with motion estimation from previous and next frames.

In High Efficiency Video Coding (HEVC) version of H.264, macroblocks are organized in slices, which represent areas of a given picture that can be decoded independently of each other. H.264/MPEG4-HEVC supports five slice-coding types.

* I slice: The simplest I slice (where I stands for intra). In I slices all macroblocks are encoded without referring to any other pictures of the video sequence.

* P and B slice: Previously encoded images can be used to form a prediction signal for macroblocks of the predictive-coded P and B slice types (where P stands for predictive and B stands for bi-directional).

* SP and SI slices: SP (switching P) and SI (switching I) slices, switch between data streams encoded at various bitrates.
A picture comprises slices representing a complete frame or field (in the interlaced scan case). Field-based coding is often used when the scene displays fast motion. In some cases parts of a frame are more efficiently encoded in field mode, while others are more efficiently encoded in frame mode. Hence, H.264/MPEG4-HEVC supports macroblock-adaptive switching between frame and field coding (MBAFF).

Each macroblock can be transmitted in one of several coding types depending on slice coding algorithm. All intra coding types in H.264/MPEG4-HEVC rely on prediction of samples in a given block performed in the spatial domain, rather than in the transform domain as in previous video coding standards.

Robustness to data errors or losses and flexibility for operation over a variety of network environments is enabled by a number of novel design aspects (cf. Fig. 9) introduced into the H.264/HEVC standard, such as:

*   **FMO: Flexible macroblock ordering (FMO):**
    Slices are an independently-decodable subset of a slice group. Flexible macroblock ordering can significantly enhance robustness to data losses by managing the spatial relationship between areas encoded in each slice.

*   **ASO: Arbitrary slice ordering (ASO):**
    Since each slice of a coded picture can be decoded independently of other picture slices, H.264-HEVC enables sending and receiving picture slices in any order speeding up processing.

*   **Deblocking Filter:**
    Block-based video coding produces artifacts that originate from both prediction and residual difference coding stages of the decoding process. The deblocking filter in H.264-HEVC is located within the motion-compensated prediction loop, improving quality in inter-picture prediction.

*   **CABAC and CAVLC: Context-Adaptive entropy coding:**
    While arithmetic coding was previously [21] an optional feature of H.263, a more effective use of this technique is found in H.264 to create two powerful entropy coding methods known as CABAC (Context-Adaptive Binary Arithmetic Coding) and CAVLC (Context-Adaptive Variable-Length Coding) to improve performance.

*   **Data Partitioning:**
    H.264-HEVC allows data of each slice to be separated into up to three different partitions, in order to represent the video source as faithfully as possible.

<figure><img src="image_12.png" /><figcaption>**Fig. 9:** H.264-HEVC coding. MBAFF is macroblock-adaptive switching between frame and field coding. Coding is divided into Main, Baseline, Extended and High parts containing various operations leading to compress the video stream. Adapted from Wiegand *et al.* [21]</figcaption></figure>

### C. Video Broadcast

Digital Video Broadcasting (DVB) needs modulation [1], source coding to protect data against multipath interference originating from echoes during signal propagation, channel coding, compression to save bandwidth...
DVB is a successor of HDTV in which data is transformed into bit packets [1] that are processed to filter out undesirable discrete frequency components. During channel coding, data is encoded with parity symbols to enable error correction at the receiver. Coded symbols are interleaved to minimize the effects of burst-type errors produced by channel noise that can corrupt successive symbols.

The encoded data is combined with synchronization signals then 8-VSB modulated. VSB is Vestigial Side Band analog modulation that was developed historically for NTSC [1] (North-American Analog TV) and 8-VSB is a digital modulation such that a set of 8 different baseband signals are used for modulation onto a carrier frequency.

Considering that a 24.9 MHz composite HDTV video signal has a bitrate of about 1 Gb/s, an NTSC 6-MHz television channel can transmit only 12 Mb/s, implying a compression ratio of 1:80 is required.

Other modulations such as 16-QAM, 32-QAM, 64-QAM and 256-QAM (cf. Table 10) are used in the coaxial cable (DVB-C) and satellite (DVB-S) broadcast cases (cf. Table 15).

Compression algorithms can reduce spatial and temporal redundancies, moreover during transmission of each frame, only those parts of a scene that vary are actually transmitted.

<table><caption>**Table 14:** Digital modulation applications circa 1997 [20]. Cable is abbreviated Coaxial Cable. DVB-T is Digital Video Broadcast-Terrestrial. DVB-C is Digital Video Broadcast-Cable. DVB-S is Digital Video Broadcast-Satellite. Adapted from Ref. [20].</caption><thead><tr><th>Modulation format</th><th>Application</th></tr></thead><tbody><tr><td>PSK</td><td>Cable TV modems</td></tr><tr><td>QPSK</td><td>Satellite, CDMA, DVB-S, Cable TV modems</td></tr><tr><td>8, 16-VSB</td><td>North American digital TV, Broadcast, Cable</td></tr><tr><td>8-PSK</td><td>Satellite, aircraft, pilots for monitoring video systems</td></tr><tr><td>16-QAM</td><td>Microwave digital radio, modems, DVB-C, DVB-T</td></tr><tr><td>32-QAM</td><td>Terrestrial microwave, DVB-T</td></tr><tr><td>64-QAM</td><td>DVB-C, modems, broadband set top boxes</td></tr><tr><td>256-QAM</td><td>Modems, DVB-C (Europe), Digital Video (US)</td></tr></tbody></table>

<table><caption>**Table 15:** Typical DVB Transmission System Parameters. DVB-S is Digital Video Broadcast over Satellite. DVB-C is Digital Video Broadcast over coaxial Cable. DVB-T is Digital Video Broadcast-Terrestrial. Carriers in DVB-T use Quadrature Phase Shift Keying (QPSK) modulation [1, 20] and channel coding uses special coding techniques to adjust code rates. Adapted from Ref. [23].</caption><thead><tr><th>DVB System</th><th>Modulation</th><th>Band (GHz)</th><th>Signal Bandwidth (MHz)</th><th>Bitrate (Mb/s)</th></tr></thead><tbody><tr><td rowspan="3">DVB-S</td><td>16-QAM</td><td>0.23-0.47 or 0.95-2.05</td><td>5.9-7.9</td><td>18.9-25.2</td></tr><tr><td>32-QAM</td><td>0.23-0.47 or 0.95-2.05</td><td>4.7-8.0</td><td>18.9-31.9</td></tr><tr><td>64-QAM</td><td>0.23-0.47 or 0.95-2.05</td><td>3.9-8.0</td><td>18.9-38.1</td></tr><tr><td rowspan="3">DVB-C</td><td>16-QAM</td><td></td><td>2.0-7.9</td><td>7.0-27.3</td></tr><tr><td>32-QAM</td><td></td><td>2.0-8.0</td><td>8.7-34.6</td></tr><tr><td>64-QAM</td><td></td><td>2.0-7.9</td><td>10.4-41.3</td></tr><tr><td rowspan="3">DVB-T</td><td>OFDM (QPSK)</td><td>0.3-3.0</td><td>7.6</td><td>5.0-10.6</td></tr><tr><td>OFDM (16-QAM)</td><td>0.3-3.0</td><td>7.6</td><td>10.0-21.1</td></tr><tr><td>OFDM (64-QAM)</td><td>0.3-3.0</td><td>7.6</td><td>14.4-31.7</td></tr></tbody></table>

In the terrestrial broadcast (DVB-T) case, it was realized that combating channel noise can be overcome by increasing symbol duration and consequently bandwidth. Fortunately, a tradeoff between bandwidth and symbol duration is possible as done by OFDM.

In fact, OFDM achieves a larger symbol duration within same bandwidth by demultiplexing a distinct symbol into several sub-symbols. Next, sub-symbols are modulated in parallel onto different carriers. Hence, sub-symbol duration is increased while total bandwidth (sum of all carrier frequencies) remains same. Next, modulated sub-symbols are added, resulting in transmission of resulting datastream.
OFDM transmission system is able to operate in a 2K mode with a maximum of 1,705 carriers per OFDM symbol or 8K mode with a maximum of 6,817 carriers per OFDM symbol possessing a longer duration.

# APPENDIX A: HISTORICAL CODES

## 1. Morse code

Samuel Morse used variable-size codes when he designed his telegraph code (cf. Table 16) with dots (dit) and dashes (dat). It might be said that he invented for the first time binary digital technology.

Morse established the first long-distance line, between Washington and Baltimore, which opened in May 1844 with a message that is the last sentence of verse 23 of chapter 23 of the "Book of Numbers": "What hath God wrought!"

<table><caption>**Table 16:** The Morse Code for English.</caption><thead><tr><th>Letter</th><th>Code</th><th>Letter</th><th>Code</th><th>Letter</th><th>Code</th><th>Letter</th><th>Code</th></tr></thead><tbody><tr><td>A</td><td>-</td><td>N</td><td>-</td><td>1</td><td>-</td><td>Full stop</td><td>-.-.-</td></tr><tr><td>B</td><td>-...</td><td>O</td><td>-</td><td>2</td><td>-</td><td>Comma</td><td>-.-</td></tr><tr><td>C</td><td>-.-</td><td>P</td><td>-</td><td>3</td><td>-</td><td>Semi-colon</td><td>-...</td></tr><tr><td>Ch</td><td>-</td><td>Q</td><td>-</td><td>4</td><td>-</td><td>Question mark</td><td>-.-</td></tr><tr><td>D</td><td>-..</td><td>R</td><td>-</td><td>5</td><td>-</td><td>Apostrophe</td><td>-.-</td></tr><tr><td>E</td><td>-</td><td>S</td><td>-</td><td>6</td><td>-</td><td>Hyphen</td><td>-....</td></tr><tr><td>F</td><td>-..</td><td>T</td><td>-</td><td>7</td><td>-</td><td>Dash</td><td>-.-</td></tr><tr><td>G</td><td>-</td><td>U</td><td>-</td><td>8</td><td>-</td><td>Dollar sign</td><td>-.-.-</td></tr><tr><td>H</td><td>...</td><td>V</td><td>-</td><td>9</td><td>-</td><td>Quotation mark</td><td>-.-.-</td></tr><tr><td>I</td><td>-</td><td>W</td><td>-</td><td>0</td><td>-</td><td></td><td></td></tr><tr><td>J</td><td>-</td><td>X</td><td>-</td><td></td><td></td><td></td><td></td></tr><tr><td>K</td><td>-.-</td><td>Y</td><td>-</td><td></td><td></td><td></td><td></td></tr><tr><td>L</td><td>-..</td><td>Z</td><td>-</td><td></td><td></td><td></td><td></td></tr><tr><td>M</td><td>-</td><td></td><td></td><td></td><td></td><td></td><td></td></tr></tbody></table>

The duration of a dot is taken to be one unit and for a dash it is three units. The interval between the dots and dashes of a single character is one unit. Interval between characters is three units and between words it is six units (five for automatic transmission).

## 2. Baudot code

<table><caption>**Table 17:** The Baudot Code. LS: Letter Shift, FS: figure Shift, CR: Carriage Return, LF: Line Feed, ER: Error, NA: Not Assigned, SP: Space.</caption><thead><tr><th>Letter</th><th>Code</th><th>Character</th><th>Letter</th><th>Code</th><th>Character</th></tr></thead><tbody><tr><td>A</td><td>10000</td><td>1</td><td>Q</td><td>10111</td><td>/</td></tr><tr><td>B</td><td>00110</td><td>8</td><td>R</td><td>00111</td><td>-</td></tr><tr><td>C</td><td>10110</td><td>9</td><td>S</td><td>00101</td><td>SP</td></tr><tr><td>D</td><td>11110</td><td>0</td><td>T</td><td>10101</td><td>NA</td></tr><tr><td>E</td><td>01000</td><td>2</td><td>U</td><td>10100</td><td>4</td></tr><tr><td>F</td><td>01110</td><td>NA</td><td>V</td><td>11101</td><td>'</td></tr><tr><td>G</td><td>01010</td><td>7</td><td>W</td><td>01101</td><td>?</td></tr><tr><td>H</td><td>11010</td><td>+</td><td>X</td><td>01001</td><td>,</td></tr><tr><td>I</td><td>01100</td><td>NA</td><td>Y</td><td>00100</td><td>3</td></tr><tr><td>J</td><td>10010</td><td>6</td><td>Z</td><td>11001</td><td>:</td></tr><tr><td>K</td><td>10011</td><td>(</td><td>LS</td><td>00001</td><td>LS</td></tr><tr><td>L</td><td>11011</td><td>=</td><td>FS</td><td>00010</td><td>FS</td></tr><tr><td>M</td><td>01011</td><td>)</td><td>CR</td><td>11000</td><td>CR</td></tr><tr><td>N</td><td>01111</td><td>NA</td><td>LF</td><td>10001</td><td>LF</td></tr><tr><td>O</td><td>11100</td><td>5</td><td>ER</td><td>00011</td><td>ER</td></tr><tr><td>P</td><td>11111</td><td>%</td><td>NA</td><td>00000</td><td>NA</td></tr></tbody></table>
Baudot code (cf. Table 17) is a 5-bit code developed by J. M. E. Baudot in about 1880 for telegraph communication and later used in modems.

It became popular and by 1950 was designated the International Telegraph Code No. 1. It was used in many first- and second-generation computers. The code uses 5 bits per character but encodes more than 32 characters. Each 5-bit code can be the code of two characters, a letter and a figure. The "letter shift" and "figure shift" codes are used to shift between letters and figures.

Using this technique, the Baudot code can represent 32 x 2 -2 = 62 characters (each code can have two meanings except the LS and FS codes). The actual number of characters, however, is smaller than that, because five of the codes have one meaning each, and some codes are not assigned.

Baudot code is not reliable because no parity bit is used. A bad bit can transform a character into another character. In particular, a bad bit in a shift character causes a wrong interpretation of all the characters following, up to the next shift. If the data includes just integers, each decimal digit may be represented in 4 bits, with two digits assembled in a byte.

[1] A. B. Carlson and P. B. Crilly, *Communication systems: An Introduction to Signals and Noise in Electrical Communication*, 5th Edition, McGraw-Hill, New York (2010).

[2] D. Jarman, *A brief introduction to Sigma Delta conversion*, AN9504, Intersil Corporation, Melbourne (Florida, USA) (1995).

[3] S. Orfanidis, *Introduction to Signal Processing*, Prentice-Hall (2010).

[4] H. Nakajima, *Digital Audio Technology*, Tab Books (1983).

[5] D. Salomon, *Data Compression: The Complete Reference*, Fourth Edition, Springer-Verlag London Limited (2007).

[6] W. H. Press, W. T. Vetterling, S. A. Teukolsky and B. P. Flannery, *Numerical Recipes in C: The Art of Scientific Computing* Third Edition, Cambridge University Press, New-York (2007).

[7] J. Herre, *Temporal Noise Shaping, Quantization and Coding Methods in Perceptual Audio Coding: A Tutorial Introduction*, AES 17 International conference on High Quality Audio Coding (1999), N. S. Jayant, J. D. Johnston and R. J. Safranek, *Signal Compression Based on Models of Human Perception*, Proc. **IEEE**, 81, 10 (1993).

[8] M. Nelson, *The data compression book*, Prentice Hall, (1991).

[9] J. Weiss and D. Schremp, *Putting data on a diet*, **IEEE Spectrum**, 36 (1993).

[10] A. Spanias, *Speech coding: A tutorial Review*, Proc. **IEEE**, 1541, (1994).

[11] W. C. Chu, *Speech Coding Algorithms: Foundation and Evolution of Standardized Coders*, Wiley-Interscience (2003).

[12] D. E. Comer, *Computer Networks and Internets*, Fifth Edition, Pearson Education, Inc. Upper Saddle River, New Jersey (2009).

[13] M. Hasegawa-Johnson and A. Alwan, *Speech Coding: Fundamentals and applications*, in *Wiley Encyclopedia of Telecommunications*, edited by J. G. Proakis, John Wiley & Sons, Inc. (2003).

[14] K. Brandenburg and M. Bosi, *Overview of MPEG audio: Current and future standards for low bitrate audio coding*, J. Audio Eng. Soc., 45(1/2), 4 (1997). K. Brandenburg, O. Kunz and A. Sugiyama, *MPEG-4 natural audio coding*, Signal Processing: Image Communication 15, 423 (2000).

[15] J. G. Proakis, *Wiley Encyclopedia of Telecommunications*, John Wiley & Sons, Inc. (2003).

[16] J. Watkinson, *The MPEG Handbook MPEG1, MPEG2, MPEG4*, Focal Press, An imprint of Butterworth-Heinemann Linacre House, a division of Reed Educational and Professional Publishing Ltd (2001).

[17] D. Y. Pan, *Digital Audio Compression*, Digital Technical Journal, Vol. 5, No. 2, 1 (1993).

[18] T. Painter and A. Spanias, *Perceptual Coding of Digital Audio*, Proc. **IEEE**, 88, (2000).

[19] M. Bosi, *Multichannel Audio Coding and its applications in DAB and DVB*, Journal of the Audio Engineering Society, Vol. 46, number 6 (2000).

[20] *Digital Modulation in Communications Systems: An Introduction*, Hewlett-Packard Application Note 1298 (1997).

[21] G. J. Sullivan, J. R. Ohm, W. J. Han and T. Wiegand, *Overview of the High Efficiency Video Coding (HEVC) Standard*, **IEEE Trans. Circuits and Systems for Video Technology**, **CSVT-22**, 1649 (2012).

[22] I. Daubechies, *Wavelets*, SIAM, Philadelphia (1992).

[23] R. de Bruin and J.M. Smits, *Digital Video Broadcasting: Technology, Standards, and Regulations*, Artech House Publishers (2003).