# Lossy Joint Source-Channel Coding Using Raptor Codes

Authors: O. Y. Bursalioglu; M. Fresia; G. Caire; H. V. Poor
Journal: International Journal of Digital Multimedia Broadcasting
 (2008)

---

## Abstract

The straightforward application of Shannon's separation principle may entail a significant
suboptimality in practical systems with limited coding delay and complexity.
This is particularly evident when the lossy source code is based on entropy-coded quantization.
In fact, it is well known that entropy coding is not robust to residual channel errors.
In this paper, a joint source-channel coding scheme is advocated that combines the advantages and simplicity of
entropy-coded quantization with the robustness of linear codes. The idea is to combine entropy coding and channel coding into
a single linear encoding stage. If the channel is symmetric, the scheme can asymptotically achieve the optimal
rate-distortion limit. However, its advantages are more clearly evident under finite coding delay and complexity.
The sequence of quantization indices is decomposed into bitplanes, and each bitplane is independently mapped onto a sequence of channel coded symbols. The coding rate of each bitplane is chosen according to the bitplane conditional entropy rate.
The use of systematic raptor encoders is proposed, in order to obtain a continuum of coding rates with a single basic encoding
algorithm.
Simulations show that the proposed scheme can outperform the separated baseline scheme
for finite coding length and comparable complexity and, as expected, it is much more robust to channel errors in the case of channel capacity mismatch.


---

## Body

## 1. Introduction

A stationary ergodic source can be transmitted over an
information-stable channel with end-to-end average distortionD with bandwidth expansion factorb not lower than R(D)/C channel symbols per source sample, where R(D) is the source rate distortion function and C is the channel capacity. Shannon's
source-channel separation principle [1] ensures that this
optimal performance can be approached by independently designing the
source coding and the
channel coding schemes.
The bandwidth expansion factor b is defined as the number of channel symbols
per source symbol. If a block of K source symbols is transmitted through the
channel in N channel uses, then b=N/K.
This provides a definite architectural advantage in practical systems, where
typically (lossy) source coding is implemented at the application layer, while
channel coding is designed and optimized for the physical layer.On the other hand, this separated source-channel
coding (SSCC) approach may incur substantial suboptimality, due to the nonideal
behavior of finite length, finite complexity, source and channel codes. In
fact, source codes designed without taking into account the presence of channel
decoding errors are typically very fragile and this might impose unnecessarily
restrictive constraints on the performance of channel coding. In such cases,
joint source-channel coding (JSCC) may lead to performance improvement (i.e., a
better(b,D) operating point) for the same level of
complexity.Most practical lossy source coding schemes for natural
sources (e.g., images, audio, video) are based on the idea oftransform coding [2]. Source blocks are projected
onto a suitable basis by a linear transformation, such that the source is well
described by only a small number of significant transform coefficients. Then,
the coefficients are scalar-quantized, and finally the resulting sequence of
quantization indices are entropy coded. The theoretical foundation of this
approach relies on the universality of entropy-coded quantization, and dates
back to the work of Ziv [3]. In general, the linear transform is adapted to the
given class of sources (e.g., wavelet transforms for images [4]). The statistics of the
quantization indices is not known a priori. However, the memory structure of
the underlying discrete source is fixed and it is typically described as a
finite-memory tree-source (e.g., the context structure of JPEG2000 [5, 6]). Then, data compression is obtained by using an
adaptive entropy coding scheme that estimates the transition probabilities of
the source statistical model. For example, arithmetic coding [7] with Krichevsky-Trofimov
(KT) sequential probability estimation is a common choice [8].For the sake of simplicity, this paper treats only
independent and identically distributed (i.i.d.) sources with known statistics,
that is, it neither deals with the transform coding aspect, nor with the
universal implementation of entropy coding. However, our results can be
generalized along the lines of what done in [5, 6]. Even in the nonuniversal case, classical lossless
compression is catastrophic: a
small Hamming distortion (number of bits in error) in the entropy-coded
sequence is mapped into a large distortion in the reconstructed source
sequence. This imposes a very strict target error probability on the channel
coding stage, thus involving both complex channel coding and operating points
that may be quite far from the theoretical limits. This is even more evident in
applications where the coding delay is limited, thus preventing the use of very
large block lengths.It was shown in [9] that fixed-to-fixed length data compression of a
discrete source with linear codes is asymptotically optimal, in the sense that
compression up to the source entropy rate can be achieved. This is strongly
related to transmission using the same linear code on a discrete additive noise
channel where the noise has the same statistics as the discrete source. This
analogy can be exploited in order to design a JSCC scheme. We wish to maintain
the simplicity of the transform coding approach while improving the robustness
of the scheme. The rationale behind the proposed design is the following: since
linear codes achieve the entropy rate of discrete sources and the capacity of
symmetric channels, we can combine the entropy coding stage and the coding
stage into a single linear encoding stage. The advantage of this approach is
that the design of noncatastrophic linear encoders is very well understood.
Therefore, the proposed scheme can approach the optimal separation limit for
large block length, while achieving better robustness to channel errors at
finite decoding delay and complexity.In [5], this JSCC approach was applied to the transmission of JPEG2000-like encoded images (in the sense that the
wavelet transform, the quantization scheme and the tree-source memory structure
were borrowed from JPEG2000), by using a family of progressively punctured
turbo codes to map directly the redundant quantization bits into channel
symbols. As stated above, here we focus on simpler i.i.d. sources with
perfectly known statistics (i.e., the nonuniversal case) and investigate in
greater detail the performance analysis and the comparison with the baseline
SSCC approach. In this work, we use raptor codes [10] in order to map the
redundant quantization bits into channel-coded symbols.Our scheme works as follows. A source block of lengthK is quantized symbol by symbol. The sequence of
quantization indices, represented as binary vectors, are partitioned into
bitplanes. The bitplanes are separately encoded into channel symbols by a bank
of binary raptor encoders. Each bitplane is encoded at a rate that depends on its
conditional entropy rate given the bitplanes previously encoded. At the
decoder, the bitplanes are decoded in sequence using a multistage decoder,
where in each stage we use a belief propagation (BP) iterative decoder that
takes into account both the already decoded bits from previous planes, and the
a priori statistics of the current bitplane as well as the received channel
output.Raptor codes are a particularly useful class of
rateless codes. The advantage of using a rateless code is clear: with a single
basic encoding machine we can generate a continuum of coding rates. Therefore,
the scheme can adapt naturally to the entropy rate of the source and to the
capacity of the channel. Although we do not pursue the universal setting in
this work, we notice here that the proposed architecture allows a very fine
rate matching between the (unknown a priori) source entropy and the channel
capacity without resorting to a library of progressively punctured codes as is
done in [5].We express the performance of a source-channel coding
scheme in terms of its peak signal-to-noise ratio (PSNR), expressed in dB,
defined as(1)PSNR≜−10log10⁡(D). In particular, we will focus on
a standard Gaussian i.i.d. source S∼𝒩(0,1) and on the mean-squared distortion D=𝔼[|S−S^|2].
In this case, the distortion-rate function is D=2−2R.
At the Shannon separation limit, that is, letting R=bC,
we have (2)PSNRShannon=20log⁡10(2)bC=(6dB)×bC.Our aim is to design a family of practical schemes
that operate close to the curvePSNRShannon versus b.
Notice that we do not pursue here the design of embedded schemes, that is, of
single coding schemes that achieve multiple (PSNR,b) points. Nevertheless, the bitplane layered
structure of the proposed encoder and the proposed multistage decoder lend
themselves quite naturally to the design of embedded JSCC schemes. We leave
this aspect for future work and comment on it further in the concluding
section.The rest of this paper is organized as follows. In
Section2, we review the limits of scalar entropy-coded quantization and define
the target “operational Shannon limit” of our scheme. In Section 3 we present
a comprehensive analysis of the baseline SSCC scheme which represents our term of comparison.Section 4 presents the details of the proposed
scheme, its analysis and an algorithm for progressive incremental redundancy in
order to optimize the coding rates at each bitplane. Section 5 presents some
additional numerical comparisons between the baseline SSCC and the JSCC
schemes, and in Section 5 we present some concluding remarks. Raptor codes, BP
decoding, EXIT chart analysis and some ancillary results are presented in the
appendices.


## 2. Entropy-Coded Scalar Quantization

A source sequence of lengthK,s∈ℝK,
is quantized by applying componentwise the scalar quantizer 𝒬B:ℝ→𝔽2B+1,
where B bits are used to represent the magnitude and
one bit represents the sign. Let u=𝒬B(s) denote the sequence of quantization indices
and let up,k:p=0,…,B denote the bits forming the kth index. The sequence u can be thought as a (B+1)×K binary array, where each row is called a
“bitplane." Without loss of generality, we associate the 0th row with the sign bit and the rows from 1
to B with the magnitude bits, with the convention that the first bitplane is
the least significant and the Bth bitplane is the most significant.As anticipated in the Introduction, we fix the
quantizer and compare the performance of an SSCC approach based on the
concatenation of a conventional entropy coding stage with a conventional
channel code, with the performance of a JSCC that merges the two operations
into a single linear encoding map. Therefore, in the absence of channel
residual errors, both schemes achieve the same minimum distortion due to the
quantizer, denoted byD𝒬(B).
Letting HB(U) denote the entropy rate of u,
measured in bit per quantization index, we have that the
point (3)(b=HB(U)C,PSNR=−10log⁡10(D𝒬(B))) is the best achievable point for
any scheme based on the fixed quantizer 𝒬B.
Following [2], we
refer to this point as the “operational Shannon limit” for schemes with fixed
quantizers.We consider uniform scalar quantizers where the
interval size is chosen in order to minimize the mean-squared distortion of the
Gaussian unit variance i.i.d. source, for a fixed number2B+1 of intervals. In [3], Ziv showed that a coding
scheme formed by a scalar uniform quantizer followed by entropy coding yields a
rate penalty of no more than 0.754 bits per sample with respect to the R(D) limit. Thus, constraining the quantizer to be
a uniform scalar quantizer should cost no more then a 0.754/C channel symbols per source symbols.In Figure2 we compare the PSNR versus b curves for the Shannon limit, the Ziv bound
and the operational Shannon limit for a family of optimized uniform scalar
quantizers with B=1,2,3,4,5, and 6,
and for channel capacity C=0.5.
All results in this paper make use of this family of quantizers.


## 3. Analysis of the Baseline SSCC Scheme

In this section, we study the performance of nonideal
SSCC. First, we consider the performance degradation due to nonideal source and
channel codes that operate at source coding rateRs=R(D)+δs and channel coding rate Rc=C−δc,
respectively, where δs and δc are positive rate gaps. This analysis assumes
no errors at the output of the channel decoder.Then, we introduce the channel decoding error
probability, and obtain a distortion upper bound as a function ofδc and δs,
closely following the analysis of [11]. This analysis is based on the random coding exponent
for channel codes, and essentially validates the error-free rate-gap analysis
even for moderately large block length K.Finally, we consider a very practical scheme, based on
the concatenation of arithmetic entropy coding and a conventional binary raptor
code. We provide a very accurate semi-analytic approximation for the achievable
PSNR of this scheme and show that the achieved results follow closely the error
free rate-gap analysis by matching the parametersδc and δs.
We also notice that for finite block length K the practical scheme suffers from an
additional performance degradation, especially visible at high resolution
(large PSNR). We quantify this additional performance degradation by looking at
the finite-length versus infinite-length error performance of raptor codes.


### 3.1. Rate-Gap Analysis

Consider a separated scheme that makes use of channel
coding at rateRc=C−δc and source coding at rate Rs=R(D)+δs,
where δc,δs>0 are rate gaps, and where the residual
bit-error rate (BER) at the output of the channel decoder is (essentially)
zero. Using D=2−2(Rs−δs) and b=Rs/Rc we obtain (4)PSNR=(6dB)×(b(C−δc)−δs). We notice that the slope of the
straight line characterizing PSNR versus b decreases with the channel coding gap δc,
while the source coding rate gap involves only a horizontal shift. As a result,
an SSCC whose channel coding stage achieves negligible BER works further and
further away from the Shannon limit as PSNR increases (high resolution).


### 3.2. SSCC with Codes Achieving Positive Error Exponent

In order to take into account channel decoding errors,
we modify slightly the approach of [11] and obtain the achievable PSNR lower bound (we omit
the details since the derivation follows trivially from [11]): (5)PSNR≥(6dB)×(b(C−δc)−δs)−10log⁡10(1+2−KbEr(C−δc)+1+2b(C−δc)−2δs), where Er(Rc) denotes the random coding error exponent for a
given coding ensemble over the considered transmission channel. Notice that Er(Rc)>0 for all Rc<C,
and therefore, the error exponent is positive for all rate gaps δc>0.
For values of K,δc,δs,b,C such that 2−KbEr(C−δc)+1+2b(C−δc)−2δs≪1,
(5) essentially coincides with (4).Figure3 compares (4) and (5) for different values of δc and for block length K=10000 (which is the finite source block length that
we will use throughout this paper), and δs=0.3821.
This value of δs is chosen in order to match the rate gap
attained by the quantizers (see Figure 5). In these results, we considered a
binary symmetric channel (BSC) with capacity C=0.5 (cross-over probability ϵ=0.11). The exponent Er(Rc) for the BSC can be found, for example, in
[12]. For the
parameters of Figure 3 we notice that (4) and (5) do not coincide for too small δc (e.g., δc=0.01 in the figure) while they coincide for large
enough δc (in this case, δc≥δc*=0.016). For finite but large block lengths, as in
this case, the threshold δc* is given by the minimum value of the channel
coding rate gap above which the exponent T(C,δc,δs,K,b)≜−KbEr(C−δc)+1+2b(C−δc)−2δs becomes negative. In Figure 4, we plot T(C,δc,δs,K,b) and Er(C−δc) versus δc,
when the parameters of Figure 3 are considered. It
has been observed that for different values of b given in the range of Figure 3, δc is constant. Thus in Figure 4, we use b=7.2036.


### 3.3. SSCC with Arithmetic Coding and Raptor Codes

We provide an accurate approximated analysis of the
performance of a practical SSCC scheme that can be regarded as our baseline
scheme, since its encoding and decoding complexity is very similar to that of
the JSCC scheme examined in the next section. With reference to the block
diagram of Figure1, we consider the concatenation of an optimized uniform
scalar quantizer with B+1 quantization bits with an arithmetic encoder.
The resulting entropy-coded bits are then channel encoded using a raptor code
of suitable rate.Figure 1
Conceptual block diagram of the
conventional SSCC and the proposed JSCC schemes. The two schemes coincide but
for the fact that the concatenation of entropy coding and channel coding (SSCC)
is replaced by a single linear encoding block (JSCC).Figure 2
PSNR versusb for the Shannon Limit, the Ziv bound and the
operational Shannon limit for the considered family of scalar quantizers for B=1,…,6 and for channel capacity C=0.5.Figure 3
The SSCC rate-gap approximation
(4) is compared with the PSNR upperbound (5) for C=0.5 and δs=0.3821.T(C,δc,δs,K,b) and Er(Rc) versus δc for C=0.5, K=10000, δs=0.03821, and b=7.2036.
(a)(b)Figure 5
Performance of
the concatenation of arithmetic coding and raptor code for infinite channel
coding length, source lengthK=10000,
and a BSC with C=0.5. δs=0.3821, and δc=0 are the values of rate-gaps for the
operational Shannon limit (4). For the base line SSCC, these rate-gap values are
empirically found to be δs=0.3821 and δc=0.035.A sufficiently large interleaver is placed between the
entropy coding and the channel coding stages, such that the bit decoding errors
at the input of the arithmetic decoder can be considered to be i.i.d.. Since
the arithmetic encoder has perfect knowledge of the probability distribution of
the discrete sourceu at the quantizer output, it can approach very
closely the source entropy rate HB(U) even for moderate source block length K.We approximate the performance of such a scheme by
assuming that the arithmetic decoder produces random data after the first bit
error at its input. LetM=KHB(U) denote the number of entropy-coded bits
produced by the arithmetic encoder. These bits are channel encoded and decoded.
Let m denote the position of the last correctly
decoded bit before the first bit error at the arithmetic decoder input. Under
the assumption of i.i.d. bit errors, m is a truncated geometric random variable with
probability mass function (6)P(m=i)=(1−Pb)iPb for i=0,1,…,M−1,
and P(m=M)=1−∑i=0M−1(1−Pb)iPb,
where Pb denotes the BER at the output of the channel
decoder. We approximate the number of correctly decoded quantization indices by m/HB(U) (neglecting integer effects). After the first
bit error, the arithmetic decoder produces random symbols distributed as the
quantization indices (i.e., according to the given discrete-source probability
distribution) but essentially statistically independent of the source sequence.
Therefore, the average distortion in this case is given by(7)σ2=𝔼[(S−S˜)2]=𝔼[S2]+𝔼[S˜2]=1+σ𝒬2, where S˜ denotes a random variable distributed as the
quantizer reconstruction points, and σ𝒬2 denotes its variance. On the other hand, before
the first bit error, the system reconstructs the correct quantization points S^ correctly; therefore the average distortion in
this case coincides with the quantization distortion D𝒬(B).Eventually, the total average distortion of the system
is approximated by(8)D≈P(m=M)D𝒬(B)+∑i=0M−1(1−Pb)iPb[D𝒬(B)i/HB(U)K+σ2K−i/HB(U)K]. The approximate analysis
requires the evaluation of the residual BER at the channel decoder output. This
can be obtained by simulation of the stand-alone raptor code with given finite
length, or by using any suitable approximation or semi-analytic technique, such
as Density Evolution or EXIT chart methods [13–17]. In particular, we make use of the EXIT chart
approximation, reviewed in Appendix B.In Figure5 we report the PSNR versus b (obtained by using (8)) for different values
of B when K=10000,
a BSC with capacity C=0.5 and where the raptor code output BER is
approximated via the EXIT chart method. These results assume implicitly
infinite channel coding block length. In order to validate the approximated
distortion analysis of (8), we run simulations of the arithmetic decoder and
quantization reconstruction, fed with the quantization bits corrupted by
independent bit errors at a rate equal to the raptor code output BER. As seen
in Figure 5, the results of the simulated arithmetic decoder match remarkably
well the approximation (8), thus showing that the arithmetic decoder indeed
produces approximately random data after the first bit error.The results in Figure5 show that for the case of very
large channel coding block length the performance of the baseline SSCC scheme
is remarkably close to the operational Shannon limit and therefore the scheme
is hard to beat by any scheme using the same set of quantizers. However, the
picture changes when we consider a finite channel coding block length. In
particular, we consider independent encoding of each source block, so that the
system latency is dictated by the source block length K and not by the channel coding block length.
This corresponds to choosing the raptor code input bits block length equal to M=KHB(U).
For the system parameters as before, the PSNR results in this case are shown in
Figure 6. We notice that the finite channel coding block length yields an
additional degradation that increases with PSNR.Figure 6
Performance of the
concatenation of arithmetic coding and a raptor code for finite channel coding
length, source lengthK=10000,
and a BSC with C=0.5.We can explain and quantify the increasing bandwidth
expansion gapΔb(B) shown in Figure 6 as follows. Let Rinf and Rfin denote the channel coding rates needed by the
raptor code to reach a small BER such that the effective distortion is
virtually identical to the quantization distortion. For example, Figure 7 plots
the PSNR corresponding to the distortion (8) as a function of Pb.
We notice that for Pb=10−7 the quantization distortion (corresponding to
the maximum achievable PSNR) is essentially reached. Then, Figure 8 plots the
raptor code BER for the BSC with capacity C=0.5,
as a function of the reciprocal of the channel coding rate 1/Rc.
Notice that the raptor code is a rateless code, and therefore we can generate
as many coded symbols as we like. In order to generate Figure 8 we keep the
channel parameter ϵ=0.11 fixed (corresponding to C=0.5) and run encoding and decoding for smaller
and smaller coding rates. The infinite block length case is obtained by using
the EXIT chart approximation.Figure 7
Output PSNR as a function of the
channel decoding residual BERPb.Figure 8
Raptor code output BER for the
infinite block length case (EXIT approximation) and for the finite length case,
obtained by simulation, as a function of the reciprocal of the coding rate forC=0.5.
The Finite length is taken to be 27606=KH2(U) which is the approximate number of bits at the
output of arithmetic encoder, since the Gaussian block has entropy rate 2.7606 when quantized with 𝒬2.Figure8 shows that the target BER of 10−7 is reached at certain rates Rinf and Rfin for the cases of infinite and finite block
length, respectively, and allows us to find the difference 1/Rinf−1/Rfin,
shown in the figure.Finally, we can quantify the bandwidth expansion gaps
shown in Figure6 by noticing that, since b=HB(U)/Rc,
we have (9)Δb(B)=HB(U)(1Rinf−1Rfin). It is clear that the gap Δb(B) is increasing with the quantizer resolution B,
and therefore with PSNR. This is a further confirmation of the fact that, in
practice, it becomes more and more difficult to approach the Shannon limit as
the resolution increases.


## 3.1. Rate-Gap Analysis

Consider a separated scheme that makes use of channel
coding at rateRc=C−δc and source coding at rate Rs=R(D)+δs,
where δc,δs>0 are rate gaps, and where the residual
bit-error rate (BER) at the output of the channel decoder is (essentially)
zero. Using D=2−2(Rs−δs) and b=Rs/Rc we obtain (4)PSNR=(6dB)×(b(C−δc)−δs). We notice that the slope of the
straight line characterizing PSNR versus b decreases with the channel coding gap δc,
while the source coding rate gap involves only a horizontal shift. As a result,
an SSCC whose channel coding stage achieves negligible BER works further and
further away from the Shannon limit as PSNR increases (high resolution).


## 3.2. SSCC with Codes Achieving Positive Error Exponent

In order to take into account channel decoding errors,
we modify slightly the approach of [11] and obtain the achievable PSNR lower bound (we omit
the details since the derivation follows trivially from [11]): (5)PSNR≥(6dB)×(b(C−δc)−δs)−10log⁡10(1+2−KbEr(C−δc)+1+2b(C−δc)−2δs), where Er(Rc) denotes the random coding error exponent for a
given coding ensemble over the considered transmission channel. Notice that Er(Rc)>0 for all Rc<C,
and therefore, the error exponent is positive for all rate gaps δc>0.
For values of K,δc,δs,b,C such that 2−KbEr(C−δc)+1+2b(C−δc)−2δs≪1,
(5) essentially coincides with (4).Figure3 compares (4) and (5) for different values of δc and for block length K=10000 (which is the finite source block length that
we will use throughout this paper), and δs=0.3821.
This value of δs is chosen in order to match the rate gap
attained by the quantizers (see Figure 5). In these results, we considered a
binary symmetric channel (BSC) with capacity C=0.5 (cross-over probability ϵ=0.11). The exponent Er(Rc) for the BSC can be found, for example, in
[12]. For the
parameters of Figure 3 we notice that (4) and (5) do not coincide for too small δc (e.g., δc=0.01 in the figure) while they coincide for large
enough δc (in this case, δc≥δc*=0.016). For finite but large block lengths, as in
this case, the threshold δc* is given by the minimum value of the channel
coding rate gap above which the exponent T(C,δc,δs,K,b)≜−KbEr(C−δc)+1+2b(C−δc)−2δs becomes negative. In Figure 4, we plot T(C,δc,δs,K,b) and Er(C−δc) versus δc,
when the parameters of Figure 3 are considered. It
has been observed that for different values of b given in the range of Figure 3, δc is constant. Thus in Figure 4, we use b=7.2036.


## 3.3. SSCC with Arithmetic Coding and Raptor Codes

We provide an accurate approximated analysis of the
performance of a practical SSCC scheme that can be regarded as our baseline
scheme, since its encoding and decoding complexity is very similar to that of
the JSCC scheme examined in the next section. With reference to the block
diagram of Figure1, we consider the concatenation of an optimized uniform
scalar quantizer with B+1 quantization bits with an arithmetic encoder.
The resulting entropy-coded bits are then channel encoded using a raptor code
of suitable rate.Figure 1
Conceptual block diagram of the
conventional SSCC and the proposed JSCC schemes. The two schemes coincide but
for the fact that the concatenation of entropy coding and channel coding (SSCC)
is replaced by a single linear encoding block (JSCC).Figure 2
PSNR versusb for the Shannon Limit, the Ziv bound and the
operational Shannon limit for the considered family of scalar quantizers for B=1,…,6 and for channel capacity C=0.5.Figure 3
The SSCC rate-gap approximation
(4) is compared with the PSNR upperbound (5) for C=0.5 and δs=0.3821.T(C,δc,δs,K,b) and Er(Rc) versus δc for C=0.5, K=10000, δs=0.03821, and b=7.2036.
(a)(b)Figure 5
Performance of
the concatenation of arithmetic coding and raptor code for infinite channel
coding length, source lengthK=10000,
and a BSC with C=0.5. δs=0.3821, and δc=0 are the values of rate-gaps for the
operational Shannon limit (4). For the base line SSCC, these rate-gap values are
empirically found to be δs=0.3821 and δc=0.035.A sufficiently large interleaver is placed between the
entropy coding and the channel coding stages, such that the bit decoding errors
at the input of the arithmetic decoder can be considered to be i.i.d.. Since
the arithmetic encoder has perfect knowledge of the probability distribution of
the discrete sourceu at the quantizer output, it can approach very
closely the source entropy rate HB(U) even for moderate source block length K.We approximate the performance of such a scheme by
assuming that the arithmetic decoder produces random data after the first bit
error at its input. LetM=KHB(U) denote the number of entropy-coded bits
produced by the arithmetic encoder. These bits are channel encoded and decoded.
Let m denote the position of the last correctly
decoded bit before the first bit error at the arithmetic decoder input. Under
the assumption of i.i.d. bit errors, m is a truncated geometric random variable with
probability mass function (6)P(m=i)=(1−Pb)iPb for i=0,1,…,M−1,
and P(m=M)=1−∑i=0M−1(1−Pb)iPb,
where Pb denotes the BER at the output of the channel
decoder. We approximate the number of correctly decoded quantization indices by m/HB(U) (neglecting integer effects). After the first
bit error, the arithmetic decoder produces random symbols distributed as the
quantization indices (i.e., according to the given discrete-source probability
distribution) but essentially statistically independent of the source sequence.
Therefore, the average distortion in this case is given by(7)σ2=𝔼[(S−S˜)2]=𝔼[S2]+𝔼[S˜2]=1+σ𝒬2, where S˜ denotes a random variable distributed as the
quantizer reconstruction points, and σ𝒬2 denotes its variance. On the other hand, before
the first bit error, the system reconstructs the correct quantization points S^ correctly; therefore the average distortion in
this case coincides with the quantization distortion D𝒬(B).Eventually, the total average distortion of the system
is approximated by(8)D≈P(m=M)D𝒬(B)+∑i=0M−1(1−Pb)iPb[D𝒬(B)i/HB(U)K+σ2K−i/HB(U)K]. The approximate analysis
requires the evaluation of the residual BER at the channel decoder output. This
can be obtained by simulation of the stand-alone raptor code with given finite
length, or by using any suitable approximation or semi-analytic technique, such
as Density Evolution or EXIT chart methods [13–17]. In particular, we make use of the EXIT chart
approximation, reviewed in Appendix B.In Figure5 we report the PSNR versus b (obtained by using (8)) for different values
of B when K=10000,
a BSC with capacity C=0.5 and where the raptor code output BER is
approximated via the EXIT chart method. These results assume implicitly
infinite channel coding block length. In order to validate the approximated
distortion analysis of (8), we run simulations of the arithmetic decoder and
quantization reconstruction, fed with the quantization bits corrupted by
independent bit errors at a rate equal to the raptor code output BER. As seen
in Figure 5, the results of the simulated arithmetic decoder match remarkably
well the approximation (8), thus showing that the arithmetic decoder indeed
produces approximately random data after the first bit error.The results in Figure5 show that for the case of very
large channel coding block length the performance of the baseline SSCC scheme
is remarkably close to the operational Shannon limit and therefore the scheme
is hard to beat by any scheme using the same set of quantizers. However, the
picture changes when we consider a finite channel coding block length. In
particular, we consider independent encoding of each source block, so that the
system latency is dictated by the source block length K and not by the channel coding block length.
This corresponds to choosing the raptor code input bits block length equal to M=KHB(U).
For the system parameters as before, the PSNR results in this case are shown in
Figure 6. We notice that the finite channel coding block length yields an
additional degradation that increases with PSNR.Figure 6
Performance of the
concatenation of arithmetic coding and a raptor code for finite channel coding
length, source lengthK=10000,
and a BSC with C=0.5.We can explain and quantify the increasing bandwidth
expansion gapΔb(B) shown in Figure 6 as follows. Let Rinf and Rfin denote the channel coding rates needed by the
raptor code to reach a small BER such that the effective distortion is
virtually identical to the quantization distortion. For example, Figure 7 plots
the PSNR corresponding to the distortion (8) as a function of Pb.
We notice that for Pb=10−7 the quantization distortion (corresponding to
the maximum achievable PSNR) is essentially reached. Then, Figure 8 plots the
raptor code BER for the BSC with capacity C=0.5,
as a function of the reciprocal of the channel coding rate 1/Rc.
Notice that the raptor code is a rateless code, and therefore we can generate
as many coded symbols as we like. In order to generate Figure 8 we keep the
channel parameter ϵ=0.11 fixed (corresponding to C=0.5) and run encoding and decoding for smaller
and smaller coding rates. The infinite block length case is obtained by using
the EXIT chart approximation.Figure 7
Output PSNR as a function of the
channel decoding residual BERPb.Figure 8
Raptor code output BER for the
infinite block length case (EXIT approximation) and for the finite length case,
obtained by simulation, as a function of the reciprocal of the coding rate forC=0.5.
The Finite length is taken to be 27606=KH2(U) which is the approximate number of bits at the
output of arithmetic encoder, since the Gaussian block has entropy rate 2.7606 when quantized with 𝒬2.Figure8 shows that the target BER of 10−7 is reached at certain rates Rinf and Rfin for the cases of infinite and finite block
length, respectively, and allows us to find the difference 1/Rinf−1/Rfin,
shown in the figure.Finally, we can quantify the bandwidth expansion gaps
shown in Figure6 by noticing that, since b=HB(U)/Rc,
we have (9)Δb(B)=HB(U)(1Rinf−1Rfin). It is clear that the gap Δb(B) is increasing with the quantizer resolution B,
and therefore with PSNR. This is a further confirmation of the fact that, in
practice, it becomes more and more difficult to approach the Shannon limit as
the resolution increases.


## 4. Joint Source-Channel Coding Scheme

In this section we describe the encoder and decoder of
the proposed JSCC scheme. Then, we discuss an incremental redundancy rate
allocation procedure that allows the optimization of the scheme. We hasten to
say that this rate allocation procedure is run off-line, and serves to design
the coding scheme for given source and channel statistics. More generally, an
adaptive scheme that allocates coded bits to the bitplanes on the fly,
depending on the empirical entropy rate of the source and on the capacity of
the channel may be envisaged in a universal JSCC setting, where the source
statistics are not known a priori and are learned instead from the source
sequence itself. However, we do not pursue this approach here.Figure9 shows the encoder block diagram. Each
bitplane (row of the binary array u of quantization indices produced by the
quantizer), is mapped into a sequence of coded symbols. Here we consider binary
coding, and a BSC. Letting up denote the pth row of u,
the corresponding block of coded symbols is given by xp=upGp, where Gp is a suitable encoding matrix of size K×Np.
Then, the encoded blocks x0,…,xB are transmitted in sequence over the BSC. The
resulting bandwidth expansion factor is (10)b=∑p=0BNpK. Given the source symmetry, it is
clear that the sign bit is equiprobable and has entropy H(U0)=1.
Furthermore, it is independent of the magnitude bits. Hence, the target nominal
rate for the encoder of the sign bit is K/N0=C.
As for the pth magnitude bit, we allocate a nominal target
rate equal to K/Np=C/H(Up∣Up+1,…,UB),
where H(Up∣Up+1,…,UB) denotes the conditional entropy rate of the pth bitplane, conditioned on the bitplanes p+1,p+2,…,B.
It follows that the nominal bandwidth expansion is given by(11)b=K/C+∑p=1BKH(Up∣Up+1,…,UB)/CK=1+∑p=1BH(Up∣Up+1,…,UB)C=HB(U)C, which is optimal.Figure 9
Diagram of the proposed JSCC encoder.In order to be able to decode at these rates, we
consider a multistage decoder as shown in Figure10, that considers the
bitplanes in sequence. The sign bit is independently decoded. The magnitude
bits are decoded in sequence, starting from the Bth plane. At each decoding stage p,
the hard decisions of the already decoded planes are used by the BP decoder to
compute the conditional a priori probabilities of the pth bitplane, as explained in Appendix A.
Assuming that at each level p,
the previous levels are correctly decoded, then the rates K/Np=C/H(Up∣Up+1,…,UB) are achievable.Figure 10
Multistage decoder for successive bitplane decoding
and source reconstruction.In practice, due to the fact that the raptor codes do
not achieve sufficiently low BER if their rate is too close to the nominal rate
limit, we must allocate the rates allowing for some gap. The rate allocation
problem is made more complicated by the fact that in the multistage decoder the
decoding of the different planes is not independent. In particular, if thepth plane fails with many bits in errors, then
it is very likely that all the planes p−1,p−2,…,1 will also fail, since their decoders are fed
with incorrect a priori conditional probabilities. We will address the problem
of rate allocation for the multistage decoder at the end of this section.Next, let us examine in more detail how encoding of
thepth plane is implemented with raptor codes
[10]. Raptor codes can
substantially be viewed as an extension of Luby Transform codes (LT codes)
[18], since they are
based on the concatenation of an outer linear code (in our case we consider
low-density-parity check (LDPC) codes) with an inner LT code (see Appendix A
for details). We use raptor codes in systematic form. In particular, let Sp be a K×K full-rank binary matrix given by Sp=GldpcAp, where Ap is a submatrix of the LT code generator matrix
at encoding level p and Gldpc is the generator matrix of the LDPC code (see
[10] for details). The
encoder produces a vector of K intermediate symbols, denoted by up′=upSp−1.
Then, the intermediate symbols are expanded by high-rate LDPC encoding, into up′′=up′Gldpc.
Finally, the encoded symbols x are obtained from u′′,
by applying nonsystematic rateless encoding, that is, the symbols x1,x2,…,xNp are produced in sequence, and each xi is given as the sum of elements of u′′ selected at random according to the LT degree
distribution Ω.Notice thatup=up′Sp.
Therefore, in the Tanner graph representing the code [19, 20] the nodes corresponding to
source symbols bp have a degree distribution identical to that
of a standard nonsystematic raptor code. Furthermore, although Sp is sparse, its inverse is sufficiently dense
such that the symbols up′ are close to being uniform and random i.i.d.
Notice that this is essential to the scheme, since in order to drive the
channel with the correct input distribution we need to send the nonsystematic
symbols xp through the channel, and their distribution
should be as close as possible to i.i.d. and equiprobable.A key component in the systematic raptor code design
consists of finding a suitable nonsingularK×K matrix Sp,
with given column weight distribution, and such that its inverse looks as much
as possible like a random binary matrix. As for the LDPC code (often referred
to as the “precode” in the raptor coding literature), we used a regular code
with parameters (2,100).Let us focus now on decoding and source
reconstruction. The multistage decoder of Figure10 is based on BP at each
stage p in order to approximately compute the
symbol-by-symbol posterior marginal Log-Likelihood Ratios (LLRs) {λp,k:k=1,…,K} defined as (12)λp,k=log⁡P(up,k=0∣yp,up+1,…,uB)P(up,k=1∣yp,up+1,…,uB), where yp denotes the channel output corresponding to
the input xp,
and the conditioning is with respect to the already decoded bitplanes. This is
obtained by feeding the hard decisions from the planes p+1,…,B to the BP decoder at level p.
An iterative version of the multistage decoder where soft messages in the form
of a posteriori LLRs are exchanged instead of hard decisions was also
considered, but it was observed that this does not provide any significant
improvement and therefore was not pursued further, given its much greater
complexity.The information about the already decoded bitplanes is
incorporated into the BP decoder for bitplanep in the following way. As explained in Appendix
A, the BP algorithm is initialized with input messages at all the source and
coded nodes in the Tanner graph of the code. The coded nodes (corresponding to
the coded symbols xp), receive their input message from the
corresponding channel observation. In the case of a BSC, this is given
by (13)μp,i=(−1)yp,ilog⁡1−ϵϵ,i=1,…,Np. The source nodes (corresponding
to the source bits up), are associated with the input
messages (14)νp,k=log⁡P(up,k=0∣u^p+1,k,…,u^B,k)P(up,k=1∣u^p+1,k,…,u^B,k), where u^p+1,k,…,u^B,k are the hard decisions obtained from the
previous stages.The BP decoder at each stage runs for a given desired
number of iterations, and eventually outputs both hard decisions to be passed
to the next stage and soft outputs in the form of the posterior LLRs given by
(12). Once all bitplanes have been decoded, the source is reconstructed as
follows. Consider without loss of generality the inverse quantization
mapping (15)𝒬B−1(uk)=(−1)u0,kΔB2([∑p=1Bup,k2p]+1) that yields the mid-point of
each quantization interval given the set of quantization bits.Then, we can either consider hard reconstruction,
which consists of using the hard decisionsu^p,k in (15), or soft reconstruction, which makes
use of the (approximate) posterior LLRs in order to compute the
minimum-mean-square-error (MMSE) estimator of the source samples given the
channel output, that is, the conditional mean estimator s^k=𝔼[sk|y].
Treating the decoder estimated posterior LLRs as if they were the true
posterior LLRs, we obtain (16)s^k=ΔB2tanh⁡(λ0,k2)([∑p=1B2p1+eλp,k]+1).In AppendixA, we prove an interesting isomorphism
between the BP decoder of the joint source-channel problem as described above
and a related standard channel coding problem. Let us focus on a single binary
independent source sequence u of length K,
with probabilities pk≜P(uk=1) for k=1,…,K.
This is encoded into a binary codeword x=uG,
of length N,
where G is a K×N raptor encoder as previously described. Let us
transmit x through a BSC with cross-over probability ϵ,
and let y=x⊕e denote the corresponding output. The result holds for any binary input symmetric
output channel, but here we focus on the BSC for simplicity of
exposition. Then, the BP decoder for this problem
is isomorphic to a decoder for the following related channel coding problem:
consider transmission of the all-zero codeword from the systematic code with generator
matrix [I|G],
of size K×(K+N) over a channel that for the first K components operates as (17)yk=xk⊕uk,k=1,…,K, where uk is the kth source symbol, and for the remaining N components operates as (18)yK+n=xK+n⊕en,n=1,…,N. In order words, there exists a
one-to-one mapping of the messages of the BP decoder for the first problem
(joint source channel) and the messages of the BP decoder for the second
problem (channel only), for every edge of the decoder graph and every decoder
iteration.This means that the source-channel BP decoding can be
analyzed, for example, by using the EXIT chart method, by considering the
associated “virtual” channel, where the all-zero codeword from the associated
systematic code is transmitted partly on a binary additive noise channel with
noise realization identical to the source realization of the source-channel
problem, and partly on thesame BSC (with the same noise realization) of the source-channel problem. We use
this BP isomorphism result in order to derive a simple EXIT chart analysis of
the BP decoder at each stage of the multistage decoder, under the assumption
that the hard decisions from previous stages are correct.


### 4.1. Rate Allocation Algorithm

The rate allocation of each bitplane encoder is
established offline by using the greedy algorithm described below. Again, we
notice that we do not consider adaptive rate allocation: given the source and
channel statistics, we run the greedy allocation algorithm in order to design
the JSCC coding scheme.Allocating the number of coded symbols according to
the optimal limits, that is,(19)Np=H(Up∣Up+1,…,UB)KC yields very bad performance even
at very large block length, since it is known that raptor codes converge to
very small BER at a fixed (small) gap from capacity on general binary-input
symmetric output channels [14].
Therefore, we have to allow for some increment in the coded block lengths,
normally referred to as “overhead” in the raptor coding literature. The
problem is how to allocate a total overhead among the B+1 stages. In order to do so, we propose the
following greedy overhead allocation algorithm.We initialize the lengthsNp(0) according to their nominal value given by
(19). At each iteration of the allocation algorithm, we allocate a given number ΔN of additional coded symbols to one of the B+1 codes. Let D(N0,…,NB) denote the achieved average distortion of the
JSCC scheme when coding lengths N0,…,NB are used and let D(0)=D(N0(0),…,NB(0)).
Then, for iterations i=1,2,…,
do the following.(i)
For allp=0,…,B,
compute (20)Dp(i)=D(N0(i−1),…,Np(i−1)+ΔN,…,NB(i−1)).(ii)
Findp^=arg⁡minp=0,…,B⁡Dp(i).(iii)
LetNp(i)←Np(i−1) for all p≠p^,
and Np^(i)←Np^(i−1)+ΔN.(iv)
If|Dp^(i)−D𝒬(B)|≤δ,
exit. Otherwise, let D(i)←Dp^(i) and go back to 1. The quantity δ>0 is the tolerance within which we wish to
achieve the target quantization distortion.In essence, the above algorithm allocates at each
iteration a packet ofΔN-coded bits to the bitplane raptor encoder
that yields the largest decrease in the overall average distortion. The
distortion can be computed either by Monte Carlo simulation, or by using the
EXIT chart approximation. The latter method is much faster, but cannot take
into account the effect of finite block length and the error propagation
between the stages of the multistage decoder.In Figure11,
we report the comparison between the finite length simulation and the infinite
length EXIT approximation for the same setting of B ranging from 1 to 6, the BSC with capacity C=0.5,
and source block length K=10000 used throughout the paper. As we can see, the
two cases yield almost identical results. This allows us to use the infinite
length EXIT approximation to estimate (with very good approximation) a suitable
rate allocation among the B+1 stages for the finite length case. Finally, in Figure 12, for the case of B=3,
we report the relative overhead Np(i)/Np(0) versus PSNR(i)=−10log⁡10D(i) produced by the greedy allocation algorithm. As one might expect, the
greedy algorithm starts increasing the overhead of the sign bitplane and then
continues from the most significant to least significant magnitude bitplanes.
Eventually, each bitplane is allocated a coding length between 12% and 18%
larger than the nominal length (19), in line with standard raptor coding
reported results. Furthermore, notice that this scheme tends to give larger
overheads to most significant bitplanes, that is, it implicitly implements
unequal error protection across the layers, which is a very well-known design
approach for multilevel coded modulation with multistage decoding [21].Figure 11
PSNR versusb comparison of JSCC
scheme for finite block length simulation and infinite block length EXIT
approximation for C=0.5.Figure 12
Relative overheadNp(i)/Np(0) versus PSNR(i)=−10log⁡10D(i) produced by the greedy allocation algorithm for the case B=3.
We notice that the bitplane coding overheads are incremented one at a time, in
sequence.


## 4.1. Rate Allocation Algorithm

The rate allocation of each bitplane encoder is
established offline by using the greedy algorithm described below. Again, we
notice that we do not consider adaptive rate allocation: given the source and
channel statistics, we run the greedy allocation algorithm in order to design
the JSCC coding scheme.Allocating the number of coded symbols according to
the optimal limits, that is,(19)Np=H(Up∣Up+1,…,UB)KC yields very bad performance even
at very large block length, since it is known that raptor codes converge to
very small BER at a fixed (small) gap from capacity on general binary-input
symmetric output channels [14].
Therefore, we have to allow for some increment in the coded block lengths,
normally referred to as “overhead” in the raptor coding literature. The
problem is how to allocate a total overhead among the B+1 stages. In order to do so, we propose the
following greedy overhead allocation algorithm.We initialize the lengthsNp(0) according to their nominal value given by
(19). At each iteration of the allocation algorithm, we allocate a given number ΔN of additional coded symbols to one of the B+1 codes. Let D(N0,…,NB) denote the achieved average distortion of the
JSCC scheme when coding lengths N0,…,NB are used and let D(0)=D(N0(0),…,NB(0)).
Then, for iterations i=1,2,…,
do the following.(i)
For allp=0,…,B,
compute (20)Dp(i)=D(N0(i−1),…,Np(i−1)+ΔN,…,NB(i−1)).(ii)
Findp^=arg⁡minp=0,…,B⁡Dp(i).(iii)
LetNp(i)←Np(i−1) for all p≠p^,
and Np^(i)←Np^(i−1)+ΔN.(iv)
If|Dp^(i)−D𝒬(B)|≤δ,
exit. Otherwise, let D(i)←Dp^(i) and go back to 1. The quantity δ>0 is the tolerance within which we wish to
achieve the target quantization distortion.In essence, the above algorithm allocates at each
iteration a packet ofΔN-coded bits to the bitplane raptor encoder
that yields the largest decrease in the overall average distortion. The
distortion can be computed either by Monte Carlo simulation, or by using the
EXIT chart approximation. The latter method is much faster, but cannot take
into account the effect of finite block length and the error propagation
between the stages of the multistage decoder.In Figure11,
we report the comparison between the finite length simulation and the infinite
length EXIT approximation for the same setting of B ranging from 1 to 6, the BSC with capacity C=0.5,
and source block length K=10000 used throughout the paper. As we can see, the
two cases yield almost identical results. This allows us to use the infinite
length EXIT approximation to estimate (with very good approximation) a suitable
rate allocation among the B+1 stages for the finite length case. Finally, in Figure 12, for the case of B=3,
we report the relative overhead Np(i)/Np(0) versus PSNR(i)=−10log⁡10D(i) produced by the greedy allocation algorithm. As one might expect, the
greedy algorithm starts increasing the overhead of the sign bitplane and then
continues from the most significant to least significant magnitude bitplanes.
Eventually, each bitplane is allocated a coding length between 12% and 18%
larger than the nominal length (19), in line with standard raptor coding
reported results. Furthermore, notice that this scheme tends to give larger
overheads to most significant bitplanes, that is, it implicitly implements
unequal error protection across the layers, which is a very well-known design
approach for multilevel coded modulation with multistage decoding [21].Figure 11
PSNR versusb comparison of JSCC
scheme for finite block length simulation and infinite block length EXIT
approximation for C=0.5.Figure 12
Relative overheadNp(i)/Np(0) versus PSNR(i)=−10log⁡10D(i) produced by the greedy allocation algorithm for the case B=3.
We notice that the bitplane coding overheads are incremented one at a time, in
sequence.


## 5. Numerical Results

In this section, we provide both finite length and infinite
length results. We considered source block lengthK=10000 for the finite length results. In all the
numerical results of this paper, we considered raptor codes with the “LT”
degree distribution [14](21)Ω(x)=0.008x+0.494x2+0.166x3+0073x4+0.083x5+0.056x8+0.037x9+0.056x19+0.025x65+0.003x66. As outer code
we used a regular high-rate LDPC code with degrees (2,100) and rate 0.98. The source symbols are
estimated after running 100 iterations of the decoding algorithm.We would like to stress the fact that the LT and LDPC
degree distribution polynomials have been chosen without considering any
optimization method and that we have averaged over the ensemble of randomly
generated raptor codes with the given parameters. In practice, one would
carefully design an LDPC graph with good properties for the desired lengthK and degree distributions.This section is subdivided in two parts. In the first
part we described the results obtained by varying of the bandwidth expansion
factor, when the capacity of the BSC is fixed toC=0.5,
corresponding to crossover probability ϵ=0.11.
The aim of this section is to compare the performance of families of SSCC and
JSCC codes for different values of b,
and to see how they approach the operational Shannon limit.In the second part we examine the behavior of asingle fixed code, designed for a nominal
channel crossover probability and target PSNR, when we vary the channel
crossover probability. This set of results illustrates the robustness of a
given coding scheme to nonideal channel conditions.In both subsections we provide results for infinite
and finite codeword length cases. The infinite case results have been generated
by using the EXIT chart approximation of AppendixB.


### 5.1. Approaching the Operational Shannon Limit

In Figure13, we plot the performance comparison
between the proposed JSSC scheme and the SSCC scheme, when infinite codeword
length is considered. In this case, the SSCC scheme outperforms the proposed
scheme in the sense that it reaches the quantization distortion at slightly
lower values of b,
for all B=1,…,6.
The SSCC schemes show a very sharp transition (“all or nothing” behavior). In
contrast, the JSCC schemes reach their quantization PSNR more gradually: as we
increase the overhead, the performance gradually improves.Figure 13
JSCC and SSCC
infinite block length comparison forB=1,2,3,4,5,6 and C=0.5.The situation radically changes when we consider
finite codeword length. In Figure14 we plot the performance of JSSC and SSCC
schemes for finite block length. In this case, the JSCC schemes outperform
their SSCC counterpart. In particular, as we have already remarked, the JSCC
performance is almost identical to that for infinite block length, while the
SSCC suffers much more evidently from the residual BER due to finite length
practical codes. This also hints that the EXIT approximated analysis yields very
faithful results for the JSCC scheme, while it provides optimistic results for
the SSCC scheme. This can be explained by the fact that the BER performance of
infinite length codes exhibits a very sharp “waterfall” threshold, beyond
which the BER is zero, while for finite length the waterfall is smoother.Figure 14
JSCC and SSCC finite
block length comparison forB=1,2,3,4,5,6 and C=0.5.An important advantage of the JSCC is that the PSNR
value gradually increases asb increases, while a sharp threshold effect can
be seen in the case of SSCC. In [5] it was shown that, with natural sources such as
images, PSNR values lower than peak value were still perceptually acceptable
for the JSCC scheme, while the SSCC scheme degrades abruptly also from the
perceptual viewpoint.


### 5.2. Robustness

In the previous set of results, we have fixed the
channel capacity and the (quantized) source entropy rate and we have examined
families of codes operating at different(b,PSNR) points. Now, we take a complementary view and
fix the channel code while letting the channel capacity vary. This setting is
relevant when a given code, designed for some nominal channel conditions, is
used on a channel of variable quality, and therefore we are interested in the
robustness of the performance with respect to the channel parameters. Also,
this setting is more akin to the standard way of studying the performance of
channel coding, where the BER is plotted as a function of the channel
parameters (ϵ in the case of a BSC), for a given channel
code.In order to have a fair comparison between the two
schemes, the bandwidth expansion factor (i.e., the code used) has been fixed in
the following way: we keep the minimum value ofb such that both schemes reach the quantization
PSNR on the previous set of results (see Figure 14). In particular we keep b=4.3565 and b=14.8079 for B=1 and B=6,
respectively. Since the JSCC scheme needs lower values of b to reach the quantization PSNR in both cases,
we add some extra bits to reach the same values of b.We have examined the two extreme cases of low
resolution (B=1) and high resolution (B=6). In Figures 15 and 16 we notice that in both
cases the JSCC scheme outperforms the SSCC scheme in terms of PSNR. Moreover,
as expected, the PSNR of the SSCC scheme degrades sharply, while the PSNR of
the JSCC scheme degrades gradually as the channel crossover probability
increases. For example, considering B=6,
if ϵ increases from its nominal value 0.11 to a higher value 0.115 the JSCC scheme loses about 6 dB in PSNR,
while the SSCC loses 24 dB. We interpret this sharp degradation as an effect of
the catastrophic behavior of the entropy coding stage in SSCC, which is greatly
mitigated by the linear coding stage in the proposed JSCC scheme.Figure 15
Comparison of performance degradation of JSCC and SSCC
as the cross-over probability of the BSC increases forK=10000 and B=1.Figure 16
Comparison of performance degradation of JSCC and SSCC
as the cross-over probability of the BSC increases forK=10000 and B=6.


## 5.1. Approaching the Operational Shannon Limit

In Figure13, we plot the performance comparison
between the proposed JSSC scheme and the SSCC scheme, when infinite codeword
length is considered. In this case, the SSCC scheme outperforms the proposed
scheme in the sense that it reaches the quantization distortion at slightly
lower values of b,
for all B=1,…,6.
The SSCC schemes show a very sharp transition (“all or nothing” behavior). In
contrast, the JSCC schemes reach their quantization PSNR more gradually: as we
increase the overhead, the performance gradually improves.Figure 13
JSCC and SSCC
infinite block length comparison forB=1,2,3,4,5,6 and C=0.5.The situation radically changes when we consider
finite codeword length. In Figure14 we plot the performance of JSSC and SSCC
schemes for finite block length. In this case, the JSCC schemes outperform
their SSCC counterpart. In particular, as we have already remarked, the JSCC
performance is almost identical to that for infinite block length, while the
SSCC suffers much more evidently from the residual BER due to finite length
practical codes. This also hints that the EXIT approximated analysis yields very
faithful results for the JSCC scheme, while it provides optimistic results for
the SSCC scheme. This can be explained by the fact that the BER performance of
infinite length codes exhibits a very sharp “waterfall” threshold, beyond
which the BER is zero, while for finite length the waterfall is smoother.Figure 14
JSCC and SSCC finite
block length comparison forB=1,2,3,4,5,6 and C=0.5.An important advantage of the JSCC is that the PSNR
value gradually increases asb increases, while a sharp threshold effect can
be seen in the case of SSCC. In [5] it was shown that, with natural sources such as
images, PSNR values lower than peak value were still perceptually acceptable
for the JSCC scheme, while the SSCC scheme degrades abruptly also from the
perceptual viewpoint.


## 5.2. Robustness

In the previous set of results, we have fixed the
channel capacity and the (quantized) source entropy rate and we have examined
families of codes operating at different(b,PSNR) points. Now, we take a complementary view and
fix the channel code while letting the channel capacity vary. This setting is
relevant when a given code, designed for some nominal channel conditions, is
used on a channel of variable quality, and therefore we are interested in the
robustness of the performance with respect to the channel parameters. Also,
this setting is more akin to the standard way of studying the performance of
channel coding, where the BER is plotted as a function of the channel
parameters (ϵ in the case of a BSC), for a given channel
code.In order to have a fair comparison between the two
schemes, the bandwidth expansion factor (i.e., the code used) has been fixed in
the following way: we keep the minimum value ofb such that both schemes reach the quantization
PSNR on the previous set of results (see Figure 14). In particular we keep b=4.3565 and b=14.8079 for B=1 and B=6,
respectively. Since the JSCC scheme needs lower values of b to reach the quantization PSNR in both cases,
we add some extra bits to reach the same values of b.We have examined the two extreme cases of low
resolution (B=1) and high resolution (B=6). In Figures 15 and 16 we notice that in both
cases the JSCC scheme outperforms the SSCC scheme in terms of PSNR. Moreover,
as expected, the PSNR of the SSCC scheme degrades sharply, while the PSNR of
the JSCC scheme degrades gradually as the channel crossover probability
increases. For example, considering B=6,
if ϵ increases from its nominal value 0.11 to a higher value 0.115 the JSCC scheme loses about 6 dB in PSNR,
while the SSCC loses 24 dB. We interpret this sharp degradation as an effect of
the catastrophic behavior of the entropy coding stage in SSCC, which is greatly
mitigated by the linear coding stage in the proposed JSCC scheme.Figure 15
Comparison of performance degradation of JSCC and SSCC
as the cross-over probability of the BSC increases forK=10000 and B=1.Figure 16
Comparison of performance degradation of JSCC and SSCC
as the cross-over probability of the BSC increases forK=10000 and B=6.


## 6. Conclusions

Unlike most JSCC schemes
presented in the literature, which are carefully targeted for specific source
and channel pairs, the scheme proposed here can closely approach the
rate-distortion separation limit for virtually any well-behaved source under
quadratic distortion and any symmetric channel, owing to the universality of
entropy-coded quantization and the optimality of linear codes for both data
compression and channel coding. Furthermore, we have demonstrated that beyond
operating close to optimal, the proposed scheme is better and more robust than
a separated approach, especially in the practical case of finite block length.We wish to conclude this paper with some
considerations for future work. Following [5], the JSCC scheme can be applied to any class of
sources for which efficient transform coding has been designed. In particular,
images, audio and video are natural and relevant candidates. The scheme takes
advantage of the know-how and careful source statistical characterization
developed in designing lossy coding standards, and preserves the structure of
the transform coder. This makes it easy to introduce the JSCC scheme into
practical applications, for example, by introducing a trans-coding stage at the
physical layer, while preserving the network architecture and the source coding
standards developed at the application layer.Although we have not pursued this aspect here, the
bitplane layered encoding and multistage successive decoding architectures of
the proposed scheme lend themselves quite naturally to a multiresolution, or
“embedded," implementation. For example, it is sufficient to use an embedded
scalar quantizer in order to obtain such a scheme: bitplanes will be
transmitted in sequence, and the resolution of the reconstructed source
improves at each additional layer received.A different route for future investigation involves
the use of nonbinary linear codes. Also for the proposed JSCC scheme, the gap
from the Shannon limit increases with the PSNR (resolution). This is due to the
fact that each layer needs to be encoded with a fixed overhead, such that the
overall overhead increases with the number of layers. As an alternative, we may
wish to use a nonbinary raptor code operating over symbols ofB+1 bits, and mapping directly the quantization
indices over the channel symbols. The hope is that the overhead of such
nonbinary codes does not depend (or at least depends in a sublinear way) on B.
This may lead to better bandwidth expansion gaps at high resolution.



---

*Source: 124685-2008-09-15.xml*