advances in residual vector quantization a review advances in residual vector quantization rvq are surveyed definitions of joint encoder optimality and joint decoder optimality are discussed design techniques for rvqs with large numbers of stages and generally different encoder and decoder codebooks are elaborated and extended fixed rate rvqs and variable rate rvqs that employ entropy coding are examined predictive and finite state rvqs designed and integrated into neural network based source coding structures are revisited successive approximation rvqs that achieve embedded and refinable coding are reviewed a new type of successive approximation rvq that varies the instantaneous block rate by using different numbers of stages on different blocks is introduced and applied to image waveforms and a scalar version of the new residual quantizer is applied to image subbands in an embedded wavelet transform coding system

on l_ infty properties of multiresolution scalar quantizers we investigate the max norm l properties of multiresolution scalar quantizers mrsq the multiresolution requirement imposes nontrivial constraints on the maximum distortion at each level of the quantizer to quantify these constraints we define the overall multiresolution l distortion of an mrsq to be a weighted sum of l distortions over all refinement levels of the mrsq we then seek mrsq constructions that minimize this average max distortion measure an interesting relationship between this problem and the structure of huffman code trees is established lower bounds for the average max distortion are derived based on this relationship the derivation of these lower bounds also lead to efficient dynamic programming heuristic solutions

failure of successive refinement for symmetric gaussian mixtures we show that information refinement fails for a continuous alphabet source whose input distribution is the symmetric mixture of normal densities with means spl mu and spl mu and common variance spl sigma sup 2

a survey of the theory of source coding with a fidelity criterion the purpose of this paper is threefold 1 to acquaint the reader with the types of problems that have been considered in the area of source coding with a fidelity criterion 2 to survey results that have been obtained on these problems and 3 to outline future research trends in the area

universal multiresolution source codes a multiresolution source code is a single code giving an embedded source description that can be read at a variety of rates and thereby yields reproductions at a variety of resolutions the resolution of a source reproduction here refers to the accuracy with which it approximates the original source thus a reproduction with low distortion is a high resolution reproduction while a reproduction with high distortion is a low resolution reproduction this paper treats the generalization of universal lossy source coding from single resolution source codes to multiresolution source codes results described in this work include new definitions for weakly minimax universal strongly minimax universal and weighted universal sequences of fixed and variable rate multiresolution source codes that extend the corresponding notions from lossless coding and single resolution quantization to multiresolution quantizers a variety of universal multiresolution source coding results follow including necessary and sufficient conditions for the existence of universal multiresolution codes rate of convergence bounds for universal multiresolution coding performance to the theoretical bound and a new multiresolution approach to two stage universal source coding

joint wyner ziv dirty paper coding by modulo lattice modulation the combination of source coding with decoder side information the wyner ziv problem and channel coding with encoder side information the gel fand pinsker problem can be optimally solved using the separation principle in this work we show an alternative scheme for the quadratic gaussian case which merges source and channel coding this scheme achieves the optimal performance by applying a modulo lattice modulation to the analog source thus it saves the complexity of quantization and channel decoding and remains with the task of ldquoshapingrdquo only furthermore for high signal to noise ratio snr the scheme approaches the optimal performance using an snr independent encoder thus it proves for this special case the feasibility of universal joint source channel coding

general embedded quantization for wavelet based lossy image coding embedded quantization is a mechanism employed by many lossy image codecs to progressively refine the distortion of a transformed image currently the most common approach to do so in the context of wavelet based image coding is to couple uniform scalar deadzone quantization usdq with bitplane coding bpc usdq bpc is convenient for its practicality and has proved to achieve competitive coding performance but the quantizer established by this scheme does not allow major variations this paper introduces a multistage quantization scheme named general embedded quantization geq that provides more flexibility to the quantizer geq schemes can be devised for specific decoding rates achieving optimal coding performance practical approaches of geq schemes achieve coding performance similar to that of usdq bpc while requiring fewer quantization stages the performance achieved by geq is evaluated in this paper through experimental results carried out in the framework of modern image coding systems

estimate of the solution of the dirichlet problem for parabolic equations and applications in the present paper we consider the dirichlet problem for quasilinear nonuniformly parabolic equations a new sufficient condition which guarantees the a priori estimate of the maximum of the modulus of the solution is formulated a several applications of this estimate are given

variable rate vector quantization for medical image compression three techniques for variable rate vector quantizer design are applied to medical images the first two are extensions of an algorithm for optimal pruning in tree structured classification and regression due to breiman et al the code design algorithms find subtrees of a given tree structured vector quantizer tsvq each one optimal in that it has the lowest average distortion of all subtrees of the tsvq with the same or lesser average rate since the resulting subtrees have variable depth natural variable rate coders result the third technique is a joint optimization of a vector quantizer and a noiseless variable rate code this technique is relatively complex but it has the potential to yield the highest performance of all three techniques

rate distortion theory for causal video coding characterization computation algorithm and comparison causal video coding is considered from an information theoretic point of view where video source frames x1 x2 xn are encoded in a frame by frame manner the encoder for each frame xk can use all previous frames and all previous encoded frames while the corresponding decoder can use only all previous encoded frames and each frame xk itself is modeled as a source xk xk i i 1 a novel computation approach is proposed to analytically characterize numerically compute and compare the minimum total rate of causal video coding rc d1 dn required to achieve a given distortion quality level d1 dn 0 among many other things the computation approach includes an iterative algorithm with global convergence for computing rc d1 dn the global convergence of the algorithm further enables us to demonstrate a somewhat surprising result dubbed the more and less coding theorem under some conditions on source frames and distortion the more frames need to be encoded and transmitted the less amount of data after encoding has to be actually sent with the help of the algorithm it is also shown by example that rc d1 dn is in general much smaller than the total rate offered by the traditional greedy coding method as a by product an extended markov lemma is established for correlated ergodic sources

successive refinement of information characterization of the achievable rates let r spl middot be the rate distortion function assume that we want to describe a source with distortion no larger than spl delta sub 1 from the rate distortion theory we know that we need to do so at a rate r sub 1 no smaller than r spl delta sub 1 spl lsqb bits symbol spl rsqb if it turns out that a more accurate description at distortion spl delta sub 2 spl delta sub 2

practical codes for collaborative estimation in collaborative estimation a network of nodes each indirectly observing an underlying source communicate with each other to form improved estimates of the underlying source this paper derives an efficient collaborative estimation algorithm achieving high performance estimates with low communication and complexity utilizing tools from multiterminal source coding theory and modern practical coding theory relevant theoretical limits based on achievability from multiterminal source coding are first presented in order to characterize efficient tradeoffs between communication and estimation performance and inspire the architecture of the collaborative estimation code next low complexity tools from modern practical coding theory are utilized to get a practical collaborative estimation algorithm approaching these performance limits the developed algorithm utilizes successively refined trellis coded quantization sr tcq to provide necessary diverse descriptions of the source and low density parity check ldpc codes to provide an efficient means of compressing these descriptions for low complexity belief propagation decoding with side information comparing the communication versus estimate quality tradeoff performance attained by the developed low complexity scheme with that obtainable with an inner bound an average distortion gap of only 1 0 and 1 6 db at average rates of 3 32 and 2 33 b s respectively is observed

the rate distortion region for multiple descriptions without excess rate during recent years there has been strong interest in a certain source coding problem which some authors call the problem of multiple descriptions old and new wringing techniques enable us to establish a single letter characterization of the rate distortion region in the case of no excess rate for the joint description

nested linear lattice codes for structured multiterminal binning network information theory promises high gains over simple point to point communication techniques at the cost of higher complexity however lack of structured coding schemes limited the practical application of these concepts so far one of the basic elements of a network code is the binning scheme wyner 1974 1978 and other researchers proposed various forms of coset codes for efficient binning yet these schemes were applicable only for lossless source or noiseless channel network coding to extend the algebraic binning approach to lossy source or noisy channel network coding previous work proposed the idea of nested codes or more specifically nested parity check codes for the binary case and nested lattices in the continuous case these ideas connect network information theory with the rich areas of linear codes and lattice codes and have strong potential for practical applications we review these developments and explore their tight relation to concepts such as combined shaping and precoding coding for memories with defects and digital watermarking we also propose a few novel applications adhering to a unified approach

asymmetric multilevel diversity coding and asymmetric gaussian multiple descriptions we consider the asymmetric multilevel diversity a mld coding problem where a set of 2k 1 information sources ordered in a decreasing level of importance is encoded into k messages or descriptions there are 2k 1 decoders each of which has access to a nonempty subset of the encoded messages each decoder is required to reproduce the information sources up to a certain importance level depending on the combination of descriptions available to it we obtain a single letter characterization of the achievable rate region for the 3 description problem in contrast to symmetric multilevel diversity coding source separation coding is not sufficient in the asymmetric case and ideas akin to network coding need to be used strategically based on the intuitions gained in treating the a mld problem we derive inner and outer bounds for the rate region of the asymmetric gaussian multiple description md problem with three descriptions both the inner and outer bounds have a similar geometric structure to the rate region template of the a mld coding problem and moreover we show that the gap between them is constant which results in an approximate characterization of the asymmetric gaussian three description rate region

image coding based on a morphological representation of wavelet data an experimental study of the statistical properties of wavelet coefficients of image data is presented as well as the design of two different morphology based image coding algorithms that make use of these statistics a salient feature of the proposed methods is that by a simple change of quantizers the same basic algorithm yields high performance embedded or fixed rate coders another important feature is that the shape information of morphological sets used in this coder is encoded implicitly by the values of wavelet coefficients thus avoiding the use of explicit and rate expensive shape descriptors these proposed algorithms while achieving nearly the same objective performance of state of the art zerotree based methods are able to produce reconstructions of a somewhat superior perceptual quality due to a property of joint compression and noise reduction they exhibit

network vector quantization we present an algorithm for designing locally optimal vector quantizers for general networks we discuss the algorithm s implementation and compare the performance of the resulting network vector quantizers to traditional vector quantizers vqs and to rate distortion r d bounds where available while some special cases of network codes e g multiresolution mr and multiple description md codes have been studied in the literature we here present a unifying approach that both includes these existing solutions as special cases and provides solutions to previously unsolved examples

a lossy source coding interpretation of wyner s common information wyner s common information was originally defined for a pair of dependent discrete random variables its significance is largely reflected in and also confined to several existing interpretations in various source coding problems this paper attempts to expand its practical significance by providing a new operational interpretation in the context of the gray wyner network it is established that wyner s common information has a new lossy source coding interpretation specifically it is established that under suitable conditions wyner s common information equals to the smallest common message rate when the total rate is arbitrarily close to the rate distortion function with joint decoding for the gray wyner network a surprising observation is that such equality holds independent of the values of distortion constraints as long as the distortions are within some distortion region the new lossy source coding interpretation provides the first meaningful justification for defining wyner s common information for continuous random variables and the result can also be extended to that of multiple variables examples are given for characterizing the rate distortion region for the gray wyner lossy source coding problem and for identifying conditions under which wyner s common information equals that of the smallest common rate as a by product the explicit expression for the common information between a pair of gaussian random variables is obtained

lossy compression via sparse linear regression computationally efficient encoding and decoding we propose computationally efficient encoders and decoders for lossy compression using a sparse regression code the codebook is defined by a design matrix and codewords are structured linear combinations of columns of this matrix the proposed encoding algorithm sequentially chooses columns of the design matrix to successively approximate the source sequence it is shown to achieve the optimal distortion rate function for independent identically distributed i i d gaussian sources under the squared error distortion criterion for a given rate the parameters of the design matrix can be varied to tradeoff distortion performance with encoding complexity an example of such a tradeoff as a function of the block length n is the following with computational resource space or time per source sample of o n log n 2 for a fixed distortion level above the gaussian distortion rate function the probability of excess distortion decays exponentially in n the sparse regression code is robust in the following sense for any ergodic source the proposed encoder achieves the optimal distortion rate function of an i i d gaussian source with the same variance simulations show that the encoder has good empirical performance especially at low and moderate rates

ldgm based multiple description coding for finite alphabet sources this work presents an ldgm based practical successive coding scheme for the multiple description md problem for finite alphabet sources the scheme which targets the zhang berger zb rate distortion region is shown to be asymptotically optimal with joint typicality encoding while as a practical encoding solution a message passing algorithm is adopted we further discuss in more detail the application of the coding scheme in three cases of the md problem with the hamming distortion measure 1 no excess sum rate for binary sources 2 successive refinement and 3 no excess marginal rate for the uniform binary source in the no excess sum rate case some progress is made in the characterization of fundamental limits by deriving the analytical expression of the distortion region for general binary sources and of the auxiliary variables needed to achieve its boundary the exact expression of the zhang berger upper bound to the central distortion is also provided for the case of no excess marginal rate for the uniform binary source the proposed ldgm based coding scheme is tested in practice for all three aforementioned cases the experimental results show very good performance demonstrating its ability to approach the theoretical rate distortion limits or the available upper bounds

distortion minimization in gaussian layered broadcast coding with successive refinement a transmitter without channel state information wishes to send a delay limited gaussian source over a slowly fading channel the source is coded in superimposed layers with each layer successively refining the description in the previous one the receiver decodes the layers that are supported by the channel realization and reconstructs the source up to a distortion the expected distortion is minimized by optimally allocating the transmit power among the source layers for two source layers the allocation is optimal when power is first assigned to the higher layer up to a power ceiling that depends only on the channel fading distribution all remaining power if any is allocated to the lower layer for convex distortion cost functions with convex constraints the minimization is formulated as a convex optimization problem in the limit of a continuum of infinite layers the minimum expected distortion is given by the solution to a set of linear differential equations in terms of the density of the fading distribution as the number of channel uses per source symbol tends to zero the power distribution that minimizes expected distortion converges to the one that maximizes expected capacity

optimal pruning with applications to tree structured source coding and modeling an algorithm introduced by l breiman et al 1984 in the context of classification and regression trees is reinterpreted and extended to cover a variety of applications in source coding and modeling in which trees are involved these include variable rate and minimum entropy tree structured vector quantization minimum expected cost decision trees variable order markov modeling optimum bit allocation and computer graphics and image processing using quadtrees a concentration on the first of these and a detailed analysis of variable rate tree structured vector quantization are provided it is found that variable rate tree structured vector quantization outperforms not only the fixed rate variety but also full search vector quantization the successive approximation character of variable rate tree structured vector quantization permits it to degrade gracefully if the rate is reduced at the encoder this has applications to the problem of buffer overflow

guessing subject to distortion we investigate the problem of guessing a random vector x within distortion level d our aim is to characterize the best attainable performance in the sense of minimizing in some probabilistic sense the number of required guesses g x until the error falls below d the underlying motivation is that g x is the number of candidate codewords to be examined by a rate distortion block encoder until a satisfactory codeword is found in particular for memoryless sources we provide a single letter characterization of the least achievable exponential growth rate of the spl rho th moment of g x as the dimension of the random vector x grows without bound in this context we propose an asymptotically optimal guessing scheme that is universal both with respect to the information source and the value of spl rho we then study some properties of the exponent function e d spl rho along with its relation to the source coding exponents finally we provide extensions of our main results to the gaussian case guessing with side information and sources with memory

n channel entropy constrained multiple description lattice vector quantization in this paper we derive analytical expressions for the central and side quantizers which under high resolution assumptions minimize the expected distortion of a symmetric multiple description lattice vector quantization md lvq system subject to entropy constraints on the side descriptions for given packet loss probabilities we consider a special case of the general n channel symmetric multiple description problem where only a single parameter controls the redundancy tradeoffs between the central and the side distortions previous work on two channel md lvq showed that the distortions of the side quantizers can be expressed through the normalized second moment of a sphere we show here that this is also the case for three channel md lvq furthermore we conjecture that this is true for the general n channel md lvq for given source target rate and packet loss probabilities we find the optimal number of descriptions and construct the md lvq system that minimizes the expected distortion we verify theoretical expressions by numerical simulations and show in a practical setup that significant performance improvements can be achieved over state of the art two channel md lvq by using three channel md lvq

new results in binary multiple descriptions an encoder whose input is a binary equiprobable memoryless source produces one output of rate r 1 and another of rate r 2 let d 1 d 2 and d 0 respectively denote the average error frequencies with which the source data can be reproduced on the basis of the encoder output of rate r l only the encoder output of rate r 2 only and both encoder outputs the two descriptions problem is to determine the region r of all quintuples r 1 r 2 d 1 d 2 d 0 that are achievable in thc usual shannon sense let r d 1 d log 2 d 1 d log 2 1 d denote the error frequency rate distortion function of the source the no excess rate case prevails when r 1 r 2 r d 0 and the excess rate case when r 1 r 2 r d 0 denote the section of r at r 1 r 2 d 0 by d r 1 r 2 d 0 d 1 d 2 r 1 r 2 d 1 d 2 d 0 in r in the no excess rate case we show that a portion of the boundary of d r 1 r 2 d 0 coincides with the curve frac 1 2 d 1 2d 0 frac 1 2 d 2 2d 0 frac 1 2 1 2d 0 2 this curve is an extension of witsenhausen s hyperbola bound to the case d 0 0 it follows that the projection of r onto the d 1 d 2 plane at fixed d 0 consists of all d 1 geq d 0 and d 2 geq d 0 that lie on or above this hyperbola in the excess rate case we show by counterexample that the achievable region of el gamal and cover is not tight

successive refinement of vector sources under individual distortion criteria the successive refinement problem is extended to vector sources where individual distortion constraints are posed on each vector component for vector gaussian sources with squared error distortion a single letter rate distortion characterization is inherited from the previously studied gaussian multiple descriptions problem with covariance distortion constraints though this characterization is amenable to well known numerical convex optimization techniques an analytical solution is difficult to obtain in full generality even for 2 d sources in this work the special case of successive refinability is addressed analytically specifically vector gaussian sources are shown to be not successively refinable everywhere unlike scalar gaussian sources it is also shown that for 2 d gaussian sources the rate loss at the second stage can be as high as 0 5 b sample in a degenerate scenario corresponding to what is known as sequential coding of correlated sources finally analysis of 2 d binary symmetric sources with hamming distortion reveals that the behavior of these sources with respect to successive refinability exhibits remarkable similarity to their 2 d gaussian counterparts

an improvement to multiple description transform coding a multiple description transform coding mdtc method has been reported previously the redundancy rate distortion rrd performance of this coding scheme for the independent and identically distributed i i d two dimensional 2 d gaussian source has been analyzed using mean squared error mse as the distortion measure at the small redundancy region the mdtc scheme can achieve excellent rrd performance because a small increase in redundancy can reduce the single description distortion at a rate faster than exponential but the performance of mdtc becomes increasingly poor at larger redundancies this paper describes a generalization of the mdtc gmdtc scheme which introduces redundancy both by transform and through correcting the error resulting from a single description its rrd performance is closer to the theoretical bound in the entire range of redundancy analysis both for a single pair of variables and for multiple variables is presented

progressive coding of a gaussian source using matching pursuit in this paper the application of matching pursuit mp in progressive coding of memoryless gaussian sources is studied in addition a detailed analysis of the rate distortion performance of our proposed mp encoder is presented and the distortion of the mp encoder is derived in terms of dictionary size and number of quantization levels and the optimum parameters are calculated our analysis is based on a probabilistic model for matching pursuit residual vectors our simulation results verify the accuracy of our analysis and show that matching pursuit can produce an embedded bitstream with comparable quality to existing quantizers the mp encoder outperforms quantizers that are capable of producing embedded bitstreams

tracking the best quantizer an algorithm is presented for online prediction that allows to track the best expert efficiently even when the number of experts is exponentially large provided that the set of experts has a certain additive structure as an example we work out the case where each expert is represented by a path in a directed graph and the loss of each expert is the sum of the weights over the edges in the path these results are then used to construct universal limited delay schemes for lossy coding of individual sequences in particular we consider the problem of tracking the best scalar quantizer that is adaptively matched to the source sequence with piecewise different behavior a randomized algorithm is presented which can perform on any source sequence asymptotically as well as the best scalar quantization algorithm that is matched to the sequence and is allowed to change the employed quantizer for a given number of times the complexity of the algorithm is quadratic in the sequence length but at the price of some deterioration in performance the complexity can be made linear analogous results are obtained for sequential multiresolution and multiple description scalar quantization of individual sequences

multiple description wavelet based image coding we consider the problem of coding images for transmission over error prone channels the impairments we target are transient channel shutdowns as would occur in a packet network when a packet is lost or in a wireless system during a deep fade when data is delivered it is assumed to be error free but some of the data may never reach the receiver the proposed algorithms are based on a combination of multiple description scalar quantizers with techniques successfully applied to the construction of some of the most efficient subband coders a given image is encoded into multiple independent packets of roughly equal length when packets are lost the quality of the approximation computed at the receiver depends only on the number of packets received but does not depend on exactly which packets are actually received when compared with previously reported results on the performance of robust image coders based on multiple descriptions on standard test images our coders attain similar psnr values using typically about 50 60 of the bit rate required by these other state of the art coders while at the same time providing significantly more freedom in the mechanism for allocation of redundancy among descriptions

hybrid digital analog hda joint source channel codes for broadcasting and robust communications we consider the problem of transmitting a band limited gaussian source on an additive band limited gaussian noise channel the well known threshold effect dictates that the more powerful a code is the more sensitive it is to the exact knowledge of the channel noise a code is said to be robust if it is asymptotically optimal for a wide range of channel noise thus robust codes have a graceful degradation characteristic and are free of the threshold effect it is demonstrated that robust codes exist whenever the source and channel bandwidths are equal in the unequal bandwidth case a collection of nearly robust joint source channel codes is constructed using a hybrid digital analog hda coding technique for designing nearly robust codes a matched tandem code whose channel encoder s output is partially fully matched to its input is proposed and the existence of an asymptotically optimal matched tandem code is shown the nearly robust codes achieve the shannon limit theoretically optimum distortion and have a less severe threshold effect finally for the case of two different noise conditions the distortion regions of these codes are determined

variable rate tree structured vector quantizers in general growth algorithms for optimal tree structured vector quantizers do not exist in this paper we show that if the source satisfies certain conditions namely that of diminishing marginal returns optimal growth algorithms do exist we present such an algorithm and compare its performance with that of other tree growth algorithms even for sources that do not meet the necessary conditions for the growth algorithm to be optimal such as for speech with unknown statistics it is seen by simulation that the algorithm outperforms other known growth algorithms for sources that do not satisfy the required conditions the algorithm presented here can also be used to grow the initial tree for the pruning process the performance of such pruned trees is superior to that of trees pruned from full trees of the same rate

multiuser successive refinement and multiple description coding in this correspondence we consider the multiuser successive refinement msr problem where the users are connected to a central server via links with different noiseless capacities and each user wishes to reconstruct in a successive refinement fashion an achievable region is given for the two user two layer case and it provides the complete rate distortion region for the gaussian source under the mse distortion measure the key observation is that this problem includes the multiple description md problem with two descriptions as a subsystem and the techniques useful in the md problem can be extended to this case it is shown that the coding scheme based on the universality of random binning is suboptimal because multiple gaussian side informations only at the decoders do incur performance loss in contrast to the case of single side information at the decoder it is further shown that unlike the single user case when there are multiple users the loss of performance by a multistage coding approach can be unbounded for the gaussian source the result suggests that in such a setting the benefit of using successive refinement is not likely to justify the accompanying performance loss the msr problem is also related to the source coding problem where each decoder has its individual side information while the encoder has the complete set of the side informations the msr problem further includes several variations of the md problem for which the specialization of the general result is investigated and the implication is discussed

fast variable run length coding for embedded progressive wavelet based image compression run length coding has attracted much attention in wavelet based image compression because of its simplicity and potentially low complexity the main drawback is the inferior rd performance compared to the state of the art coder spiht in this paper we concentrate on the embedded progressive run length code of tian and wells 1996 1998 we consider significance sequences drawn from the scan in the dominant pass it turns out that self similar curves for scanning the dominant pass increase the compression efficiency significantly this is a consequence of the correlation of direct neighbors in the wavelet domain this dependence can be better exploited by using groups of coefficients similar to the spiht algorithm this results in a new and very fast coding algorithm which shows performance similar to the state of the art coder spiht but with lower complexity and small and fixed memory overhead

strong successive refinability and rate distortion complexity tradeoff we investigate the second order asymptotics source dispersion of the successive refinement problem similar to the classical definition of a successively refinable source we say that a source is strongly successively refinable if successive refinement coding can achieve the second order optimum rate including the dispersion terms at both decoders we establish a sufficient condition for strong successive refinability we show that any discrete source under hamming distortion and the gaussian source under quadratic distortion are strongly successively refinable we also demonstrate how successive refinement ideas can be used in point to point lossy compression problems in order to reduce complexity we give two examples the binary hamming and gaussian quadratic cases in which a layered code construction results in a low complexity scheme that attains optimal performance for example when the number of layers grows with the block length n we show how to design an o n log n algorithm that asymptotically achieves the rate distortion bound

successive refinement of information the successive refinement of information consists of first approximating data using a few bits of information then iteratively improving the approximation as more and more information is supplied the goal is to achieve an optimal description at each stage in general an ongoing description which is rate distortion optimal whenever it is interrupted is sought it is shown that in order to achieve optimal successive refinement the necessary and sufficient conditions are that the solutions of the rate distortion problem can be written as a markov chain in particular all finite alphabet signals with hamming distortion satisfy these requirements it is also shown that the same is true for gaussian signals with squared error distortion and for laplacian signals with absolute error distortion a simple counterexample with absolute error distortion and a symmetric source distribution which shows that successive refinement is not always achievable is presented

on delayed sequential coding of correlated sources motivated by video coding applications the problem of sequential coding of correlated sources with encoding and or decoding frame delays is studied the fundamental tradeoffs between individual frame rates individual frame distortions and encoding decoding frame delays are derived in terms of a single letter information theoretic characterization of the rate distortion region for general interframe source correlations and certain types of potentially frame specific and coupled single letter fidelity criteria the sum rate distortion region is characterized in terms of generalized directed information measures highlighting their role in delayed sequential source coding problems for video sources which are spatially stationary memoryless and temporally gauss markov mse frame distortions and a sum rate constraint our results expose the optimality of idealized differential predictive coding among all causal sequential coders when the encoder uses a positive rate to describe each frame somewhat surprisingly causal sequential encoding with one frame delayed noncausal sequential decoding can exactly match the sum rate mse performance of joint coding for all nontrivial mse tuples satisfying certain positive semidefiniteness conditions thus even a single frame delay holds potential for yielding significant performance improvements generalizations to higher order markov sources are also presented and discussed a rate distortion performance equivalence between causal sequential encoding with delayed noncausal sequential decoding and delayed noncausal sequential encoding with causal sequential decoding is also established

achievable rates for multiple descriptions consider a sequence of independent identically distributed i i d random variables x l x 2 cdots x n and a distortion measure d x i x i on the estimates x i of x i two descriptions i x in 1 2 cdots 2 nr 1 and j x in 1 2 cdots 2 nr 2 are given of the sequence x x 1 x 2 cdots x n from these two descriptions three estimates i x x2 j x and hat x o i x j x are formed with resulting expected distortions e frac 1 n sum n k 1 d x k hat x mk d m m 0 1 2 we find that the distortion constraints d 0 d 1 d 2 are achievable if there exists a probability mass distribution p x p hat x 1 hat x 2 hat x 0 x with ed x hat x m leq d m such that r 1 i x hat x 1 r 2 i x hat x 2 where i cdot denotes shannon mutual information these rates are shown to be optimal for deterministic distortion measures

universal wyner ziv coding for distortion constrained general side information we investigate the wyner ziv coding in which the statistics of the principal source is known but the statistics of the channel generating the side information is unknown except that it is in a certain class the class consists of channels such that the distortion between the principal source and side information is smaller than a threshold but channels may be neither stationary nor ergodic in this situation we define a new rate distortion function as the minimum rate such that there exists a wyner ziv code that is universal for every channel in the class then we show an upper bound and a lower bound on the rate distortion function and derive a matching condition such that the upper and lower bounds coincide the relation between the new rate distortion function and rate distortion function of the heegard berger problem is also discussed

symmetric scalable multiple description scalar quantization real time data delivery over best effort error prone packet networks has invigorated the study of robust coding schemes such as scalable multiple description coding smdc in this context the paper introduces a novel generic symmetric scalable multiple description quantizer ssmdsq which generates perfectly balanced source descriptions novel embedded index assignments are proposed which are used to realize high as well as medium to low redundancy ssmdsqs compared to existing designs it is shown that the proposed quantizer constructions exhibit superior distortion rate d r performance moreover this paper describes an innovative extension of the lloyd max algorithm in order to optimize symmetric and asymmetric scalable multiple description quantizers for a family of generalized gaussian gg source distributions the proposed optimization algorithm yields on average a significant d r performance gain over unoptimized quantizers furthermore anchored in the designed ssmdsqs an smdc framework is established to realize packet based transmission over erasure channels in this framework transmission strategies are determined for scenarios wherein the average packet loss rate over the transmission link is a unknown and b can be estimated at the encoder for both scenarios smdc packetized transmission is simulated for a family of gg distributions experimental results confirm that compared to contemporary schemes the designed quantizer constructions with or without optimization account for a significant average gain in signal to noise ratio snr for a wide range of packet loss rates

optimality and approximate optimality of source channel separation in networks we consider the source channel separation architecture for lossy source coding in communication networks it is shown that the separation approach is optimal in two general scenarios and is approximately optimal in a third scenario the two scenarios for which separation is optimal complement each other the first is when the memoryless sources at source nodes are arbitrarily correlated each of which is to be reconstructed at possibly multiple destinations within certain distortions but the channels in this network are synchronized orthogonal and memoryless point to point channels the second is when the memoryless sources are mutually independent each of which is to be reconstructed only at one destination within a certain distortion but the channels are general including multi user channels such as multiple access broadcast interference and relay channels possibly with feedback the third scenario for which we demonstrate approximate optimality of source channel separation generalizes the second scenario by allowing each source to be reconstructed at multiple destinations with different distortions for this case the loss from optimality using the separation approach can be upper bounded when a difference distortion measure is taken and in the special case of quadratic distortion measure this leads to universal constant bounds

harmonic functions of subordinate killed brownian motion in this paper we study harmonic functions of subordinate killed brownian motion in a domain d we first prove that when the killed brownian semigroup in d is intrinsic ultracontractive all nonnegative harmonic functions of the subordinate killed brownian motion in d are continuous and then we establish a harnack inequality for these harmonic functions we then show that when d is a bounded lipschitz domain both the martin boundary and the minimal martin boundary of the subordinate killed brownian motion in d coincide with the euclidean boundary partial d we also show that when d is a bounded lipschitz domain a boundary harnack principle holds for positive harmonic functions of the subordinate killed brownian motion in d

on the rate loss of multiresolution source codes in the wyner ziv setting in this paper we present upper bounds on the rate loss of the multiresolution source code in the wyner ziv setting wz mrsc we define two types of rate loss the side information rate loss and the multiresolution rate loss to measure the nonsuccessive refinability the former compares the rates achievable by a given wz mrsc to the corresponding minimal rates achievable by single resolution wyner ziv codes and the latter refers to the difference between the rates achievable by a given wz mrsc and the minimal achievable rates by any wz mrsc at the same resolution we bound these two types of rate loss from above and show that the side information rate loss and corresponding successive refinability largely depend on the side information while there exist small universal constant upper bounds on the multiresolution rate loss that are generally independent of the side information

a segmentation based predictive multiresolution image coder recursive rectilinear tessellations like quadtree are widely used in image coding but a regular tessellation despite simple geometry may not suit image compression because it is too rigid to reflect the scene structure of an image the paper presents a new image pyramid formed by adaptive tree structured segmentation to be a framework of a predictive multiresolution image coder subjectively appealing compression results are obtained at different resolutions by scene adaptive tree structured segmentation and by exploiting the statistical dependency between the layers of the image pyramid the adaptive segmentation based image coder is constructed by recursive least squares piecewise functional approximation the seemingly expensive encoding process can be made efficient by an incremental least squares computation technique the decoding is simple and can be done in real time if assisted by existing hardware technology

asymptotic analysis of multiple description quantizers a high rate analysis of multiple description quantizers is presented for rth power distortions and general source densities both fixed length and variable length encoding of the quantizer indices are considered optimal companding functions are shown to be the same as for single channel quantizers as compared to the bound of ozarow 1980 a gap of 8 69 db and 3 07 db exists between the entropy constrained and level constrained cases respectively for a memoryless gaussian source and r 2

a successively refinable lossless image coding algorithm we present a compression technique that provides progressive transmission as well as lossless and near lossless compression in a single framework the proposed technique produces a bit stream that results in a progressive and ultimately lossless reconstruction of an image similar to what one can obtain with a reversible wavelet codec in addition the proposed scheme provides near lossless reconstruction with respect to a given bound after decoding of each layer of the successively refinable bit stream we formulate the image data compression problem as one of successively refining the probability density function pdf estimate of each pixel within this framework restricting the region of support of the estimated pdf to a fixed size interval then results in near lossless reconstruction we address the context selection problem as well as pdf estimation methods based on context data at any pass experimental results for both lossless and near lossless cases indicate that the proposed compression scheme that innovatively combines lossless near lossless and progressive coding attributes gives competitive performance in comparison with state of the art compression schemes

successive refinement for the wyner ziv problem and layered code design we examine successive refinement for the wyner ziv problem described in a recent paper by steinberg and merhav where the authors showed that if the side information for all stages is identical then the jointly gaussian source with squared error distortion measure is successively refinable we first extend this result to the case where the difference between the source and the side information is gaussian and independent of the side information as a byproduct we give an alternative proof that the wyner ziv problem for these sources has no rate loss a result that was recently shown by pradhan et al through invoking the duality between the gaussian wyner ziv problem and the costa problem we then perform layered wyner ziv code design for this general type of source based on nested scalar quantization bit plane coding and low density parity check ldpc code based slepian wolf coding source coding with side information we show that density evolution can be used to analyze the slepian wolf code performance provided that certain symmetry conditions which have been dubbed dual symmetry are satisfied by the hypothetical channel between the source and the side information we also show that the dual symmetry condition is indeed satisfied by the hypothetical channel in our slepian wolf coding setup this justifies the use of density evolution in our ldpc code based slepian wolf code design for wyner ziv coding for the jointly gaussian source our layered coder performs 1 29 to 3 45 db from the wyner ziv bound for rates ranging from 0 47 to 4 68 bits per sample when the side information is laplacian and the source equals the side information plus an independent gaussian noise term our layered coder performs 1 33 to 3 90 db from the wyner ziv bound for rates ranging from 0 48 to 4 64 bits per sample

an on line universal lossy data compression algorithm via continuous codebook refinement i basic results a new on line universal lossy data compression algorithm is presented for finite memoryless sources with unknown statistics its performance asymptotically approaches the fundamental rate distortion limit the codebook is generated on the fly and continuously adapted by simple rules there is no separate codebook training or codebook transmission candidate codewords are randomly generated according to an arbitrary and possibly suboptimal distribution through a carefully designed gold washing or information theoretic sieve mechanism good codewords and only good codewords are promoted to permanent status with high probability we also determine the rate at which our algorithm approaches the fundamental limit

rate distortion for correlated sources with partially separated encoders we resolve some open problems concerning the encoding of a pair of correlated sources with respect to a fidelity criterion two encoders are used one encoder observes only the output of the first source while the other is supplied both with the second source output and with partial information about the first source at some predetermined rate a general coding theorem is proved which established that cal s ast a certain region defined in terms of single letter information theoretic quantities is an inner bound to the region of all attainable vectors of rates and distortions for certain cases of interest the converse is proved too thereby establishing the rate distortion region for these cases

tiling and adaptive image compression we investigate the task of compressing an image by using different probability models for compressing different regions of the image in this task using a larger number of regions would result in better compression but would also require more bits for describing the regions and the probability models used in the regions we discuss using quadtree methods for performing the compression we introduce a class of probability models for images the k rectangular tilings of an image that is formed by partitioning the image into k rectangular regions and generating the coefficients within each region by using a probability model selected from a finite class of n probability models for an image of size n spl times n we give a sequential probability assignment algorithm that codes the image with a code length which is within o k log nn k of the code length produced by the best probability model in the class the algorithm has a computational complexity of o nn sup 3 an interesting subclass of the class of k rectangular tilings is the class of tilings using rectangles whose widths are powers of two this class is far more flexible than quadtrees and yet has a sequential probability assignment algorithm that produces a code length that is within o k log nn k of the best model in the class with a computational complexity of o nn sup 2 logn similar to the computational complexity of sequential probability assignment using quadtrees we also consider progressive transmission of the coefficients of the image

a theory of network equivalence part ii multiterminal channels a technique for bounding the capacities of networks of independent channels is introduced parts i and ii treat point to point and multiterminal channels respectively bounds are derived using a new tool called a bounding model channel 1 is an upper lower bounding model for channel 2 if replacing channel 2 by channel 1 in any network yields a new network whose capacity region is a superset subset of the capacity region of the original network this paper derives bounding models from noiseless links with lower bounding models corresponding to points in the channel s capacity region and upper bounding models corresponding to points in a new channel characterization called an emulation region replacing all channels in a network by their noiseless upper lower bounding models yields a network of lossless links whose capacity region is a superset subset of the capacity region for the original network this converts a general often stochastic network into a network coding instance enabling the application of tools and results derived in that domain a channel s upper and lower bounding models differ when the channel can carry more information in some networks than in others bounding the difference between upper and lower bounding models bounds both the accuracy of the technique and the price of separating source network coding from channel coding

a broadcast approach for a single user slowly fading mimo channel a broadcast transmission strategy for the slowly fading gaussian multiple input multiple output mimo channel is introduced this broadcast strategy is an extension of the single input single output siso broadcast approach perfect channel state information csi is assumed known at the receiver end only this strategy facilitates to adapt the reliably decoded rate to the actual channel state without having any feedback link to the transmitter transmission of layered coded information is motivated by the theory of majorization we derive the basic equations characterizing achievable rates of the strategy several ad hoc approximations to the achievable region are considered and their performance is compared with the siso setting and the ergodic capacity it has been demonstrated that a single layer outage approach is reasonably efficient in the mimo setting in terms of the average reliably decoded rate a multiple access channel mac broadcast approach is also applied for the mimo case and demonstrated to be relatively efficient

combinatorial message sharing and a new achievable region for multiple descriptions this paper presents a new achievable rate distortion region for the general l channel multiple descriptions mds problem a well known general region for this problem is due to venkataramani kramer and goyal vkg their encoding scheme is an extension of the el gamal cover ec and zhang berger zb coding schemes to the l channel case and includes a combinatorial number of refinement codebooks one for each subset of the descriptions as in zb the scheme also allows for a single common codeword to be shared by all descriptions this paper proposes a novel encoding technique involving combinatorial message sharing cms where every subset of the descriptions may share a distinct common message this introduces a combinatorial number of shared codebooks along with the refinement codebooks of these shared codebooks provide a more flexible framework to tradeoff redundancy across the messages for resilience to descriptions loss we derive an achievable rate distortion region for the proposed technique and show that it subsumes the vkg region for general sources and distortion measures we further show that cms provides a strict improvement of the achievable region for any source and distortion measures for which some two description subset is such that zb achieves points outside the ec region we then show a more surprising result cms outperforms vkg for a general class of sources and distortion measures including scenarios where the zb and ec regions coincide for all two description subsets in particular we show that cms strictly improves on vkg for the l channel quadratic gaussian md problem for all l geq 3 despite the fact that the ec region is complete for the corresponding two descriptions problem consequently the correlated quantization scheme an extreme special case of vkg that has been proven to be optimal for several cross sections of the l channel quadratic gaussian md problem is strictly suboptimal in general using the encoding principles derived we show that the cms scheme achieves the complete rate distortion region for several asymmetric cross sections of the l channel quadratic gaussian md problem

the progressive transmission disadvantage the disadvantage of progressive transmission compared to transmission in a single step is analyzed for the high rate case using a polytope that is self similar under binary subdivision this disadvantage is calculated to be 0 3424 db or 0 057 bit compared to scalar quantization obtained with a step size of 1 bit per ensemble

on two stage sequential coding of correlated sources we study the problem of two stage sequential coding tssc which is an extension of sequential coding of correlated sources let x and y be dependent random variables the network contains two encoders and two decoders 1 a y encoder with input y 2 an x encoder with inputs x and y 3 a y decoder that reconstructs y and 4 an x decoder that reconstructs x the first stage is traditional sequential coding where the y encoder describes y to both the x decoder and y decoder and the x encoder describes x and y to the x decoder at the second stage the y encoder refines the description of y and the x encoder refines the description of x the tssc model is a theoretical abstraction of scalable video coding here y and x represent successive frames of a video sequence and the two stages together give an embedded description that allows the video to be decoded at two distinct rates we give an inner bound on the rate distortion region for this tssc model the tight bound on the rate distortion region is derived when y must be reconstructed losslessly in the usual shannon sense in the second stage we also study the minimum total rate of the tssc model and show that the minimum total rate of one stage sequential coding cannot be achieved at both stages for jointly gaussian sources this theoretical result can shed light on the rate distortion performance behavior of scalable video coding widely noted by practitioners

n channel symmetric multiple descriptions part i n k source channel erasure codes in this two part paper we present a new achievable rate region for the general n channel symmetric multiple descriptions problem in part i inspired by the concept of maximum distance separable mds erasure channel codes we consider a special case of this rate region where the source is encoded into n descriptions each with rate r these descriptions are transmitted over n bandwidth constrained and errorless channels during transmission a subset of these channels can break down thus erasing the corresponding descriptions the decoder is interested in recovering the source with the reception of at least k descriptions thus the encoder is allowed to sample only one realization of this breakdown process during the entire transmission for gaussian sources we have the following interesting result when any k descriptions arrive the achievable distortion exactly matches the optimal distortion rate performance corresponding to a source rate of kr bits with the reception of any m k descriptions the source reconstruction quality is strictly better the improvement being nearly linear in the number of descriptions received

joint source channel coding of a gaussian mixture source over the gaussian broadcast channel suppose that we want to send a description of a single source to two listeners through a gaussian broadcast channel where the channel is used once per source sample the problem of joint source channel coding is to design a communication system to minimize the distortion d sub 1 at receiver 1 and at the same time minimize the distortion d sub 2 at receiver 2 if the source is gaussian the optimal solution is well known and it is achieved by an uncoded analog scheme we consider a gaussian mixture source we derive inner and outer bounds for the distortion region of all d sub 1 d sub 2 pairs that are simultaneously achievable the outer bound is based on the entropy power inequality while the inner bound is attained by a digital over analog encoding scheme which we present we also show that if the modes of the gaussian mixture are highly separated our bounds are tight and hence our scheme attains the entire distortion region this optimal region exceeds the region attained by separating source and channel coding although it does not contain the ideal point d sub 1 d sub 2 r sup 1 c sub 1 r sup 1 c sub 2

on the role of the refinement layer in multiple description coding and scalable coding we clarify the relationship among several existing achievable multiple description rate distortion regions by investigating the role of refinement layer in multiple description coding specifically we show that the refinement layer in the el gamal cover egc scheme and the venkataramani kramer goyal vkg scheme can be removed as a consequence the egc region is equivalent to the egc region an antecedent version of the egc region while the vkg region when specialized to the 2 description case is equivalent to the zhang berger zb region moreover we prove that for multiple description coding with individual and hierarchical distortion constraints the number of layers in the vkg scheme can be significantly reduced when only certain weighted sum rates are concerned the role of refinement layer in scalable coding a special case of multiple description coding is also studied

successive wyner ziv coding scheme and its application to the quadratic gaussian ceo problem in this paper we introduce a distributed source coding scheme called successive wyner ziv coding we show that every point in the rate region of the quadratic gaussian ceo problem can be achieved via successive wyner ziv coding the concept of successive refinement in single source coding is generalized to the distributed source coding scenario which we refer to as distributed successive refinement for the quadratic gaussian ceo problem we establish a necessary and sufficient condition for distributed successive refinement where the successive wyner ziv coding scheme plays an important role

the generalized random energy model and its application to the statistical physics of ensembles of hierarchical codes in an earlier work the statistical physics associated with finite temperature decoding of code ensembles along with the relation to their random coding error exponents were explored in a framework that is analogous to derrida s random energy model rem of spin glasses according to which the energy levels of the various spin configurations are independent random variables the generalized rem grem extends the rem in that it introduces correlations between energy levels in an hierarchical structure in this paper we explore some analogies between the behavior of the grem and that of code ensembles which have parallel hierarchical structures in particular in analogy to the fact that the grem may have different types of phase transition effects depending on the parameters of the model then the above mentioned hierarchical code ensembles behave substantially differently in the various domains of the design parameters of these codes we make an attempt to explore the insights that can be imported from the statistical mechanics of the grem and be harnessed to serve for code design considerations and guidelines

side information scalable source coding we consider the problem of side information scalable si scalable source coding where the encoder constructs a two layer description such that the receiver with high quality side information will be able to use only the first layer to reconstruct the source in a lossy manner while the receiver with low quality side information will have to receive both layers in order to decode we provide inner and outer bounds to the rate distortion r d region for general discrete memoryless sources the achievable region is tight when either one of the decoders requires a lossless reconstruction and when the distortion measures are degraded and deterministic furthermore the gap between the inner and the outer bounds can be bounded by certain constants when the squared error distortion measure is used the notion of perfect scalability is introduced for which necessary and sufficient conditions are given for sources satisfying a mild support condition using si scalable coding and successive refinement wyner ziv coding as basic building blocks we provide a complete characterization of the rate distortion region for the important quadratic gaussian source with multiple jointly gaussian side informations where the side information quality is not necessarily monotonic along the scalable coding order a partial result is provided for the doubly symmetric binary source under the hamming distortion measure when the worse side information is a constant for which one of the outer bounds is strictly tighter than the other

the analysis and design of windowed fourier frame based multiple description source coding schemes in this paper the windowed fourier encoding decoding scheme applied to the multiple description compression problem is analyzed in the general case four window functions are needed to define the encoder and decoder although this number can be reduced to three or two by using time shift or frequency shift division schemes the encoding coefficients are next divided into two groups according to the eveness of either the modulation or translation index the distortion on each channel is analyzed using the zak transform for the optimal windows explicit representation formulas are obtained and nonlocalization results are proved asymptotic formulas of the total distortion and transmission rate are established and the redundancy is shown to trade off between these two