approximately optimal solutions to the finite dimensional witsenhausen counterexample recently a vector version of witsenhausen s counterexample was considered and it was shown that in the asymptotic limit of infinite vector length certain vector quantization based control strategies are provably within a constant factor of the asymptotically optimal cost for all possible problem parameters while suggestive a constant factor result for the finite dimensional problem has remained elusive in this paper we provide a resolution to this issue by applying a large deviation sphere packing philosophy we derive a lower bound to the optimal cost for the finite dimensional case that uses appropriate shadows of an existing vector lower bound that is the same for all dimensions using this new lower bound we show that good lattice quantization based control strategies achieve within a constant factor of the optimal cost uniformly over all possible problem parameters including the vector length for witsenhausen s original problem which is the scalar case the gap between regular lattice quantization based strategies and the lower bound is provably never more than a factor of 100 and computer calculations strongly suggest that the factor in fact may be no larger than 8 finally to obtain a numerical understanding of the possible room for improvement in costs using alternative strategies we also include numerical comparison with strategies that are conjectured to be optimal using this comparison we posit that there is more room for improvement in our lower bounds than in our upper bounds

a distributed newton method for network utility maximization i algorithm most existing works use dual decomposition and first order methods to solve network utility maximization num problems in a distributed manner which suffer from slow rate of convergence properties this paper develops an alternative distributed newton type fast converging algorithm for solving num problems by using novel matrix splitting techniques both primal and dual updates for the newton step can be computed using iterative schemes in a decentralized manner we propose a stepsize rule and provide a distributed procedure to compute it in finitely many iterations the key feature of our direction and stepsize computation schemes is that both are implemented using the same distributed information exchange mechanism employed by first order methods we describe the details of the inexact algorithm here and in part ii of this paper we show that under some assumptions even when the newton direction and the stepsize in our method are computed within some error due to finite truncation of the iterative schemes the resulting objective function value still converges superlinearly in terms of primal iterations to an explicitly characterized error neighborhood simulation results demonstrate significant convergence rate improvement of our algorithm relative to the existing first order methods based on dual decomposition

why do block length and delay behave differently if feedback is present for output symmetric discrete memoryless channels dmcs at even moderately high rates fixed block length communication systems show no improvements in their error exponents with feedback this paper studies systems with fixed end to end delay and shows that feedback generally provides dramatic gains in the error exponents a new upper bound the uncertainty focusing bound is given on the probability of symbol error in a fixed delay communication system with feedback this bound turns out to have a form similar to viterbi s bound used for the block error probability of convolutional codes as a function of the fixed constraint length the uncertainty focusing bound is shown to be asymptotically achievable with noiseless feedback for erasure channels as well as for any output symmetric dmc that has strictly positive zero error capacity furthermore it can be achieved in a delay universal anytime fashion even if the feedback itself is delayed by a small amount finally it is shown that for end to end delay it is generally possible at high rates to beat the sphere packing bound for general dmcs thereby providing a counterexample to a conjecture of pinsker

a zero delay sequential scheme for lossy coding of individual sequences we consider adaptive sequential lossy coding of bounded individual sequences when the performance is measured by the sequentially accumulated mean squared distortion the encoder and the decoder are connected via a noiseless channel of capacity r and both are assumed to have zero delay no probabilistic assumptions are made on how the sequence to be encoded is generated for any bounded sequence of length n the distortion redundancy is defined as the normalized cumulative distortion of the sequential scheme minus the normalized cumulative distortion of the best scalar quantizer of rate r which is matched to this particular sequence we demonstrate the existence of a zero delay sequential scheme which uses common randomization in the encoder and the decoder such that the normalized maximum distortion redundancy converges to zero at a rate n sup 1 5 log n as the length of the encoded sequence n increases without bound

second order and moderate deviations asymptotics for successive refinement we derive the optimal second order coding region and moderate deviations constant for successive refinement source coding with a joint excess distortion probability constraint we consider two scenarios 1 a discrete memoryless source dms and arbitrary distortion measures at the decoders and 2 a gaussian memoryless source gms and quadratic distortion measures at the decoders for a dms with arbitrary distortion measures we prove an achievable second order coding region using type covering lemmas by kanlis and narayan and by no ingber and weissman we prove the converse using the perturbation approach by gu and effros when the dms is successively refinable the expressions for the second order coding region and the moderate deviations constant are simplified and easily computable for this case we also obtain new insights on the second order behavior compared with the scenario where separate excess distortion proabilities are considered for example we describe a dms for which the optimal second order region transitions from being characterizable by a bivariate gaussian to a univariate gaussian as the distortion levels are varied we then consider a gms with quadratic distortion measures to prove the direct part we make use of the sphere covering theorem by verger gaugry together with appropriately defined gaussian type classes to prove the converse we generalize kostina and verdu s one shot converse bound for point to point lossy source coding we remark that this proof is applicable to general successively refinable sources in the proofs of the moderate deviations results for both scenarios we follow a strategy similar to that for the second order asymptotics and use the moderate deviations principle

exact moderate deviation asymptotics in streaming data transmission in this paper a streaming transmission setup is considered where an encoder observes a new message in the beginning of each block and a decoder sequentially decodes each message after a delay of t blocks in this streaming setup the fundamental interplay between the coding rate the error probability and the blocklength in the moderate deviations regime is studied for output symmetric channels the moderate deviations constant is shown to improve over the block coding or non streaming setup by exactly a factor of t for a certain range of moderate deviations scalings for the converse proof a more powerful decoder to which some extra information is fedforward is assumed the error probability is bounded first for an auxiliary channel and this result is translated back to the original channel by using a newly developed change of measure lemma where the speed of decay of the remainder term in the exponent is carefully characterized for the achievability proof a known coding technique that involves a joint encoding and decoding of fresh and past messages is applied with some manipulations in the error analysis

uniform random number generation from markov chains non asymptotic and asymptotic analyses in this paper we derive non asymptotic achievability and converse bounds on the random number generation with without side information our bounds are efficiently computable in the sense that the computational complexity does not depend on the block length we also characterize the asymptotic behaviors of the large deviation regime and the moderate deviation regime by using our bounds which implies that our bounds are asymptotically tight in those regimes we also show the second order rates of those problems and derive single letter forms of the variances characterizing the second order rates furthermore we address the relative entropy rate and the modified mutual information rate for these problems

on optimal finite state digital transmission systems the digital transmission of signals by transmitters and receivers that are time invariant finite state machines were investigated in general form an average distortion criterion is used some general structure theorems are proved and some examples given that show improvement in performance as the memories of the transmitter and receiver are increased many conjectures and unsolved problems are mentioned and areas for further study are indicated

source coding with limited look ahead side information at the decoder we characterize the rate distortion function for the source coding with decoder side information setting when the ith reconstruction symbol is allowed to depend only on the first i lscr side information symbols for some finite look ahead lscr in addition to the index from the encoder for the case of causal side information i e lscr 0 we find that the penalty of causality is the omission of the subtracted mutual information term in the wyner ziv rate distortion function for lscr 0 we derive a computable infinite letter expression for the rate distortion function when specialized to the near lossless case our results characterize the best achievable rate for the slepian wolf source coding problem with finite side information looka head and have some surprising implications we find that side information is useless for any fixed lscr when the joint probability mass function pmf of the source and side information satisfies the positivity condition p x y 0 for all x y more generally the optimal rate depends on the distribution of the pair x y only through the distribution of x and the bipartite graph whose edges represent the pairs x y for which p x y 0 on the other hand if side information look ahead is allowed to grow faster than logarithmic in the block length then h x y is achievable finally we apply our approach to derive a computable expression for channel capacity when state information is available at the encoder with limited look ahead

distributed source coding using syndromes discus design and construction we address the problem of compressing correlated distributed sources i e correlated sources which are not co located or which cannot cooperate to directly exploit their correlation we consider the related problem of compressing a source which is correlated with another source that is available only at the decoder this problem has been studied in the information theory literature under the name of the slepian wolf 1973 source coding problem for the lossless coding case and as rate distortion with side information for the lossy coding case we provide a constructive practical framework based on algebraic trellis codes dubbed as distributed source coding using syndromes discus that can be applicable in a variety of settings simulation results are presented for source coding of independent and identically distributed i i d gaussian sources with side information available at the decoder in the form of a noisy version of the source to be coded our results reveal the promise of this approach using trellis based quantization and coset construction the performance of the proposed approach is 2 5 db from the wyner ziv 1976 bound

source coding and graph entropies a sender wants to accurately convey information to a receiver who has some possibly related data we study the expected number of bits the sender must transmit for one and for multiple instances in two communication scenarios and relate this number to the chromatic and korner 1973 entropies of a naturally defined graph

the method of types information theory the method of types is one of the key technical tools in shannon theory and this tool is valuable also in other fields in this paper some key applications are presented in sufficient detail enabling an interested nonspecialist to gain a working knowledge of the method and a wide selection of further applications are surveyed these range from hypothesis testing and large deviations theory through error exponents for discrete memoryless channels and capacity of arbitrarily varying channels to multiuser problems while the method of types is suitable primarily for discrete memoryless models its extensions to certain models with memory are also discussed

universal multiterminal source coding algorithms with asymptotically zero feedback fixed database case consider a source network in which a finite alphabet source x xi i 0 infin is to be encoded and transmitted and another finite alphabet source y xi i 0 infin correlated with x is available only to the decoder as side information traditionally the channel between the encoder and decoder in the source network is assumed to be one way this together with the fact that the encoder does not have access to y implies that the encoder has to know the achievable rates before encoding in this paper we consider universal source coding for a feedback source network in which the channel between the encoder and decoder is two way and asymmetric assume that the encoder and decoder share a random database that is independent of both x and y a string matching based variable rate block coding algorithm with simple progressive encoding and joint typicality decoding is first proposed for the feedback source network the simple progressive encoder does not need to know the achievable rates at the beginning of encoding it is proven that for any x y in a large class of sources satisfying some mixing conditions the average number of bits per letter transmitted from the encoder to the decoder compression rate goes to the conditional entropy h x y of x given y asymptotically and at the same time the average number of bits per letter transmitted from the decoder to the encoder feedback rate goes to 0 asymptotically the algorithm and the corresponding analysis results are then extended to the case where both x and y are to be encoded separately but decoded jointly finally a universal decoding algorithm is proposed to replace the joint typicality decoding and the resulting universal compression algorithm consisting of the simple progressive encoder and the universal decoding algorithm is further shown to be asymptotically optimal for the class of all jointly memoryless source side information pairs x y

recurrent multiple repetition coding for channels with feedback we consider multiple repetition strategies with fixed delay decoding for discrete memoryless channels with noiseless feedback existing binary schemes by schalkwijk and zigangirov are analyzed and their results are extended the general error exponents are computed and presented by elegant expressions in the strictly symmetric case an important class of precoded sequences so called flip sequences is found and their degrading effect on the error exponent is investigated this effect is shown negligible when the repetition parameters are chosen such that the transmission rate is maximized even when signalling at channel capacity the error exponent is shown to be strictly positive

convergence of limited communication gradient methods distributed optimization increasingly plays a central role in economical and sustainable operation of cyber physical systems nevertheless the complete potential of the technology has not yet been fully exploited in practice due to communication limitations posed by the real world infrastructures this work investigates fundamental properties of distributed optimization based on gradient methods where gradient information is communicated using a limited number of bits in particular a general class of quantized gradient methods are studied where the gradient direction is approximated by a finite quantization set sufficient and necessary conditions are provided on such a quantization set to guarantee that the methods minimize any convex objective function with lipschitz continuous gradient and a nonempty and bounded set of optimizers a lower bound on the cardinality of the quantization set is provided along with specific examples of minimal quantizations convergence rate results are established that connect the fineness of the quantization and the number of iterations needed to reach a predefined solution accuracy generalizations of the results to a relevant class of constrained problems using projections are considered finally the results are illustrated by simulations of practical systems

a comment on a rate of convergence result for a universal d semifaithful code in the above paper see ibid vol 39 no 3 p 813 20 1993 yu and speed propose a universal pointwise d semifaithful code whose expected compression ratio for discrete memoryless sources approaches the rate distortion function at a rate o n sup 1 log n they also conjecture that this is the fastest achievable convergence rate for pointwise d semifaithful codes in this correspondence we use a simple extension of kraft s inequality and prove that this conjecture is true at least for the hamming distortion measure

anytime characteristics of protograph based ldpc convolutional codes anytime transmission has been shown to be an effective means of stabilizing an unstable system over noisy channels a crucial issue in implementing anytime transmission strategies is the design of anytime codes that offer good finite length performance and asymptotic anytime properties this paper introduces an efficient anytime code derived from protograph based low density parity check convolutional codes known as extended s ldpcc xs ldpcc codes the asymptotic anytime properties of xs ldpcc codes over both the binary erasure channel bec and the binary input additive white gaussian noise biawgn channel are demonstrated through density evolution by means of certain observations verified by simulations we provide accurate estimates of the delay exponents of xs ldpcc codes over the bec and the biawgn channel through simulations the finite length performance of xs ldpcc codes and the effects of different code design parameters are quantified

the error exponent of variable length codes over markov channels with feedback the error exponent of markov channels with feedback is studied in the variable length block coding setting burnashev s classic result is extended to finite state ergodic markov channels for these channels a single letter characterization of the reliability function is presented under the assumption of full causal output feedback and full causal observation of the channel state both at the transmitter and at the receiver side tools from stochastic control theory are used in order to treat channels with intersymbol interference isi specifically the convex analytic approach to markov decision processes is adopted in order to handle problems with stopping time horizons induced by variable length coding schemes

optimal causal coding decoding problems the symbols produced by a finite markov source are causally encoded so as to be transmitted through a noisy memoryless channel the encoder is assumed to have channel feedback information and the decoder to be causal the feedback information is shown to be useful in general separation results are derived and used to prove that encoding is useless for a class of symmetric channels

scalar estimation and control with noisy binary observations we consider a simple one dimensional system whose observations are sent to a state estimator over a noisy binary communication link the interesting thing about the system is that it is unstable the problem is to design an encoding scheme and a decoder such that the estimation error is stable we explicitly construct a simple but efficient estimator for the binary symmetric channel bsc we are not aware of any such previous codes for the bsc we compare our results to the nonconstructive bounds of sahai

variations on a theme by schalkwijk and kailath schalkwijk and kailath 1966 developed a class of block codes for gaussian channels with ideal feedback for which the probability of decoding error decreases as a second order exponent in block length for rates below capacity this well known but surprising result is explained and simply derived here in terms of a result by elias 1956 concerning the minimum mean square distortion achievable in transmitting a single gaussian random variable over multiple uses of the same gaussian channel a simple modification of the schalkwijk kailath scheme is then shown to have an error probability that decreases with an exponential order which is linearly increasing with block length in the infinite bandwidth limit this scheme produces zero error probability using bounded expected energy at all rates below capacity a lower bound on error probability for the finite bandwidth case is then derived in which the error probability decreases with an exponential order which is linearly increasing in block length at the same rate as the upper bound

moderate deviations in channel coding we consider block codes whose rate converges to the channel capacity with increasing blocklength at a certain speed and examine the best possible decay of the probability of error for discrete memoryless channels we prove that a moderate deviation principle holds for all convergence rates between the large deviation and the central limit theorem regimes

a distributed newton method for network utility maximization part ii convergence the existing distributed algorithms for network utility maximization num problems are mostly constructed using dual decomposition and first order gradient or subgradient methods which suffer from a slow rate of convergence part i of this paper proposed an alternative distributed newton type algorithm for solving num problems with self concordant utility functions for each primal iteration this algorithm features distributed exact stepsize calculation with finite termination and decentralized computation of the dual variables using a finitely truncated iterative scheme obtained through novel matrix splitting techniques this paper analyzes the convergence properties of a broader class of algorithms with potentially different stepsize computation schemes in particular we allow for errors in the stepsize computation we show that if the error levels in the newton direction resulting from finite termination of dual iterations and stepsize calculation are below a certain threshold then the algorithm achieves local quadratic convergence rate in primal iterations to an error neighborhood of the optimal solution where the size of the neighborhood can be explicitly characterized by the parameters of the algorithm and the error levels

discrete lossy gray wyner revisited second order asymptotics large and moderate deviations in this paper we revisit the discrete lossy gray wyner problem in particular we derive its optimal second order coding rate region its error exponent reliability function and its moderate deviations constant under mild conditions on the source to obtain the second order asymptotics we extend some ideas from watanabe s work in particular we leverage the properties of an appropriate generalization of the conditional distortion tilted information density which was first introduced by kostina and verdu the converse part uses a perturbation argument by gu and effros in their strong converse proof of the discrete gray wyner problem the achievability part uses two novel elements 1 a generalization of various type covering lemmas and 2 the uniform continuity of the conditional rate distortion function in both the source joint distribution and the distortion level to obtain the error exponent for the achievability part we use the same generalized type covering lemma and for the converse we use the strong converse together with a change of measure technique finally to obtain the moderate deviations constant we apply the moderate deviations theorem to probabilities defined in terms of information spectrum quantities

moderate deviation asymptotics for variable length codes with feedback we consider data transmission across discrete memoryless channels dmcs using variable length codes with feedback we consider the family of such codes whose rates are rho n below the channel capacity c where rho n is a positive sequence that tends to zero slower than the reciprocal of the square root of the expectation of the random blocklength n this is known as the moderate deviations regime and we establish the optimal moderate deviations constant we show that in this scenario the error probability decays sub exponentially with speed exp b c n rho n where b is the maximum relative entropy between output distributions of the dmc

an information spectrum approach to weak variable length source coding with side information this paper studies variable length vl source coding of general sources with side information novel one shot coding bounds for slepian wolf sw coding which give nonasymptotic tradeoff between the error probability and the codeword length of vl sw coding are established one shot results are applied to asymptotic analysis and a general formula for the optimal coding rate achievable by weakly lossless vl sw coding i e vl sw coding with vanishing error probability is derived our general formula reveals how the encoder side information and or vl coding improve the optimal coding rate in the general setting in addition it is shown that if the encoder side information is useless in weakly lossless vl coding then it is also useless even in the case where the error probability may be positive asymptotically

hidden markov processes an overview of statistical and information theoretic aspects of hidden markov processes hmps is presented an hmp is a discrete time finite state homogeneous markov chain observed through a discrete time memoryless invariant channel in recent years the work of baum and petrie 1966 on finite state finite alphabet hmps was expanded to hmps with finite as well as continuous state spaces and a general alphabet in particular statistical properties and ergodic theorems for relative entropy densities of hmps were developed consistency and asymptotic normality of the maximum likelihood ml parameter estimator were proved under some mild conditions similar results were established for switching autoregressive processes these processes generalize hmps new algorithms were developed for estimating the state parameter and order of an hmp for universal coding and classification of hmps and for universal decoding of hidden markov channels these and other related topics are reviewed

real time coding with limited lookahead a real time coding system with lookahead consists of a memoryless source a memoryless channel an encoder which encodes the source symbols sequentially with knowledge of future source symbols up to a fixed finite lookahead d with or without feedback of the past channel output symbols and a decoder which sequentially constructs the source symbols using the channel output the objective is to minimize the expected per symbol distortion for a fixed finite lookahead d 1 we invoke the theory of controlled markov chains to obtain an average cost optimality equation acoe the solution of which denoted by d d is the minimum expected per symbol distortion with increasing d d d bridges the gap between causal encoding d 0 where symbol by symbol encoding decoding is optimal and the infinite lookahead case d where shannon theoretic arguments show that separation is optimal we extend the analysis to a system with finite state decoders with or without noise free feedback for a bernoulli source and binary symmetric channel under hamming loss we compute the optimal distortion for various source and channel parameters and thus obtain computable bounds on d d we also identify regions of source and channel parameters where symbol by symbol encoding decoding is suboptimal finally we demonstrate the wide applicability of our approach by applying it in additional coding scenarios such as the case where the sequential decoder can take cost constrained actions affecting the quality or availability of side information about the source

on the dispersions of three network information theory problems we analyze the dispersions of distributed lossless source coding the slepian wolf problem the multiple access channel and the asymmetric broadcast channel for the two encoder slepian wolf problem we introduce a quantity known as the entropy dispersion matrix which is analogous to the scalar dispersions that have gained interest recently we prove a global dispersion result that can be expressed in terms of this entropy dispersion matrix and provides intuition on the approximate rate losses at a given blocklength and error probability to gain better intuition about the rate at which the nonasymptotic rate region converges to the slepian wolf boundary we define and characterize two operational dispersions 1 the local dispersion and 2 the weighted sum rate dispersion the former represents the rate of convergence to a point on the slepian wolf boundary whereas the latter represents the fastest rate for which a weighted sum of the two rates converges to its asymptotic fundamental limit interestingly when we approach either of the two corner points the local dispersion is characterized not by a univariate gaussian but a bivariate one as well as a subset of off diagonal elements of the aforementioned entropy dispersion matrix finally we demonstrate the versatility of our achievability proof technique by providing inner bounds for the multiple access channel and the asymmetric broadcast channel in terms of dispersion matrices all our proofs are unified by a so called vector rate redundancy theorem which is proved using the multidimensional berry esseen theorem

on haroutunian s exponent for parallel channels and an application to fixed delay codes without feedback the haroutunian exponent arises in the study of channel reliability functions for both block coding with feedback and fixed delay coding without feedback for asymmetric channels such as the z channel the haroutunian exponent is strictly larger than the sphere packing exponent the spherepacking exponent is believed to be an upper bound for the reliability function in the two aforementioned communication problems but in attempting to prove this one gets stuck at the haroutunian exponent because of entanglements between the channel behavior and the input distribution in this paper we present a characteristic of the haroutunian exponent that differentiates it from the random coding and sphere packing exponents we consider the parallel channel the repeated use of the original discrete memoryless channel independently some number of times it is well known that the capacity of the parallel channel is l times the capacity of the original channel and the random coding and sphere packing exponents of the l use parallel channel decompose into l times the exponents of the original channel the main result of this paper is to show that the appropriately normalized haroutunian exponent of the parallel channel asymptotically decomposes to the sphere packing exponent of the original channel as opposed to the haroutunian exponent of the original channel this fact is then used to prove two results first an upper bound for the reliability function for fixed blocklength coding with delayed feedback is proved this upper bound converges to the sphere packing exponent as the delay in the feedback path tends to infinity second the reliability function for fixed delay coding without feedback is shown to be upper bounded by the sphere packing exponent

a dichotomy of functions in distributed coding an information spectral approach the problem of distributed data compression for function computation is considered where 1 the function to be computed is not necessarily symbolwise function and 2 the information source has memory and may not be stationary nor ergodic we introduce the class of smooth sources and give a sufficient condition on functions so that the achievable rate region for computing coincides with the slepian wolf region i e the rate region for reproducing the entire source for any smooth sources moreover for symbolwise functions the necessary and sufficient condition for the coincidence is established our result for the full side information case is a generalization of the result by ahlswede and csiszar to sources with memory our dichotomy theorem is different from han and kobayashi s dichotomy theorem which reveals an effect of memory in distributed function computation all results are given not only for fixed length coding but also for variable length coding in a unified manner furthermore for the full side information case the error probability in the moderate deviation regime is also investigated

markov properties for linear causal models with correlated errors a linear causal model with correlated errors represented by a dag with bi directed edges can be tested by the set of conditional independence relations implied by the model a global markov property specifies by the d separation criterion the set of all conditional independence relations holding in any model associated with a graph a local markov property specifies a much smaller set of conditional independence relations which will imply all other conditional independence relations which hold under the global markov property for dags with bi directed edges associated with arbitrary probability distributions a local markov property is given in richardson 2003 which may invoke an exponential number of conditional independencies in this paper we show that for a class of linear structural equation models with correlated errors there is a local markov property which will invoke only a linear number of conditional independence relations for general linear models we provide a local markov property that often invokes far fewer conditional independencies than that in richardson 2003 the results have applications in testing linear structural equation models with correlated errors

rateless codes for avc models the arbitrarily varying channel avc is a channel model whose state is selected maliciously by an adversary fixed blocklength coding assumes a worst case bound on the adversary s capabilities which leads to pessimistic results this paper defines a variable length perspective on this problem for which achievable rates are shown that depend on the realized actions of the adversary specifically rateless codes are constructed which require a limited amount of common randomness these codes are constructed for two kinds of avc models in the first the channel state cannot depend on the channel input and in the second it can as a by product the randomized coding capacity of the avc with state depending on the transmitted codeword is found and shown to be achievable with a small amount of common randomness the results for this model are proved using a randomized strategy based on list decoding

optimal zero delay coding of markov sources stationary and finite memory codes the optimal zero delay coding of a finite state markov source is considered the existence and structure of optimal codes are studied using a stochastic control formulation prior results in the literature established the optimality of deterministic markov walrand varaiya type coding policies for the finite time horizon problem and the optimality of both deterministic nonstationary and randomized stationary policies for the infinite time horizon problem our main result here shows that for any irreducible and aperiodic markov source with a finite alphabet deterministic and stationary markov coding policies are optimal for the infinite horizon problem in addition the finite block length time horizon performance of an optimal stationary and markov coding policy is shown to approach the infinite time horizon optimum at a rate o 1 t the results are extended to systems where zero delay communication takes place across a noisy channel with noiseless feedback

lossless coding for distributed streaming sources distributed source coding is traditionally viewed in a block coding context wherein all source symbols are known in advance by the encoders however many modern applications to which distributed source coding ideas are applied are better modeled as having streaming data in a streaming setting source symbol pairs are revealed to separate encoders in real time and need to be reconstructed at the decoder with subject to some tolerable end to end delay in this paper a causal sequential random binning encoder is introduced and paired with maximum likelihood ml and universal decoders the latter uses a novel weighted empirical suffix entropy decoding rule we derive a lower bounds on the error exponent with delay for each decoder we also provide upper bounds for the special case of streaming with decoder side information and discuss when upper and lower bounds match we show that both ml and universal decoders achieve the same positive error exponents for all rate pairs inside the slepian wolf achievable rate region the dominant error events in streaming are different from those in block coding and result in different exponents because the sequential random binning scheme is also universal over delays the resulting code eventually reconstructs every source symbol correctly with probability one

the streaming dmt of fading channels we study sequential transmission of a stream of messages over a block fading multi input multi output mimo channel a new message arrives in each coherence block and the decoder is required to output each message after a delay of t coherence blocks we establish the optimal diversity multiplexing tradeoff dmt and show that it can be achieved using a simple interleaving of messages the converse is based on an outage amplification technique which appears to be new we also discuss another coding scheme based on a sequential tree code this coding scheme only requires the knowledge of delay at the decoder and yet realizes the optimal dmt we finally discuss some extensions when multiple messages at uniform intervals arrive within each coherence period

on causal source codes with side information we study the effect of the introduction of side information into the causal source coding setting of neuhoff and gilbert we find that the spirit of their result namely the sufficiency of time sharing scalar quantizers followed by appropriate lossless coding for attaining optimum performance within the family of causal source codes extends to many scenarios involving availability of side information at both encoder and decoder or only on one side for example in the case where side information is available at both encoder and decoder we find that time sharing side information dependent scalar quantizers at most two for each side information symbol attains optimum performance this remains true even when the reproduction sequence is allowed noncausal dependence on the side information and even for the case where the source and the side information rather than consisting of independent and identically distributed i i d pairs form respectively the output of a memoryless channel and its stationary ergodic input

feedback in the non asymptotic regime without feedback the backoff from capacity due to non asymptotic blocklength can be quite substantial for blocklengths and error probabilities of interest in many practical applications in this paper novel achievability bounds are used to demonstrate that in the non asymptotic regime the maximal achievable rate improves dramatically thanks to variable length coding and feedback for example for the binary symmetric channel with capacity 1 2 the blocklength required to achieve 90 of the capacity is smaller than 200 compared to at least 3100 for the best fixed blocklength code even with noiseless feedback virtually all the advantages of noiseless feedback are shown to be achievable even if the feedback link is used only to send a single signal informing the encoder to terminate the transmission stop feedback it is demonstrated that the non asymptotic behavior of the fundamental limit depends crucially on the particular model chosen for the end of packet control signal fixed blocklength codes and related questions concerning communicating with a guaranteed delay are discussed in which situation feedback is demonstrated to be almost useless even non asymptotically

computing maximum likelihood estimates in recursive linear models with correlated errors in recursive linear models the multivariate normal joint distribution of all variables exhibits a dependence structure induced by a recursive or acyclic system of linear structural equations these linear models have a long tradition and appear in seemingly unrelated regressions structural equation modelling and approaches to causal inference they are also related to gaussian graphical models via a classical representation known as a path diagram despite the models long history a number of problems remain open in this paper we address the problem of computing maximum likelihood estimates in the subclass of bow free recursive linear models the term bow free refers to the condition that the errors for variables i and j be uncorrelated if variable i occurs in the structural equation for variable j we introduce a new algorithm termed residual iterative conditional fitting ricf that can be implemented using only least squares computations in contrast to existing algorithms ricf has clear convergence properties and yields exact maximum likelihood estimates after the first iteration whenever the mle is available in closed form

variable length source dispersions differ under maximum and average error criteria variable length compression without prefix free constraints and with side information available at both encoder and decoder is considered instead of requiring the code to be error free we allow for it to have a non vanishing error probability we derive one shot bounds on the optimal average codeword length by proposing two new information quantities namely the conditional and unconditional varepsilon cutoff entropies using these one shot bounds we obtain the second order asymptotics of the problem under two different formalisms the average and maximum probabilities of error over the realization of the side information while the first order terms in the asymptotic expansions for both formalisms are identical we find that the source dispersion under the average error formalism is in most cases strictly smaller than its maximum error counterpart applications to a certain class of guessing problems previously studied by kuzuoka 2020 are also discussed

optimum tradeoffs between the error exponent and the excess rate exponent of variable rate slepian wolf coding we analyze the optimal tradeoff between the error exponent and the excess rate exponent for variable rate slepian wolf codes in particular we first derive upper converse bounds on the optimal error and excess rate exponents and then lower achievable bounds via a simple class of variable rate codes which assign the same rate to all source blocks of the same type class then using the exponent bounds we derive bounds on the optimal rate functions namely the minimal rate assigned to each type class needed in order to achieve a given target error exponent the resulting excess rate exponent is then evaluated iterative algorithms are provided for the computation of both bounds on the optimal rate functions and their excess rate exponents the resulting slepian wolf codes bridge between the two extremes of fixed rate coding which has minimal error exponent and maximal excess rate exponent and average rate coding which has maximal error exponent and minimal excess rate exponent

on the error exponent of convolutionally coded arq forney s 1968 exponent is effectively the best one for the block coded automatic repeat request systems arq and it has been conjectured that a counterpart of forney s exponent is attainable by convolutional codes with reasonable decoder complexity in the paper it is show that the scheme of yamamoto and itoh 1980 which is known to be one of the simplest convolutionally coded arq schemes actually gives a better performance than ever expected and that it has an error exponent quite close to the conjectured counterpart of forney s exponent at relatively low rates it is further shown that the likelihood ratio test used by yamamoto and itoh gives forney s exponent in the case of block coded arq

a simple class of capacity achieving strategies for discrete memoryless channels with feedback a class of multiple repetition strategies is presented for discrete memoryless channels with noiseless feedback this extends results obtained by schalkwijk 1971 to nonbinary asymmetric memoryless channels an interesting characterization of the attainability of a rate is presented in terms of a generating polynomial from which follows that for each strategy in our class there exists a discrete memoryless channel such that capacity can be achieved on this channel

channel combining and splitting for cutoff rate improvement the cutoff rate r0 w of a discrete memoryless channel dmc w is often used as a figure of merit alongside the channel capacity c w if a channel w is split into two possibly correlated subchannels w1 w2 the capacity function always satisfies c w1 c w2 lesc w while there are examples for which r0 w1 r0 w2 r0 w the fact that cutoff rate can be created by channel splitting was noticed by massey in his study of an optical modulation system this paper gives a general framework for achieving similar gains in the cutoff rate of arbitrary dmcs by methods of channel combining and splitting the emphasis is on simple schemes that can be implemented in practice we give several examples that achieve significant gains in cutoff rate at little extra system complexity theoretically as the complexity grows without bound the proposed framework is capable of boosting the cutoff rate of a channel to arbitrarily close to its capacity in a sense made precise in the paper apart from massey s work the methods studied here have elements in common with forney s concatenated coding idea a method by pinsker for cutoff rate improvement and certain coded modulation techniques namely ungerboeck s set partitioning idea and imai hirakawa multilevel coding these connections are discussed in the paper

what can we learn from the path equations identifiability constraints equivalence procedures are given for determining identified parameters finding constraints on the covariances and checking equivalence in acyclic recursive linear path models with correlated error terms disturbances by inspection of the path equations aided by simple recursions this provides a useful and general alternative to the employment of directed acyclic graph theory for such purposes

streaming data transmission in the moderate deviations and central limit regimes we consider streaming data transmission over a discrete memoryless channel a new message is given to the encoder at the beginning of each block and the decoder decodes each message sequentially after a delay of t blocks in this streaming setup we study the fundamental interplay between the rate and error probability in the central limit and moderate deviations regimes and show that 1 in the moderate deviations regime the moderate deviations constant improves over the block coding or non streaming setup by a factor of t and 2 in the central limit regime the second order coding rate improves by a factor of approximately sqrt t for a wide range of channel parameters for both the regimes we propose coding techniques that incorporate a joint encoding of fresh and previous messages in particular for the central limit regime we propose a coding technique with truncated memory to ensure that a summation of constants which arises as a result of applications of the central limit theorem does not diverge in the error analysis furthermore we explore interesting variants of the basic streaming setup in the moderate deviations regime we first consider a scenario with an erasure option at the decoder i e the decoder can output an erasure symbol instead of a message estimate and show that both the exponents of the total error and the undetected error probabilities improve by factors of t next by utilizing the erasure option we show that the exponent of the total error probability can be improved to that of the undetected error probability in the order sense at the expense of a variable decoding delay

on zero error source coding with decoder side information let x y be a pair of random variables distributed over a finite product set v spl times w according to a probability distribution p x y the following source coding problem is considered the encoder knows x while the decoder knows y and wants to learn x without error the minimum zero error asymptotic rate of transmission is shown to be the complementary graph entropy of an associated graph thus previous results in the literature provide upper and lower bounds for this minimum rate further these bounds are tight for the important class of perfect graphs the algorithmic aspects of instantaneous code design are considered next it is shown that optimal code design is np hard an optimal code design algorithm is derived polynomial time suboptimal algorithms are also presented and their average and worst case performance guarantees are established

fast iterative coding techniques for feedback channels a class of capacity achieving low complexity high reliability variable rate coding schemes is developed for communication over discrete memoryless channels with noiseless feedback algorithms for encoding and decoding that require computations growing linearly with the number of channel inputs used are developed the error exponent associated with the scheme is shown to be optimal and implies that capacity is achievable simulations are performed and support the analytically predicted high performance and low complexity

on the linear codebook level duality between slepian wolf coding and channel coding in this paper it is shown that each slepian wolf coding problem is related to a dual channel coding problem in the sense that the sphere packing exponents random coding exponents and correct decoding exponents in these two problems are mirror symmetrical to each other this mirror symmetry is interpreted as a manifestation of the linear codebook level duality between slepian wolf coding and channel coding furthermore this duality in conjunction with a systematic analysis of the expurgated exponents reveals that nonlinear slepian wolf codes can strictly outperform linear slepian wolf codes in terms of rate error tradeoff at high rates the linear codebook level duality is also established for general sources and channels

the redundancy of source coding with a fidelity criterion 1 known statistics the problem of redundancy of source coding with respect to a fidelity criterion is considered for any fixed rate r 0 and any memoryless source with finite source and reproduction alphabets and a common distribution p the nth order distortion redundancy d sub n r of fixed rate coding is defined as the minimum of the difference between the expected distortion per symbol of any block code with length n and rate r and the distortion rate function d p r of the source p it is demonstrated that for sufficiently large n d sub n r is equal to spl part spl part r d p r ln n 2n o ln n n where spl part spl part r d p r is the partial derivative of d p r evaluated at r and assumed to exist for any fixed distortion level d 0 and any memoryless source p the nth order rate redundancy r sub n d of coding at fixed distortion level d or by using d semifaithful codes is defined as the minimum of the difference between the expected rate per symbol of any d semifaithful code of length n and the rate distortion function r p d of p evaluated at d it is proved that for sufficiently large n r sub n d is upper bounded by ln n n o ln n n and lower bounded by in n 2n o in n n as a by product the lower bound of r sub n d derived in this paper gives a positive answer to a conjecture proposed by yu and speed 1993

achievable moderate deviations asymptotics for streaming compression of correlated sources motivated by streaming multi view video coding and wireless sensor networks we consider the problem of blockwise streaming compression of a pair of correlated sources which we term streaming slepian wolf coding we study the moderate deviations regime in which the rate pairs of a sequence of codes converge along a straight line to various points on the boundary of the slepian wolf region at a speed slower than the inverse square root of the blocklength n while the error probability decays subexponentially fast in n our main result focuses on the directions of approaches to corner points of the slepian wolf region it states that for each correlated source and all corner points there exists a non empty subset of directions of approaches such that the moderate deviations constant the constant of proportionality for the subexponential decay of the error probability is enhanced over the non streaming case by at least a factor of t the block delay of decoding source block pairs we specialize our main result to the setting of streaming lossless source coding and generalize this result to the setting where we have different delay requirements for each of the two source blocks the proof of our main result involves the use of various analytical tools and amalgamates several ideas from the recent information theoretic streaming literature we adapt the so called truncated memory encoding idea from draper and khisti 2011 and lee tan and khisti 2016 to ensure that the effect of error accumulation is nullified in the limit of large block lengths we also adapt the use of the so called minimum weighted empirical suffix entropy decoder which was used by draper chang and sahai 2014 to derive achievable error exponents for symbolwise streaming slepian wolf coding

the awgn red alert problem consider the following unequal error protection scenario one special message dubbed the red alert message is required to have an extremely small probability of missed detection the remainder of the messages must keep their average probability of error and probability of false alarm below a certain threshold the goal then is to design a codebook that maximizes the error exponent of the red alert message while ensuring that the average probability of error and probability of false alarm go to zero as the blocklength goes to infinity this red alert exponent has previously been characterized for discrete memoryless channels this paper completely characterizes the optimal red alert exponent for additive white gaussian noise channels with block power constraints

on the redundancy of slepian wolf coding in this paper the redundancy of both variable and fixed rate slepian wolf coding is considered given any jointly memoryless source side information pair xi yi i 1 infin with finite alphabet the redundancy rn isinn of variable rate slepian wolf coding of x1 n with decoder only side information y1 n depends on both the block length n and the decoding block error probability isinn and is defined as the difference between the minimum average compression rate of order n variable rate slepian wolf codes having the decoding block error probability less than or equal to isinn and the conditional entropy h x y where h x y is the conditional entropy rate of the source given the side information the redundancy of fixed rate slepian wolf coding of x1 n with decoder only side information y1 n is defined similarly and denoted by rf n isinn it is proved that under mild assumptions about isinn rn isinn dvradic log isinn n oradic log isinn n and rf n isinn dfradic log isinn n o radic log isinn n where df and dnu are two constants completely determined by the joint distribution of the source side information pair since dv is generally smaller than df our results show that variable rate slepian wolf coding is indeed more efficient than fixed rate slepian wolf coding