why do block length and delay behave differently if feedback is present for output symmetric discrete memoryless channels dmcs at even moderately high rates fixed block length communication systems show no improvements in their error exponents with feedback this paper studies systems with fixed end to end delay and shows that feedback generally provides dramatic gains in the error exponents a new upper bound the uncertainty focusing bound is given on the probability of symbol error in a fixed delay communication system with feedback this bound turns out to have a form similar to viterbi s bound used for the block error probability of convolutional codes as a function of the fixed constraint length the uncertainty focusing bound is shown to be asymptotically achievable with noiseless feedback for erasure channels as well as for any output symmetric dmc that has strictly positive zero error capacity furthermore it can be achieved in a delay universal anytime fashion even if the feedback itself is delayed by a small amount finally it is shown that for end to end delay it is generally possible at high rates to beat the sphere packing bound for general dmcs thereby providing a counterexample to a conjecture of pinsker

a tight upper bound on the second order coding rate of the parallel gaussian channel with feedback this paper investigates the asymptotic expansion for the maximum rate of fixed length codes over a parallel gaussian channel with feedback under the following setting a peak power constraint is imposed on every transmitted codeword and the average error probabilities of decoding the transmitted message are non vanishing as the blocklength increases the main contribution of this paper is a self contained proof of an upper bound on the first and second order asymptotics of the parallel gaussian channel with feedback the proof techniques involve developing an information spectrum bound followed by using curtiss theorem to show that a sum of dependent random variables associated with the information spectrum bound converges in distribution to a sum of independent random variables thus facilitating the use of the usual central limit theorem combined with existing achievability results our result implies that the presence of feedback does not improve the first and second order asymptotics

exact moderate deviation asymptotics in streaming data transmission in this paper a streaming transmission setup is considered where an encoder observes a new message in the beginning of each block and a decoder sequentially decodes each message after a delay of t blocks in this streaming setup the fundamental interplay between the coding rate the error probability and the blocklength in the moderate deviations regime is studied for output symmetric channels the moderate deviations constant is shown to improve over the block coding or non streaming setup by exactly a factor of t for a certain range of moderate deviations scalings for the converse proof a more powerful decoder to which some extra information is fedforward is assumed the error probability is bounded first for an auxiliary channel and this result is translated back to the original channel by using a newly developed change of measure lemma where the speed of decay of the remainder term in the exponent is carefully characterized for the achievability proof a known coding technique that involves a joint encoding and decoding of fresh and past messages is applied with some manipulations in the error analysis

variable length coding over an unknown channel burnashev in 1976 gave an exact expression for the reliability function of a discrete memoryless channel dmc with noiseless feedback a coding scheme that achieves this exponent needs in general to know the statistics of the channel suppose now that the coding scheme is designed knowing only that the channel belongs to a family q of dmcs is there a coding scheme with noiseless feedback that achieves burnashev s exponent uniformly over q at a nontrivial rate we answer the question in the affirmative for two families of channels binary symmetric and z for these families we show that for any given fraction there is a feedback coding strategy such that for any member of the family i guarantees this fraction of its capacity as rate and ii guarantees the corresponding burnashev s exponent therefore for these families in terms of delay and error probability the knowledge of the channel becomes asymptotically irrelevant in feedback code design there are blind schemes that perform as well as the best coding scheme designed with the foreknowledge of the channel under use however a converse result shows that in general even for families that consist of only two channels such blind schemes do not exist

phase transition in the 1d random field ising model with long range interaction we study one dimensional ising spin systems with ferromagnetic long range interaction decaying as n 2 alpha in left frac 12 frac ln 3 ln2 1 right in the presence of external random fields we assume that the random fields are given by a collection of symmetric independent identically distributed real random variables gaussian or subgaussian we show for temperature and strength of the randomness variance small enough with ip 1 with respect to the random fields that there are at least two distinct extremal gibbs measures

optimum power control at finite blocklength this paper investigates the maximal channel coding rate achievable at a given blocklength n and error probability epsilon when the codewords are subjected to a long term i e averaged over all codeword power constraint the second order term in the large n expansion of the maximal channel coding rate is characterized both for additive white gaussian noise awgn channels and for quasi static fading channels with perfect channel state information available at both the transmitter and the receiver it is shown that in both the cases the second order term is proportional to n 1 ln n 1 2 for the quasi static fading case this second order term is achieved by truncated channel inversion namely by concatenating a dispersion optimal code for an awgn channel subject to a short term power constraint with a power controller that inverts the channel whenever the fading gain is above a certain threshold easy to evaluate approximations of the maximal channel coding rate are developed for both the awgn and the quasi static fading case

a simple converse of burnashev s reliability function in a remarkable paper published in 1976 burnashev determined the reliability function of variable length block codes over discrete memoryless channels dmcs with feedback subsequently an alternative achievability proof was obtained by yamamoto and itoh via a particularly simple and instructive scheme their idea is to alternate between a communication and a confirmation phase until the receiver detects the codeword used by the sender to acknowledge that the message is correct we provide a converse that parallels the yamamoto itoh achievability construction besides being simpler than the original the proposed converse suggests that a communication and a confirmation phase are implicit in any scheme for which the probability of error decreases with the largest possible exponent the proposed converse also makes it intuitively clear why the terms that appear in burnashev s exponent are necessary

active sequential hypothesis testing consider a decision maker who is responsible to dynamically collect observations so as to enhance his information about an underlying phenomena of interest in a speedy manner while accounting for the penalty of wrong declaration due to the sequential nature of the problem the decision maker relies on his current information state to adaptively select the most informative sensing action among the available ones in this paper using results in dynamic programming lower bounds for the optimal total cost are established the lower bounds characterize the fundamental limits on the maximum achievable information acquisition rate and the optimal reliability moreover upper bounds are obtained via an analysis of two heuristic policies for dynamic selection of actions it is shown that the first proposed heuristic achieves asymptotic optimality where the notion of asymptotic optimality due to chernoff implies that the relative difference between the total cost achieved by the proposed policy and the optimal total cost approaches zero as the penalty of wrong declaration hence the number of collected samples increases the second heuristic is shown to achieve asymptotic optimality only in a limited setting such as the problem of a noisy dynamic search however by considering the dependency on the number of hypotheses under a technical condition this second heuristic is shown to achieve a nonzero information acquisition rate establishing a lower bound for the maximum achievable rate and error exponent in the case of a noisy dynamic search with size independent noise the obtained nonzero rate and error exponent are shown to be maximum

strong converses for group testing from finite blocklength results we prove new strong converse results in a variety of group testing settings generalizing a result of baldassini et al first in the non adaptive case we mimic the hypothesis testing argument introduced in the finite blocklength channel coding regime by polyanskiy et al and using joint source channel coding arguments of kostina and verdu in the adaptive case we combine this approach with a novel model formulation based on causal probability and directed information theory in both cases we prove results which are valid for finite sized problems and imply capacity results in the asymptotic regime these results are illustrated graphically for a range of models

on gaussian channels with feedback under expected power constraints and with non vanishing error probabilities in this paper we consider single and multi user gaussian channels with feedback under expected power constraints and with non vanishing error probabilities in the first of two contributions we study asymptotic expansions for the additive white gaussian noise awgn channel with feedback under the average error probability formalism by drawing ideas from gallager and nakiboglu s work for the direct part and the meta converse for the converse part we establish the varepsilon capacity and show that it depends on varepsilon in general and so the strong converse fails to hold furthermore we provide bounds on the second order term in the asymptotic expansion we show that for any positive integer l the second order term is bounded between a term proportional to ln l n where ln l cdot is the l fold nested logarithm function and a term proportional to n ln n 1 2 where n is the blocklength the lower bound on the second order term shows that feedback does provide an improvement in the maximal achievable rate over the case where no feedback is available in our second contribution we establish the varepsilon capacity region for the awgn multiple access channel with feedback under the expected power constraint by combining ideas from hypothesis testing information spectrum analysis ozarow s coding scheme and power control

channel coding rate in the finite blocklength regime this paper investigates the maximal channel coding rate achievable at a given blocklength and error probability for general classes of channels new achievability and converse bounds are given which are tighter than existing bounds for wide ranges of parameters of interest and lead to tight approximations of the maximal achievable rate for blocklengths n as short as 100 it is also shown analytically that the maximal rate achievable with error probability isclosely approximated by c v n q 1 where c is the capacity v is a characteristic of the channel referred to as channel dispersion and q is the complementary gaussian cumulative distribution function

variable length compression allowing errors this paper studies the fundamental limits of the minimum average length of lossless and lossy variable length compression allowing a nonzero error probability epsilon for lossless compression we give nonasymptotic bounds on the minimum average length in terms of erokhin s rate distortion function and we use those bounds to obtain a gaussian approximation on the speed of approach to the limit which is quite accurate for all but small blocklengths 1 epsilon k h mathsf s k v mathsf s 2 pi 1 2 mathop rm exp q 1 left epsilon right 2 2 where q 1 left cdot right is the functional inverse of the standard gaussian complementary cumulative distribution function and v mathsf s is the source dispersion a nonzero error probability thus not only reduces the asymptotically achievable rate by a factor of 1 epsilon but this asymptotic limit is approached from below i e larger source dispersions and shorter blocklengths are beneficial variable length lossy compression under an excess distortion constraint is shown to exhibit similar properties

refinement of the sphere packing bound asymmetric channels we provide a refinement of the sphere packing bound for constant composition codes over asymmetric discrete memoryless channels that improves the subexponential factor in front of the exponent the order of our subexponential factor is n 0 5 1 e r for any 0 where r is the left derivative of the sphere packing exponent at rate r and n is the blocklength

variable rate channel capacity this paper introduces the notions of variable to fixed and fixed to variable channel capacity without feedback for channels that satisfy the strong converse these notions coincide with the conventional shannon capacity for channels that do not behave ergodically the conventional fixed rate shannon capacity only depends on least favorable channel conditions while the variable rate capacity notions are able to capture the whole range of channel states and their likelihood even in the absence of any side information about channel state at the transmitter particular emphasis is placed on memoryless channels that are governed by finitely valued states we show that single user variable to fixed channel capacity is intimately connected to the capacity region of broadcast channels with degraded message sets and we give an expression for the fixed to variable capacity

feedback in the non asymptotic regime without feedback the backoff from capacity due to non asymptotic blocklength can be quite substantial for blocklengths and error probabilities of interest in many practical applications in this paper novel achievability bounds are used to demonstrate that in the non asymptotic regime the maximal achievable rate improves dramatically thanks to variable length coding and feedback for example for the binary symmetric channel with capacity 1 2 the blocklength required to achieve 90 of the capacity is smaller than 200 compared to at least 3100 for the best fixed blocklength code even with noiseless feedback virtually all the advantages of noiseless feedback are shown to be achievable even if the feedback link is used only to send a single signal informing the encoder to terminate the transmission stop feedback it is demonstrated that the non asymptotic behavior of the fundamental limit depends crucially on the particular model chosen for the end of packet control signal fixed blocklength codes and related questions concerning communicating with a guaranteed delay are discussed in which situation feedback is demonstrated to be almost useless even non asymptotically

a matrosov theorem for adversarial markov decision processes matrosov s relaxation of lyapunov conditions for uniform global asymptotic stability in time varying systems is extended to stochastic set valued discrete time systems nested matrosov functions are used to give conditions for stability that complement invariance principles for time invariant systems unlike invariance principles matrosov functions also can be applied to general time varying systems

a general formula for channel capacity a formula for the capacity of arbitrary single user channels without feedback not necessarily information stable stationary etc is proved capacity is shown to equal the supremum over all input processes of the input output inf information rate defined as the liminf in probability of the normalized information density the key to this result is a new converse approach based on a simple new lower bound on the error probability of m ary hypothesis tests among equiprobable hypotheses a necessary and sufficient condition for the validity of the strong converse is given as well as general expressions for spl epsiv capacity

an efficient feedback coding scheme with low error probability for discrete memoryless channels existing fixed length feedback communication schemes are either specialized to particular channels schalkwijk kailath horstein or apply to general channels but either have high coding complexity block feedback schemes or are difficult to analyze posterior matching this paper introduces a new fixed length feedback coding scheme which achieves the capacity for all discrete memoryless channels has an error exponent that approaches the sphere packing bound as the rate approaches the capacity and has o n log n coding complexity these benefits are achieved by judiciously combining features from previous schemes with new randomization technique and encoding decoding rule these new features make the analysis of the error probability for the new scheme easier than for posterior matching

joint source channel coding with feedback this paper quantifies the fundamental limits of variable length transmission of a general possibly analog source over a memoryless channel with noiseless feedback under a distortion constraint we consider excess distortion average distortion and guaranteed distortion d semifaithful codes in contrast to the asymptotic fundamental limit a general conclusion is that allowing variable length codes and feedback leads to a sizable improvement in the fundamental delay distortion tradeoff in addition we investigate the minimum energy required to reproduce k source samples with a given fidelity after transmission over a memoryless gaussian channel and we show that the required minimum energy is reduced with feedback and an average rather than maximal power constraint

reliability output decoding of tail biting convolutional codes we present extensions to raghavan and baum s reliability output viterbi algorithm rova to accommodate tail biting convolutional codes these tail biting reliability output algorithms compute the exact word error probability of the decoded codeword after first calculating the posterior probability of the decoded tail biting codeword s starting state one approach employs a state estimation algorithm that selects the maximum a posteriori state based on the posterior distribution of the starting states another approach is an approximation to the exact tail biting rova that estimates the word error probability a comparison of the computational complexity of each approach is discussed in detail the presented reliability output algorithms apply to both feedforward and feedback tail biting convolutional encoders these tail biting reliability output algorithms are suitable for use in reliability based retransmission schemes with short blocklengths in which terminated convolutional codes would introduce rate loss

lautum information a popular way to measure the degree of dependence between two random objects is by their mutual information defined as the divergence between the joint and product of marginal distributions we investigate an alternative measure of dependence the lautum information defined as the divergence between the product of marginal and joint distributions i e swapping the arguments in the definition of mutual information some operational characterizations and properties are provided for this alternative measure of information

saddle point in the minimax converse for channel coding a minimax metaconverse has recently been proposed as a simultaneous generalization of a number of classical results and a tool for the nonasymptotic analysis in this paper it is shown that the order of optimizing the input and output distributions can be interchanged without affecting the bound in the course of the proof a number of auxiliary results of separate interest are obtained in particular it is shown that the optimization problem is convex and can be solved in many cases by the symmetry considerations as a consequence it is demonstrated that in the latter cases the multiletter input distribution in information spectrum verdu han converse bound can be taken to be a memoryless product of single letter ones a tight converse for the binary erasure channel is rederived by computing the optimal nonproduct output distribution for discrete memoryless channels a conjecture of poor and verdu regarding the tightness of the information spectrum bound on the error exponents is resolved in the negative concept of the channel symmetry group is established and relations with the definitions of symmetry by gallager and dobrushin are investigated

extrinsic jensen shannon divergence applications to variable length coding this paper considers the problem of variable length coding over a discrete memoryless channel with noiseless feedback this paper provides a stochastic control view of the problem whose solution is analyzed via a newly proposed symmetrized divergence termed extrinsic jensen shannon ejs divergence it is shown that strictly positive lower bounds on ejs divergence provide nonasymptotic upper bounds on the expected code length this paper presents strictly positive lower bounds on ejs divergence and hence nonasymptotic upper bounds on the expected code length for the following two coding schemes 1 variable length posterior matching and 2 maxejs coding scheme that is based on a greedy maximization of the ejs divergence as an asymptotic corollary of the main results this paper also provides a rate reliability test variable length coding schemes that satisfy the condition s of the test for parameters r and e are guaranteed to achieve a rate r and an error exponent e the results are specialized for posterior matching and maxejs to obtain deterministic one phase coding schemes achieving capacity and optimal error exponent for the special case of symmetric binary input channels simpler deterministic schemes of optimal performance are proposed and analyzed