second order and moderate deviations asymptotics for successive refinement we derive the optimal second order coding region and moderate deviations constant for successive refinement source coding with a joint excess distortion probability constraint we consider two scenarios 1 a discrete memoryless source dms and arbitrary distortion measures at the decoders and 2 a gaussian memoryless source gms and quadratic distortion measures at the decoders for a dms with arbitrary distortion measures we prove an achievable second order coding region using type covering lemmas by kanlis and narayan and by no ingber and weissman we prove the converse using the perturbation approach by gu and effros when the dms is successively refinable the expressions for the second order coding region and the moderate deviations constant are simplified and easily computable for this case we also obtain new insights on the second order behavior compared with the scenario where separate excess distortion proabilities are considered for example we describe a dms for which the optimal second order region transitions from being characterizable by a bivariate gaussian to a univariate gaussian as the distortion levels are varied we then consider a gms with quadratic distortion measures to prove the direct part we make use of the sphere covering theorem by verger gaugry together with appropriately defined gaussian type classes to prove the converse we generalize kostina and verdu s one shot converse bound for point to point lossy source coding we remark that this proof is applicable to general successively refinable sources in the proofs of the moderate deviations results for both scenarios we follow a strategy similar to that for the second order asymptotics and use the moderate deviations principle

hierarchical guessing with a fidelity criterion arikan and merhav 1998 studied the problem of guessing a random vector x within distortion d and characterized the best attainable exponent e d spl rho of the spl rho th moment of the number of required guesses g x until the guessing error falls below d we extend these results to a multistage hierarchical guessing model which allows for a faster search for a codeword vector at the encoder of a rate distortion codebook in the two stage case of this model if the target distortion level is d sub 2 the guesser first makes guesses with respect to a higher distortion level d sub 1 and then upon his her first success directs the subsequent guesses to distortion d sub 2 as in the above mentioned earlier paper we provide a single letter characterization of the best attainable guessing exponent which relies heavily on well known results on the successive refinement problem we also relate this guessing exponent function to the source coding error exponent function of the two step coding process

on the joint source channel coding error exponent for discrete memoryless systems we investigate the computation of csisza spl acute r s bounds for the joint source channel coding jscc error exponent e sub j of a communication system consisting of a discrete memoryless source and a discrete memoryless channel we provide equivalent expressions for these bounds and derive explicit formulas for the rates where the bounds are attained these equivalent representations can be readily computed for arbitrary source channel pairs via arimoto s algorithm when the channel s distribution satisfies a symmetry property the bounds admit closed form parametric expressions we then use our results to provide a systematic comparison between the jscc error exponent e sub j and the tandem coding error exponent e sub t which applies if the source and channel are separately coded it is shown that e sub t spl les e sub j spl les 2e sub t we establish conditions for which e sub j e sub t and for which e sub j 2e sub t numerical examples indicate that e sub j is close to 2e sub t for many source channel pairs this gain translates into a power saving larger than 2 db for a binary source transmitted over additive white gaussian noise awgn channels and rayleigh fading channels with finite output quantization finally we study the computation of the lossy jscc error exponent under the hamming distortion measure

encoding tasks and renyi entropy a task is randomly drawn from a finite set of tasks and is described using a fixed number of bits all the tasks that share its description must be performed upper and lower bounds on the minimum pth moment of the number of performed tasks are derived the case where a sequence of tasks is produced by a source and n tasks are jointly described using nr bits is considered if r is larger than the renyi entropy rate of the source of order 1 1 provided it exists then the th moment of the ratio of performed tasks to n can be driven to one as n tends to infinity if r is smaller than the renyi entropy rate this moment tends to infinity the results are generalized to account for the presence of side information in this more general setting the key quantity is a conditional version of renyi entropy that was introduced by arimoto for iid sources two additional extensions are solved one of a rate distortion flavor and the other where different tasks may have different nonnegative costs finally a divergence that was identified by sundaresan as a mismatch penalty in the massey arikan guessing problem is shown to play a similar role here

guessing under source uncertainty this paper considers the problem of guessing the realization of a finite alphabet source when some side information is provided the only knowledge the guesser has about the source and the correlated side information is that the joint source is one among a family a notion of redundancy is first defined and a new divergence quantity that measures this redundancy is identified this divergence quantity shares the pythagorean property with the kullback leibler divergence good guessing strategies that minimize the supremum redundancy over the family are then identified the min sup value measures the richness of the uncertainty set the min sup redundancies for two examples the families of discrete memoryless sources and finite state arbitrarily varying sources are then determined

improved bounds on lossless source coding and guessing moments via renyi measures this paper provides upper and lower bounds on the optimal guessing moments of a random variable taking values on a finite set when side information may be available these moments quantify the number of guesses required for correctly identifying the unknown object and similarly to arikan s bounds they are expressed in terms of the arimoto renyi conditional entropy although arikan s bounds are asymptotically tight the improvement of the bounds in this paper is significant in the non asymptotic regime relationships between moments of the optimal guessing function and the map error probability are also established characterizing the exact locus of their attainable values the bounds on optimal guessing moments serve to improve non asymptotic bounds on the cumulant generating function of the codeword lengths for fixed to variable optimal lossless source coding without prefix constraints non asymptotic bounds on the reliability function of discrete memoryless sources are derived as well relying on these techniques lower bounds on the cumulant generating function of the codeword lengths are derived by means of the smooth renyi entropy for source codes that allow decoding errors

compression for quadratic similarity queries the problem of performing similarity queries on compressed data is considered we focus on the quadratic similarity measure and study the fundamental tradeoff between compression rate sequence length and reliability of queries performed on the compressed data for a gaussian source we show that the queries can be answered reliably if and only if the compression rate exceeds a given threshold the identification rate which we explicitly characterize moreover when compression is performed at a rate greater than the identification rate responses to queries on the compressed data can be made exponentially reliable we give a complete characterization of this exponent which is analogous to the error and excess distortion exponents in channel and source coding respectively for a general source we prove that as with classical compression the gaussian source requires the largest compression rate among sources with a given variance moreover a robust scheme is described that attains this maximal rate for any source distribution

on the conditional smooth renyi entropy and its applications in guessing and source coding a novel definition of the conditional smooth renyi entropy which is different from that of renner and wolf is introduced it is shown that our definition of the conditional smooth renyi entropy is appropriate for providing lower and upper bounds on the optimal guessing moment in a guessing problem where the guesser is allowed to stop guessing and declare an error further a general formula for the optimal guessing exponent is presented in particular a single letterized formula for a mixture of i i d sources is obtained it is also shown that our definition is appropriate to characterize the optimal exponential moment of the codeword length in the problem of source coding with common side information available at the encoder and decoder under a constraint on the probability of a decoding error

from approximating subdivision schemes for exponential splines to high performance interpolating algorithms in this work we construct three novel families of approximating subdivision schemes that generate piecewise exponential polynomials and we show how to convert these into interpolating schemes of great interest in curve design for their ability to reproduce important analytical shapes and to provide highly smooth limit curves with a controllable tension in particular throughout this paper we will focus on the derivation of 6 point interpolating schemes that turn out to be unique in combining vital ingredients like c 2 continuity simplicity of definition ease of implementation user independency tension control and ability to reproduce salient trigonometric and transcendental curves

guessing subject to distortion we investigate the problem of guessing a random vector x within distortion level d our aim is to characterize the best attainable performance in the sense of minimizing in some probabilistic sense the number of required guesses g x until the error falls below d the underlying motivation is that g x is the number of candidate codewords to be examined by a rate distortion block encoder until a satisfactory codeword is found in particular for memoryless sources we provide a single letter characterization of the least achievable exponential growth rate of the spl rho th moment of g x as the dimension of the random vector x grows without bound in this context we propose an asymptotically optimal guessing scheme that is universal both with respect to the information source and the value of spl rho we then study some properties of the exponent function e d spl rho along with its relation to the source coding exponents finally we provide extensions of our main results to the gaussian case guessing with side information and sources with memory

extremal properties of rate distortion functions an attempt is made to determine whether it is true that for a fixed distortion level delta the rate distortion function r p delta has in the distribution p no local maxima with values different from the global maximum it is shown that in general the answer is negative however the answer is positive for hamming distortion measures moreover their r is schur concave

joint source channel coding excess distortion exponent for some memoryless continuous alphabet systems we investigate the joint source channel coding jscc excess distortion exponent ej the exponent of the probability of exceeding a prescribed distortion level for some memoryless communication systems with continuous alphabets we first establish upper and lower bounds for ej for systems consisting of a memoryless gaussian source under the squared error distortion fidelity criterion and a memoryless additive gaussian noise channel with a quadratic power constraint at the channel input a necessary and sufficient condition for which the two bounds coincide is provided thus exactly determining the exponent this condition is observed to hold for a wide range of source channel parameters as an application we study the advantage in terms of the excess distortion exponent of jscc over traditional tandem separate coding for gaussian systems a formula for the tandem exponent is derived in terms of the gaussian source and gaussian channel exponents and numerical results show that jscc often substantially outperforms tandem coding the problem of transmitting memoryless laplacian sources over the gaussian channel under the magnitude error distortion is also carried out finally we establish a lower bound for ej for a certain class of continuous source channel pairs when the distortion measure is a metric

asymptotics and non asymptotics for universal fixed to variable source coding universal fixed to variable lossless source coding for memoryless sources is studied in the finite blocklength and higher order asymptotic regimes optimal three term fixed error asymptotic expressions are derived for general fixed to variable codes and for prefix codes it is shown that the non prefix type size code in which codeword lengths are chosen in ascending order of type class size achieves the optimal third order term and outperforms classical two stage codes converse results are proved making use of a result on the distribution of the empirical entropy and laplace s approximation finally the fixed to variable coding problem without a prefix constraint is shown to be essentially the same as the universal guessing problem

restriction of divisor classes to hypersurfaces in characteristic p abstract the injectivity of the restriction homomorphism on divisor class groups to hypersurfaces has been studied by grothendieck danilov lipman and griffith weston among others in particular when a is a noetherian normal domain of equicharacteristic zero and a fa satisfies r 1 spiroff established a map cl a cl a fa where a fa represents the integral closure of a fa and gave some conditions for injectivity in this paper the authors continue in the same vein but in the case of characteristic p 0 in addition when the hypersurface a fa is normal they provide further enlightenment about the kernel of cl a cl a fa finally using the second author s previous results they exhibit a new class of examples for which the kernel is non trivial

reducing guesswork via an unreliable oracle alice holds a random variable x and bob is trying to guess its value by asking questions of the form is x x alice answers truthfully and the game terminates once bob guesses correctly before the game begins bob is allowed to reach out to an oracle carole and ask her any yes no question i e a question of the form is x in a carole is known to lie with a given probability p what should bob ask carole if he would like to minimize his expected guessing time when carole is always truthful p 0 it is not difficult to check that bob should order the symbol probabilities in descending order and ask carole whether the index of x with respect to this order is even or odd we show that this strategy is almost optimal for any lying probability p up to a small additive constant upper bounded by 1 4 we discuss a connection to the cutoff rate of the bsc with feedback

successive refinement of information the successive refinement of information consists of first approximating data using a few bits of information then iteratively improving the approximation as more and more information is supplied the goal is to achieve an optimal description at each stage in general an ongoing description which is rate distortion optimal whenever it is interrupted is sought it is shown that in order to achieve optimal successive refinement the necessary and sufficient conditions are that the solutions of the rate distortion problem can be written as a markov chain in particular all finite alphabet signals with hamming distortion satisfy these requirements it is also shown that the same is true for gaussian signals with squared error distortion and for laplacian signals with absolute error distortion a simple counterexample with absolute error distortion and a symmetric source distribution which shows that successive refinement is not always achievable is presented

guesswork large deviations and shannon entropy how hard is it to guess a password massey showed that a simple function of the shannon entropy of the distribution from which the password is selected is a lower bound on the expected number of guesses but one which is not tight in general in a series of subsequent papers under ever less restrictive stochastic assumptions an asymptotic relationship as password length grows between scaled moments of the guesswork and specific renyi entropy was identified here we show that when appropriately scaled as the password length grows the logarithm of the guesswork satisfies a large deviation principle ldp providing direct estimates of the guesswork distribution when passwords are long the rate function governing the ldp possesses a specific restrictive form that encapsulates underlying structure in the nature of guesswork returning to massey s original observation a corollary to the ldp shows that expectation of the logarithm of the guesswork is the specific shannon entropy of the password selection process

minimum guesswork with an unreliable oracle we study a guessing game where alice holds a discrete random variable x and bob tries to sequentially guess its value before the game begins bob can obtain side information about x by asking an oracle carole any binary question of his choosing carole s answer is however unreliable and is incorrect with probability epsilon we show that bob should always ask carole whether the index of x is odd or even with respect to a descending order of probabilities this question simultaneously minimizes all the guessing moments for any value of epsilon in particular this result settles a conjecture of burin and shayevitz we further consider a more general setup where bob can ask a multiple choice m ary question and then observe carole s answer through a noisy channel when the channel is completely symmetric i e when carole decides whether to lie regardless of bob s question and has no preference when she lies a similar question about the ordered index of x modulo m is optimal interestingly however the problem of testing whether a given question is optimal appears to be generally difficult in other symmetric channels we provide supporting evidence for this difficulty by showing that a core property required in our proofs becomes np hard to test in the general m ary case we establish this hardness result via a reduction from the problem of testing whether a system of modular difference disequations has a solution which we prove to be np hard for m geq 3

a large deviations approach to secure lossy compression a shannon cipher system for memoryless sources in which distortion is allowed at the legitimate decoder is considered the source is compressed using a secured rate distortion code which satisfies a constraint on the compression rate as well as a constraint on the exponential rate of the excess distortion probability at the legitimate decoder secrecy is measured by the exponential rate of the exiguous distortion probability at the eavesdropper rather than by the traditional measure of equivocation the perfect secrecy exponent is defined as the maximal exiguous distortion exponent achievable when the key rate is unlimited the reproduction based estimate exponent is defined as the maximal exiguous distortion exponent achievable for a genie aided eavesdropper which knows the secret key under limited key rate it is proved that the maximal achievable exiguous distortion exponent is equal to the minimum between the key rate plus the reproduction based estimate exponent and the perfect secrecy exponent the result is generalized to a fairly general class of variable key rate and coding rate codes

noisy guesses we consider the problem of guessing a random finite alphabet secret n vector where the guesses are transmitted via a noisy channel we provide a single letter formula for the best achievable exponential growth rate of the rho th moment of the number of guesses as a function of n this formula exhibits a fairly clear insight concerning the penalty due to the noise we describe two different randomized schemes that achieve the optimal guessing exponent one of them is fully universal in the sense of being independent of source that governs the vector to be guessed the channel that corrupts the guesses and the moment power rho interestingly it turns out that in general the optimal guessing exponent function exhibits a phase transition when it is examined either as a function of the channel parameters or as a function of rho as long as the channel is not too distant in a certain sense to be defined precisely from the identity channel i e the clean channel or equivalently as long rho is larger than a certain critical value rho text c there is no penalty at all in the guessing exponent compared to the case of noiseless guessing

information theoretic applications of the logarithmic probability comparison bound a well known technique in estimating the probabilities of rare events in general and in information theory in particular used for example in the sphere packing bound is that of finding a reference probability measure under which the event of interest has the probability of order one and estimating the probability in question by means of the kullback leibler divergence a method has recently been proposed in 2 that can be viewed as an extension of this idea in which the probability under the reference measure may itself be decaying exponentially and the renyi divergence is used instead the purpose of this paper is to demonstrate the usefulness of this approach in various information theoretic settings for the problem of channel coding we provide a general methodology for obtaining matched mismatched and robust error exponent bounds as well as new results in a variety of particular channel models other applications we address include rate distortion coding and the problem of guessing

analysis of asymptotically equivalent binary subdivision schemes non stationary uniform subdivision schemes are presented including some important examples as the up function and exponential splines the multivariate analysis of subdivision schemes is extended to a class of non stationary schemes which are asymptotically equivalent to converging stationary or non stationary schemes

a new non stationary binary 6 point subdivision scheme in this paper the hyperbolic form of binary 6 point interpolating non stationary subdivision scheme has been constructed using the hyperbolic function some of the important properties of the proposed scheme has been discussed the comparison of the proposed scheme with the trigonometric form of binary 6 point interpolatory non stationary scheme is depicted through examples which indicate that the proposed scheme not only accommodates and is more consistent with the control polygon but also generates pleasing curves corresponding to the larger parametric values in and outside the parametric interval as compared to the trigonometric form of binary 6 point scheme developed by daniel et al comparison with some other interpolating non stationary subdivision schemes has also been demonstrated

list decoding random coding exponents and expurgated exponents some new results are derived concerning random coding error exponents and expurgated exponents for list decoding with a deterministic list size l two asymptotic regimes are considered the fixed list size regime where l is fixed independently of the block length n and exponential list size where l grows exponentially with n we first derive a general upper bound on the list decoding average error probability which is suitable for both regimes this bound leads to more specific bounds in the two regimes in the fixed list size regime the bound is related to known bounds and we establish its exponential tightness in the exponential list size regime we establish the achievability of the well known sphere packing lower bound an immediate byproduct of our analysis in both regimes is the universality of the maximum mutual information list decoder in the error exponent sense finally we consider expurgated bounds at low rates both using gallager s approach and csiszar korner marton approach which is equivalent for the optimal input assignment the latter expurgated bound which involves the notion of multi information is also modified to apply to continuous alphabet channels and in particular to the gaussian memoryless channel where the expression of the expurgated bound becomes quite explicit

an interpolating 6 point c2 non stationary subdivision scheme in this paper we consider a 6 point interpolating c 2 binary non stationary subdivision scheme and present some of its important properties we highlight some advantages of the scheme and demonstrate its performance by some examples

random coding lower bounds for the error exponent of joint quantization and watermarking systems we establish random coding lower bounds to the error exponent of discrete and gaussian joint quantization and private watermarking systems in the discrete system both the covertext and the attack channel are memoryless and have finite alphabets in the gaussian system the covertext is memoryless gaussian and the attack channel has additive memoryless gaussian noise in both cases our bounds on the error exponent are positive in the interior of the achievable quantization and watermarking rate region

on hierarchical type covering this correspondence focuses on a significant distinction between two hierarchical type covering strategies namely weak and strong covering and on the impact of this distinction on known results in particular it is demonstrated that the rate region for weak covering whose natural use is in scalable source coding is generally larger than the rate region for strong covering which is primarily useful in hierarchical guessing this correspondence also presents a corrected converse result for the hierarchical guessing problem

reliability in source coding with side information we study error exponents for source coding with side information both achievable exponents and converse bounds are obtained for the following two cases lossless source coding with coded information and lossy source coding with full side information wyner ziv these results recover and extend several existing results on source coding error exponents and are tight in some circumstances our bounds have a natural interpretation as a two player game between nature and the code designer with nature seeking to minimize the exponent and the code designer seeking to maximize it in the wyner ziv problem our analysis exposes a tension in the choice of test channel with the optimal test channel balancing two competing error events the gaussian and binary erasure cases are examined in detail

universal coding with minimum probability of codeword length overflow lossless block to variable length source coding is studied for finite state finite alphabet sources the aim is to minimize the probability that the normalized length of the codeword will exceed a given threshold b subject to the kraft inequality it is shown that the lempel ziv algorithm 1978 asymptotically attains the optimal performance in the sense just defined independently of the source and the value of b for the subclass of unifilar markov sources faster convergence to the asymptotic optimum performance can be accomplished by using the minimum description length universal code for this subclass it is demonstrated that these universal codes are also nearly optimal in the sense of minimizing buffer overflow probability and asymptotically optimal in a competitive sense

guessing revisited a large deviations approach the problem of guessing a random string is revisited a close relation between guessing and compression is first established then it is shown that if the sequence of distributions of the information spectrum satisfies the large deviation property with a certain rate function then the limiting guessing exponent exists and is a scalar multiple of the legendre fenchel dual of the rate function other sufficient conditions related to certain continuity properties of the information spectrum are briefly discussed this approach highlights the importance of the information spectrum in determining the limiting guessing exponent all known prior results are then re derived as example applications of our unifying approach

the shannon cipher system with a guessing wiretapper general sources the shannon cipher system is studied in the context of general sources using a notion of computational secrecy introduced by merhav and arikan bounds are derived on limiting exponents of guessing moments for general sources the bounds are shown to be tight for i i d markov and unifilar sources thus recovering some known results a close relationship between error exponents and correct decoding exponents for fixed rate source compression on the one hand and exponents for guessing moments on the other hand is established

a rate of convergence result for a universal d semifaithful code the problem of optimal rate universal coding is considered in the context of rate distortion theory a d semifaithful universal coding scheme for discrete memoryless sources is given the main result is a refined covering lemma based on the random coding argument and the method of types the average codelength of the code is shown to approach its lower bound the rate distortion function at a rate o n sup 1 log n and this is conjectured to be optimal based on a result of a j pilc 1968 issues of constructiveness and universality are addressed

exact correct decoding exponent of the wiretap channel decoder the performance of the achievability scheme for wyner s wiretap channel model is examined from the perspective of the probability of correct decoding pc at the wiretap channel decoder in particular for finite alphabet memoryless channels the exact random coding exponent of pc is derived as a function of the total coding rate r1 and the rate of each subcode r2 two different representations are given for this function and its basic properties are provided we also characterize the region of pairs of rates r1 r2 of full security in the sense of the random coding exponent of pc in other words the region where the exponent of this achievability scheme is the same as that of blind guessing at the eavesdropper side finally an analogous derivation of the correct decoding exponent is outlined for the case of the gaussian channel

joint source channel coding and guessing with application to sequential decoding we extend our earlier work on guessing subject to distortion to the joint source channel coding context we consider a system in which there is a source connected to a destination via a channel and the goal is to reconstruct the source output at the destination within a prescribed distortion level with respect to w r t some distortion measure the decoder is a guessing decoder in the sense that it is allowed to generate successive estimates of the source output until the distortion criterion is met the problem is to design the encoder and the decoder so as to minimize the average number of estimates until successful reconstruction we derive estimates on nonnegative moments of the number of guesses which are asymptotically tight as the length of the source block goes to infinity using the close relationship between guessing and sequential decoding we give a tight lower bound to the complexity of sequential decoding in joint source channel coding systems complementing earlier works by koshelev 1973 and hellman 1975 another topic explored here is the probability of error for list decoders with exponential list sizes for joint source channel coding systems for which we obtain tight bounds as well it is noteworthy that optimal performance w r t the performance measures considered here can be achieved in a manner that separates source coding and channel coding

on competitive prediction and its relation to rate distortion theory consider the normalized cumulative loss of a predictor f on the sequence x sup n x sub 1 x sub n denoted l sub f x sup n for a set of predictors g let l g x sup n min sub f spl isin g l sub f x sup n denote the loss of the best predictor in the class on x sup n given the stochastic process x x sub 1 x sub 2 we look at el g x sup n termed the competitive predictability of g on x sup n our interest is in the optimal predictor set of size m i e the predictor set achieving min sub g spl les m el g x sup n when m is subexponential in n simple arguments show that min sub g spl les m el g x sup n coincides for large n with the bayesian envelope min sub f el sub f x sup n we investigate the behavior for large n of min sub g spl les e sup nr el g x sup n which we term the competitive predictability of x at rate r we show that whenever x has an autoregressive representation via a predictor with an associated independent and identically distributed i i d innovation process its competitive predictability is given by the distortion rate function of that innovation process indeed it will be argued that by viewing g as a rate distortion codebook and the predictors in it as codewords allowed to base the reconstruction of each symbol on the past unquantized symbols the result can be considered as the source coding analog of shannon s classical result that feedback does not increase the capacity of a memoryless channel for a general process x we show that the competitive predictability is lower bounded by the shannon lower bound slb on the distortion rate function of x and upper bounded by the distortion rate function of any not necessarily memoryless innovation process through which the process x has an autoregressive representation thus the competitive predictability is also precisely characterized whenever x can be autoregressively represented via an innovation process for which the slb is tight the error exponent i e the exponential behavior of min sub g spl les exp nr pr l g x sup n d is also characterized for processes that can be autoregressively represented with an i i d innovation process