Information Theory from A Functional Viewpoint

A perennial theme of information theory is to find new methods to determine the fundamental limits of various communication systems, which potentially helps the engineers to find better designs by eliminating the deficient ones. Traditional methods have focused on the notion of “sets”: the method of types concerns the cardinality of subsets of the typical sets; the blowing-up lemma bounds the probability of the neighborhood of decoding sets; the single-shot (information-spectrum) approach uses the likelihood threshold to define sets. This thesis promotes the idea of deriving the fundamental limits using functional inequalities, where the central notion is “functions” instead of “sets”. A functional inequality follows from the entropic definition of an information measure by convex duality. For example, the Gibbs variational formula follows from the Legendre transform of the relative entropy. As a first example, we propose a new methodology of deriving converse (i.e. impossibility) bounds based on convex duality and the reverse hypercontractivity of Markov semigroups. This methodology is broadly applicable to network information theory, and in particular resolves the optimal scaling of the second-order rate for the previously open “side-information problems”. As a second example, we use the functional inequality for the so-called Eγ metric to prove non-asymptotic achievability (i.e. existence) bounds for several problems including source coding, wiretap channels and mutual covering. Along the way, we derive general convex duality results leading to a unified treatment to many inequalities and information measures such as the Brascamp-Lieb inequality and its reverse, strong data processing inequality, hypercontractivity and its reverse,transportation-cost inequalities, and Rényi divergences. Capitalizing on such dualities, we demonstrate information-theoretic approaches to certain properties of functional inequalities, such as the Gaussian optimality. This is the antithesis of the main thesis (functional approaches to information theory). iii Acknowledgements First, I thank my advisors Professor Paul Cuff and Professor Sergio Verdú for their unceasing guidance and support over the past five years. I was new to information theory at the start of my PhD, and Professor Cuff was generous with his time in helping getting on the road of this field. The puzzle meetings with Professor Cuff and other students was a great source of diversion and inspiration for our graduate life. Professor Verdú demonstrated to us the beauty of research and the charisma of teaching. He encouraged us to delve deeply into serious research topics, revise papers carefully, and work hard even on holidays, by doing so himself. Thanks, Prof. Abbe, Braverman, Cuff and Verdú, for serving as the committee; Prof. Chen, Poor, and Verdú, for being the readers; the administrative staff from ELE, for their assistance over the years. Thanks to all faculty members of Princeton who have taught me classes; what I learnt from them is reflected in every aspect of this thesis. I am indebted to Princeton University for providing a wonderful environment for studying and research. I owe sincere gratitude to my mentors and collaborators: Sanket Satpathy, Sudeep Kamath, Wei Yang for numerous illuminating discussions; Professor Emmanuel Abbe for letting me catch a glimpse of working on coding theory; Professor Thomas Courtade for showing me how to prove Gaussian optimality; Professor Ramon van Handel for advising me to look into semigroup operators, as opposed to the maximal operators which I was previously stuck with. I am thankful to all my friends at Princeton, my colleagues from the information sciences and systems group, especially my roommates, Sen Tao, Lanqing Yu and Amir Reza Asadi, for helping me out with many things in life. Last but not the least, I thank my parents for everything, and in particular for providing me a place where a large part of the thesis was written. iv