Mathematics of information and coding Machine generated contents note: Preface to the English Edition xi Chapter 1. What is Information Theory 1 1.1. Introduction 1 1.2. Information Theory So Far 2 1.3. Basic Elements and Related Fields 6 1.4. About this Book 16 Chapter 2. Basics of Information Theory 19 2.1. Entropy 19 2.2. Divergence 25 2.3. Mutual Information 31 2.4. Data Processing Inequality 37 2.5. Fanos Inequality 40 2.6. Type of Sequences Typical Sequences and AEP 41 Exercises 51 Chapter 3. Source and Coding 73 3.1. Various Information Sources 73 3.2. Entropy Rate of Source 79 3.3. Various Codes 84 3.4. AEP and Source Coding Theorem 86 3.5. Separable Codes and Prefix Codes 88 3.6. Converse Theorem for FV Coding 93 3.7. Direct Theorem for FV Coding 95 3.8. Huffman Code 99 3.9. Coding for Arbitrarily Varying Sources 106 3.10. Reliability Function for Source Coding 109 Exercises 115 Chapter 4. Arithmetic Code 127 4.1. Elias Code 127 4.2. Algorithm of Arithmetic Code 131 4.3. Asymptotic Optimality of Arithmetic Code 139 4.4. Arithmetic Code as Number System 142 4.5. Enumerative Code 145 4.6. Arithmetic Code as Tree Code 148 Exercises 151 Chapter 5. Universal Coding of Integers 157 5.1. Universal Codes of Integers 157 5.2. Search Tree of Bentley Yao 166 5.3. Bounds for Elias Codeword Length Functions 170 5.4. Properties of the Log Star Function 180 5.5. Modified Log Star Functions 182 Exercises 188 Chapter 6. Universal Coding of Texts 193 6.1. Lempel Ziv Code based on Incremental Parsing 193 6.2. Lempel Ziv Code based on Recurrence Time 202 Exercises 214 Chapter 7. Universal Coding of Compound Sources 219 7.1. Compound Source 219 7.2. Sub Probability and Codeword Length 220 7.3. Universal Code for Compound Source 222 7.4. Estimation of Probability Distribution 223 7.5. Rissanens Two Step Coding 225 7.6. Rissanens Predictive Coding 231 7.7. Stochastic Complexity 233 7.8. Bayes Code 234 7.9. Model Selection and Universal Code 239 7.10. Consistency of Model Selection 242 Exercises 248 Chapter 8. Data Analysis and MDL Principle 253 8.1. Fitting of Experimental Data 254 8.2. Histogram Approximation for Probability Density 257 8.3. Self Organization of Data 258 8.4. Self Formation of Category 262 8.5. Category Classification by Decision Tree 263 8.6. Probabilistic Category Classification 266 8.7. About AIC Criterion 269 Exercises 273 Bibliography 277 Index 283