Information Structures for Feedback Capacity of Channels With Memory and Transmission Cost: Stochastic Optimal Control and Variational Equalities

The Finite Transmission Feedback Information (FTFI) capacity is characterized for any class of channel conditional distributions <inline-formula> <tex-math notation="LaTeX">${\mathbf{P}}_{B_{i}|B^{i-1}, A_{i}}$ </tex-math></inline-formula> and <inline-formula> <tex-math notation="LaTeX">${\mathbf{P}}_{B_{i}|B_{i-M}^{i-1}, A_{i}}$ </tex-math></inline-formula>, where <inline-formula> <tex-math notation="LaTeX">$M$ </tex-math></inline-formula> is the memory of the channel, <inline-formula> <tex-math notation="LaTeX">$B^{i} \stackrel {\triangle }{=}\{B^{-1},B_{0}, \ldots, B_{i}\}$ </tex-math></inline-formula> are the channel outputs, and <inline-formula> <tex-math notation="LaTeX">$A^{i} \stackrel {\triangle }{=}\{A_{0}, A_{1}, \ldots, A_{i}\}$ </tex-math></inline-formula>, are the channel inputs, for <inline-formula> <tex-math notation="LaTeX">$i=0, \ldots, n$ </tex-math></inline-formula>. The characterizations of FTFI capacity are obtained by first identifying the information structures of the optimal channel input conditional distributions <inline-formula> <tex-math notation="LaTeX">${\mathscr P}_{[0, n]} \stackrel {\triangle }{=}\big \{ {\mathbf{P}}_{A_{i}|A^{i-1}, B^{i-1}}: i=0, \ldots, n\big \}$ </tex-math></inline-formula>, which maximize directed information <inline-formula> <tex-math notation="LaTeX">$C_{A^{n} \rightarrow B^{n}}^{FB} \stackrel {\triangle }{=}\sup _{\mathscr P_{[0, n]} } I(A^{n} \rightarrow B^{n}), \hspace {. 2in} I(A^{n} \rightarrow B^{n}) \stackrel {\triangle }{=}\sum _{i=0}^{n} I(A^{i};B_{i}|B^{i-1})$ </tex-math></inline-formula>. The main theorem states that, for any channel with memory <inline-formula> <tex-math notation="LaTeX">$M$ </tex-math></inline-formula>, the optimal channel input conditional distributions occur in the subset satisfying conditional independence <inline-formula> <tex-math notation="LaTeX">$\stackrel {\circ }{\mathscr P}_{[0, n]} \stackrel {\triangle }{=}\big \{ {\mathbf{P}}_{A_{i}|A^{i-1}, B^{i-1}}= {\mathbf{P}}_{A_{i}|B_{i-M}^{i-1}}: i=0, \ldots, n\big \}$ </tex-math></inline-formula>, and the characterization of FTFI capacity is given by <inline-formula> <tex-math notation="LaTeX">$C_{A^{n} \rightarrow B^{n}}^{FB, M} \stackrel {\triangle }{=}\sup _{ \stackrel {\circ }{\mathscr P}_{[0, n]} } \sum _{i=0}^{n} I(A_{i}; B_{i}|B_{i-M}^{i-1})$ </tex-math></inline-formula>. Similar conclusions are derived for problems with average cost constraints of the form <inline-formula> <tex-math notation="LaTeX">$\frac {1}{n+1} {\mathbf{E}}\Big \{c_{0,n}(A^{n}, B^{n-1})\Big \} \leq \kappa,\,\, \kappa \geq 0$ </tex-math></inline-formula>, for specific functions <inline-formula> <tex-math notation="LaTeX">$c_{0,n}(a^{n},b^{n-1})$ </tex-math></inline-formula>. The feedback capacity is addressed by investigating <inline-formula> <tex-math notation="LaTeX">$\lim _{n \longrightarrow \infty } \frac {1}{n+1}C_{A^{n} \rightarrow B^{n}}^{FB, M} $ </tex-math></inline-formula>. The methodology utilizes stochastic optimal control theory, to identify the control process, the controlled process, and often a variational equality of directed information, to derive upper bounds on <inline-formula> <tex-math notation="LaTeX">$I(A^{n} \rightarrow B^{n})$ </tex-math></inline-formula>, which are achievable over specific subsets of channel input conditional distributions <inline-formula> <tex-math notation="LaTeX">${\mathscr P}_{[0, n]}$ </tex-math></inline-formula>, which are characterized by conditional independence. The main results illustrate a direct analogy, in terms of conditional independence, of the characterizations of FTFI capacity and Shannon’s capacity formulae of memoryless channels. An example is presented to illustrate the role of optimal channel input process in the derivations of the direct and converse coding theorems.