SlowGuess/NAACL_2022_OCR huggingface dataset:
$$
\Leftrightarrow 2 H \left(\boldsymbol {z} _ {1}, \boldsymbol {z} _ {2}\right) + H \left(\boldsymbol {x}, \boldsymbol {z} _ {1}\right) + H \left(\boldsymbol {x}, \boldsymbol {z} _ {2}\right) \geq H \left(\boldsymbol {z} _ {1}\right) + H \left(\boldsymbol {z} _ {2}\right) + 2 H \left(\boldsymbol {x}, \boldsymbol {z} _ {1}, \boldsymbol {z} _ {2}\right) \tag {22}
$$
It's true because we have:
$$
\begin{array}{l} H \left(\boldsymbol {z} _ {1}, \boldsymbol {z} _ {2}\right) + H \left(\boldsymbol {x}, \boldsymbol {z} _ {1}\right) \\ = H \left(\boldsymbol {z} _ {2} \mid \boldsymbol {z} _ {1}\right) + H \left(\boldsymbol {x} \mid \boldsymbol {z} _ {1}\right) + 2 H \left(\boldsymbol {z} _ {1}\right) \tag {23} \\ \geq H \left(\boldsymbol {x}, \boldsymbol {z} _ {2} \mid \boldsymbol {z} _ {1}\right) + 2 H \left(\boldsymbol {z} _ {1}\right) \\ = H \left(\boldsymbol {x}, \boldsymbol {z} _ {1}, \boldsymbol {z} _ {2}\right) + H \left(\boldsymbol {z} _ {1}\right) \\ \end{array}
$$
Similarly, the following inequality also holds true:
$$
H \left(\boldsymbol {z} _ {1}, \boldsymbol {z} _ {2}\right) + H \left(\boldsymbol {x}, \boldsymbol {z} _ {2}\right) \geq H \left(\boldsymbol {x}, \boldsymbol {z} _ {1}, \boldsymbol {z} _ {2}\right) + H \left(\boldsymbol {z} _ {2}\right) \tag {24}
$$
Therefore, making sum to Eq.(23) and Eq.(24), we conclude that  $I(\pmb{x}; \pmb{z}_1, \pmb{z}_2) \geq I(\pmb{x}; \pmb{z}_1; \pmb{z}_2)$ . Hence, we finish the proof of the  $L = 2$  case.
When  $L = k$ , suppose  $I(\pmb{x};\pmb{z}_1,\dots ,\pmb{z}_k) \geq I(\pmb{x};\pmb{z}_1;\dots ;\pmb{z}_k)$ , we consider  $L = k + 1$ . In this case, based on the inductive assumption, we have:
$$
I (\boldsymbol {x}; \boldsymbol {z} _ {1}, \dots , \boldsymbol {z} _ {k + 1}) \geq I (\boldsymbol {x}; \boldsymbol {z} _ {1}, \dots , \boldsymbol {z} _ {k}) \geq I (\boldsymbol {x}; \boldsymbol {z} _ {1}; \dots ; \boldsymbol {z} _ {k}) \geq I (\boldsymbol {x}; \boldsymbol {z} _ {1}; \dots ; \boldsymbol {z} _ {k + 1}) \tag {25}
$$
Hence, the case of  $L = k + 1$  also holds true. Therefore, we conclude that  $I(\pmb{x};\pmb{z}_1,\dots,\pmb{z}_L) \geq I(\pmb{x};\pmb{z}_1;\dots;\pmb{z}_L)$ .
Now, we consider the interaction information and can obtain:
$$
\begin{array}{l} I (\boldsymbol {x}; \boldsymbol {z} _ {1}; \dots ; \boldsymbol {z} _ {L}) \\ = I \left(\boldsymbol {z} _ {L}, \boldsymbol {z} _ {L - 1}\right) - \sum_ {i = 2} ^ {L - 1} I \left(\boldsymbol {z} _ {L}; \dots ; \boldsymbol {z} _ {i} \mid \boldsymbol {z} _ {i - 1}\right) - I \left(\boldsymbol {z} _ {L}; \dots ; \boldsymbol {z} _ {1} \mid \boldsymbol {x}\right) \tag {26} \\ \geq \sum_ {i = 2} ^ {L - 1} I \left(\boldsymbol {z} _ {L}; \dots ; \boldsymbol {z} _ {i} | \boldsymbol {z} _ {i - 1}\right) - I \left(\boldsymbol {z} _ {L}; \dots ; \boldsymbol {z} _ {1} | \boldsymbol {x}\right) \\ \end{array}
$$
Finally, based on Eq.(16), (17), (25), (26), we can conclude:
$$
\begin{array}{l} \mathbb {E} _ {p (\boldsymbol {x})} [ \mathcal {L} _ {R} ] = \sum_ {i = 1} ^ {L} \mathbb {E} _ {p (\boldsymbol {x})} \mathbb {E} _ {q (\boldsymbol {z} _ {<   l} | \boldsymbol {x})} [ \mathrm {K L} (q (\boldsymbol {z} | \boldsymbol {x}, \boldsymbol {z} _ {<   l}) | | p (\boldsymbol {z} _ {l} | \boldsymbol {z} _ {<   l})) ] \\ \geq I (\boldsymbol {x}; \boldsymbol {z} _ {1}, \dots , \boldsymbol {z} _ {L}) \\ \geq I \left(\boldsymbol {x}; \boldsymbol {z} _ {1}; \dots ; \boldsymbol {z} _ {L}\right) \tag {27} \\ \geq \sum_ {i = 2} ^ {L - 1} I \left(\boldsymbol {z} _ {L}; \dots ; \boldsymbol {z} _ {i} | \boldsymbol {z} _ {i - 1}\right) - I \left(\boldsymbol {z} _ {L}; \dots ; \boldsymbol {z} _ {1} | \boldsymbol {x}\right) \\ \end{array}
$$
Figure 6: Attention weight of the Memory paradigm for layer 0 to layer 5. We plot three heatmaps in each layer. Average means averaging weights through all head. Max and min means we select the head with max and min attention weight on the memory token (latent variable). We can see the memory token tends to be ignored by most heads especially in lower layers. 715
Figure 7: Attention weight of Memory paradigm for layer 6 to layer 11.