SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\alpha$  and  $\beta$  are two trade-off hyper-parameters. We generate adversarial examples using the same method introduced by [71], where given a natural input  $x_{\mathrm{nat}}$ , generated the adversarial example  $x_{\mathrm{adv}}$  by maximizing KL-divergence term. Note that the intermediate features  $Z_{t}$  and  $Z_{s}$  are extracted from  $x_{\mathrm{adv}}$ . Finally, we utilize this new loss function to train a robust model.
The objective of IBD is comparable to certain conventional KD techniques [6, 10, 32]. However, the major difference is, that in our IBD, the teacher model must be an adversarially pre-trained model that can provide robust information. Additionally, we design the objective function following the
Table 1: Robustness comparison of the proposed approach and baseline models under different attack methods under the  $\ell_{\infty}$  norm with  $\epsilon = 8 / 255$  on different datasets. All the models are based on pre-activation ResNet-18 architecture. We choose the best checkpoint according to the highest robust accuracy on the test set under PGD-10. The best results are blodfaced.
Method	Best Checkpoint	Last Checkpoint
Clean	FGSM	PGD	CW	AA	Clean	FGSM	PGD	CW	AA
CIFAR-10 - linf - ε = 8/255
SAT [41]	82.97	57.77	50.85	50.09	47.73	85.16	53.97	43.03	43.71	41.58
TRADES [71]	83.74	59.54	52.73	50.94	49.58	84.11	58.72	49.97	49.05	47.02
CEB [19]	82.87	58.61	52.94	50.22	48.87	83.39	57.74	48.79	47.67	45.85
HBaR [64]	84.13	58.95	53.35	51.47	49.77	85.04	58.33	53.19	51.21	48.64
InfoAT [68]	83.17	60.52	54.29	51.62	49.92	83.23	60.40	53.94	51.17	49.67
ARD [21]	83.94	59.32	52.16	51.21	49.17	84.32	59.35	51.41	51.22	48.89
IAD [73]	83.24	59.34	54.24	51.92	50.63	83.76	59.17	53.84	51.60	50.17
RSLAD [74]	83.38	60.08	54.27	53.19	51.52	83.88	59.98	54.01	53.08	51.36
IBD (Ours)	83.17	60.75	55.13	53.62	52.11	82.95	60.94	55.04	53.49	52.05
CIFAR-100 - linf - ε = 8/255
SAT [41]	57.75	32.78	29.27	27.52	24.01	58.17	27.02	21.02	21.54	19.86
TRADES [71]	58.57	32.84	29.88	26.29	25.27	57.11	32.05	28.12	25.47	24.52
CEB [19]	55.17	32.36	30.12	26.35	25.36	55.74	30.84	26.59	24.51	23.16
HBaR [64]	59.53	34.46	31.82	27.42	26.62	58.26	32.41	28.23	25.78	25.43
InfoAT [68]	58.23	34.46	31.39	28.68	26.76	58.42	33.13	30.53	27.52	26.21
ARD [21]	60.58	33.43	29.07	27.54	25.62	60.79	32.67	28.11	26.76	24.62
IAD [73]	57.08	34.65	30.60	27.24	25.84	57.52	33.71	29.23	27.35	25.36
RSLAD [74]	57.72	34.23	31.01	28.27	26.73	57.83	34.09	30.55	28.07	26.41
IBD (Ours)	58.10	36.37	33.59	31.16	29.21	58.32	36.17	33.40	30.87	28.74
information bottleneck principle, which is theoretically proven to be a lower bound of the information bottleneck. Notably, previous adversarial distillation methods [21, 44, 74] only consider utilizing information from the final prediction output, while ignoring the information from intermediate features. From another perspective, motivated by [31], IBD can explore intermediate features to capture the robust information and leverage an adaptive feature distillation strategy to automatically transfer appropriate features from the teacher model to the target student model. Our experiments indicate that making full use of the intermediate features of the DNNs can improve the model's robustness effectively.
In this section, we first describe the experimental setting and implementation details. We evaluate the robustness and accuracy of various widely used benchmark datasets. Finally, we conduct a wealth of ablation experiments to provide a comprehensive understanding of the proposed IBD.
Baselines Setup. We conduct our experiments on three benchmark datasets including CIFAR-10, CIFAR-100 [36] and ImageNet [17]. We consider three types of baseline model: 1) classical adversarial training method, including standard Standard AT[41] and TRADES [71]; 2) adversarial robustness distillation, including ARD [21], IAD [73] and RSLAD [74]; 3) the models trained using variants IB, include CEB [19], HBaR [64] and InfoAT [68].
Evaluation Attack. We evaluate the defense under different white- and black-box attacks including FGSM [22], PGD [41], CW [4], and AutoAttack [14]. The black-box attacks include query-based
Table 2: The performance of the pre-trained teacher models under different attacks.
Dataset	Architecture	Natural Acc	FGSM	PGD-100	CW-100	AutoAttack
CIFAR-10	WideResNet-34-10	84.92	60.87	55.33	53.98	53.08
CIFAR-100	WideResNet-34-10	57.16	33.58	30.61	27.74	26.78
ImageNet-1K	ResNet-50	64.02	-	38.46	-	34.96
Table 3: Black-box robustness results in CIFAR-10 under the  $\ell_{\infty}$  norm with  $\epsilon = 8 / 255$ . We evaluate against transfer-based and query-based attacks.
Method	Trans-based	Query-based
FGSM	PGD-100	SPSA	Square
SAT[41]	61.34	59.83	66.35	54.16
TRADES[71]	63.14	60.31	67.56	54.74
CEB[19]	62.75	61.93	66.71	55.28
HBaR[64]	63.61	60.93	68.07	56.62
InfoAT [68]	64.23	62.42	68.73	57.35
ARD [21]	63.27	61.44	67.89	55.85
IAD [73]	63.53	62.13	68.42	56.21
RSLAD[74]	64.78	62.13	68.78	56.97
IBD	65.54	63.98	69.63	58.34
Table 4: Robustness comparison of the proposed IBD and several state-of-the-art models under standard AutoAttack.
Method	WRN	Clean	AA
SAT [41]	34-10	84.92	53.08
LBGAT [15]	34-20	88.70	53.57
TRADES [43]	34-20	86.18	54.39
LTD [7]	34-10	85.02	54.45
IBD	34-10	83.33	55.65
TRADES + AWP [66]	34-10	85.26	56.17
LASAT + AWP [33]	34-10	84.98	56.26
LTD + AWP [7]	34-10	86.28	56.94
IBD + AWP	34-10	85.21	57.18
attacks [1] and transfer-based attacks [59]. The maximum perturbation is set to  $\epsilon = 8 / 255$  for CIFAR-10 and CIFAR-100.
Implementation Details. For the robust pre-trained teacher models on the CIFAR dataset, we use the ResNet-18 [26] and WideResNet-34x10 [70] models trained with a way of TRADES [71] and AWP [66]. For the ImageNet dataset, the pre-trained teacher model (a ResNet-50) is provided by [49]. The results of the pre-trained teacher models under different attacks are shown in Table 8. The initial learning rate is 0.1 with a piece-wise schedule which is divided by 10 at epochs 100 and 150 for a total number of 200 training epochs, similar to [47]. We train all models with the SGD optimizer with a momentum of 0.9, weight decay of 0.0005, and a batch size of 128. We use ResNet-18 as the student model for most experiments by default, unless otherwise stated. We adopt the common setting that the  $\ell_{\infty}$  threat model with radius  $8 / 255$  , with the PGD attack taking 10 steps of size  $2 / 255$  . In addition, we performed standard data augmentation, including random crops and random horizontal flips during training. For the hyper-parameter, we set  $\alpha = 0.9$  and  $\beta = 0.8$  based on our ablation studies. For more details please refer to our open source code. Our implementation is based on PyTorch and the code to reproduce our results is available at https://github.com/SkyKuang/IBD.
4.2 Adversarial Robustness Evaluation
White-box Robustness. To verify the impact of the IBD on model robustness, we first train a natural model (without adversarial training), We evaluate the performance of IBD on CIFAR-10 under ResNet-18, and IBD achieves  $25.49\%$  robust accuracy against standard AA. However, other IB-based methods (CEB and HBaR) without adversarial training can not defend AA (that means the robust accuracy is 0). We further combine IBD with adversarial training and evaluate the robustness of all baseline models and our IBD against various types of attacks. Following in [47], we report the results at both the best checkpoint and the last checkpoint. The best checkpoint is selected based on the performance under the PGD-10 attack. The results are shown in Table 1. Our IBD method shows better robustness of both CIFAR-10 and CIFAR-100 against all attacks at either the best or the last checkpoints. In particular, our IBD improves the robustness by  $0.59\%$  and  $2.48\%$  on CIFAR-10 and CIFAR-100 respectively, compared to previous state-of-the-art methods under AutoAttack.
Black-box Robustness. To further verify the robustness of our method under black-box attacks, we evaluate our IBD and baseline methods against transfer-based attacks and query-based attacks. For transfer-based attacks, we choose a robust surrogate model, which is trained by standard adversarial training, to generate the adversarial examples. All the models are trained using the same setting
(referring to Section 4.1). We generate adversarial examples using both FGSM and PGD-100 attacks with attack budget  $\epsilon = 8 / 255$  on CIFAR-10. For query-based attacks, we evaluate the robustness under SPSA attack [60] and Square attack [13]. SPSA attack can make a full gradient evaluation by drawing random samples and obtaining the corresponding loss values. We set the number of random samples  $q$  as 128 for every iteration and  $\delta = 0.01$  and set iteration as 10. The square attack uses random search and does not exploit any gradient approximation to produce adversarial noise, and we set the maximum number of queries as 5000. We evaluate both transfer-based and query-based attacks on the best checkpoints. The results are shown in Table 3. We observe that the IBD-trained model is robust to different types of black-box attacks and surpasses all baseline methods. The results under the black-box attack demonstrate the robustness improvement of IBD is not caused by the obfuscated gradients.
Robustness Evaluation on WideResNet. Many works have demonstrated larger model capacity can usually lead to better adversarial robustness [23, 41, 43]. Therefore, we employ the large-capacity network, e.g., WideResNet (WRN) [70], as the student model. Table 4 reports the robustness results against AA on the CIFAR-10. We compare several state-of-the-art adversarial trained models on robust benchmark [12]. We can observe that the proposed IBD indeed improves the adversarial robustness by  $\sim 1.2\%$ . Furthermore, when combined with AWP [66], our IBD also surpasses the previously state-of-the-art models reported by the benchmark. where every small margin of improvement is significant. Note that, our method does not use any additional datasets.
Evaluation on Large-scale ImageNet. To further verify the generalization of our method, we consider a high-resolution, large-scale ImageNet dataset[17], which includes 1,000 classes and more than 1M training samples. Maintaining robustness for this dataset is particularly challenging. We use the fast adversarial training framework [65] to train all robust models. Here, the robust budgets are set as  $\epsilon = 2/255$  and  $\epsilon = 4/255$ . The results of our method for the ResNet-50 (student model) are shown in Table 5. Our IBD outperforms both Fast-AT [62] and RSLAD [74], significantly.
Table 5: Robustness comparison of the proposed approach and different fast adversarial training under different attack methods at  $\epsilon = 2/255$  and  $\epsilon = 4/255$  on ImageNet-1k. All the models are based on ResNet-50 architecture.
Method	Epsilon	Clean	PGD-100	AutoAttack
FAST-AT [65]	ε = 2	65.95	37.52	35.22
RSLAD [74]	ε = 2	63.64	40.49	38.35
FAST-IBD	ε = 2	62.03	44.31	40.94
FAST-AT [65]	ε = 4	60.16	27.46	24.82
RSLAD [74]	ε = 4	60.64	30.19	26.53
FAST-IBD	ε = 4	59.10	31.52	27.74
The importance of robust soft label. [52] proposes leveraging label smoothing during adversarial training, and [74] also determines the robust soft label is an important factor in robustness enhancement. We empirically verify the importance of robust soft labels in our IBD, by comparing the performance of models trained using different label modification schemes: a) true label (hard label); b) smoothing label crafted by label smoothing on the true label [55]; c) natural soft label that output by non-robust pre-trained model; 4) robust soft label that output by robust pre-trained model. The results are summarized in Table 6, which demonstrates that the robust soft label is beneficial to improving the model's robustness.
The role of features matching. We further investigate the role of features distillation strategy in section 3.2, we consider two features matching strategies: a) a hand-crafted feature distillation method, which uses the same level of features to guide the student model learning; b) an adaptive feature distillation method, which is based on attention mechanism to guide the student model learning. We conduct experiments with these two strategies on different network architectures ResNet-18 (Res) and WideResNet-34x10 (WRN). The results are reported in Table 7. Our adaptive feature
Table 6: Robustness of our IBD model trained using different types of labels on CIFAR-10 under different attacks.
Method	Clean	PGD	CW	AA
Hard label	85.25	51.64	50.49	48.35
Label smooth	85.43	51.92	50.56	48.71
Natural soft label	86.38	48.08	45.70	43.79
Robust soft label	83.17	55.13	53.62	52.11
RSLAD [74]	83.38	54.27	53.19	51.52
Table 7: Robustness of our IBD model trained using different features matching schedule on CIFAR-10 under the AutoAttack.
Architecture	Schedule	Clean	AA
Res → Res	No matching	83.25	50.63
WRN → Res	No matching	83.37	51.48
Res → Res	Hand-crafted	82.99	51.12
WRN → Res	Hand-crafted	83.10	51.61
Res → Res	Adaptive	83.24	51.46
WRN → Res	Adaptive	83.17	52.11
distillation performs better than hand-crafted feature distillation in both the same and different network architectures.
The impact of the teacher. We conducted an ablation experiment by using different teacher models to verify the impact of the teacher's robustness on the performance of the student model. We conduct this experiment on CIFAR-10 with two student models: ResNet-18 and WideResNet-34-10, and five different teacher models which have different robustness. The results are shown in Table 8 in the Appendix. We can observe that different robust teacher models have a significant positive benefit on the student model. For the ResNet-18 student model, we find that the robustness of the student does not increase monotonically with that of the teacher. As the teacher model (WideResNet-34-20) becomes more complex, the robustness of the student model decreases, compared to WideResNet-34-10. This may be due to the large gap in the architecture of the teacher model and the student model. This phenomenon is called robust saturation [74]. For the WideResNet-34-10 student model, we found that in most cases, the student's robustness can surpass that of the teacher model. We think there are two reasons for this, one is that the performance of the teacher model is not very strong. The other is that the teacher model provides robust soft labels to alleviate overfitting and improve performance. Therefore, in most cases, it is expected that the student model exceeds the teacher model, but when the teacher model is strong enough, it is not easy for the student model to surpass the teacher model (e.g., WideResNet-76-10).
The impact of  $\alpha$  and  $\beta$ . The  $\alpha$  is a trade-off the adversarial robustness and natural accuracy. We conduct ablation experiments to verify the trade-off. The results are shown in Figure 2(a) in the Appendix. When we set  $\alpha = 0.9$ , our method can achieve the best adversarial robustness. In the IB principle, the hyperparameters  $\beta$  are important to control the trade-off. Therefore, we finally choose different values of  $\beta$  to train the IBD, from which we choose the optimal value of  $\beta$ . The experimental results are shown in Figure 2(b) in the Appendix, where the best results are achieved when setting  $\beta = 0.8$ .
In this paper, we revisited the information bottleneck principle from the perspective of robustness distillation and then presented a new IB objective, called Information Bottleneck Distillation (IBD). IBD can be thought of as a tighter variational approximation to the IB objective than VIB. To optimize IBD effectively, we proposed two adaptive distillation strategies. Experimentally, we empirically demonstrated the advantage of IBD over existing methods, IB-based methods, and adversarial distillation methods on benchmark datasets.
This work was supported by National Key R&D Program of China (No.2022ZD0118202), the National Science Fund for Distinguished Young Scholars (No.62025603), the National Natural Science Foundation of China (No. U21B2037, No. U22B2051, No. 62176222, No. 62176223, No. 62176226, No. 62072386, No. 62072387, No. 62072389, No. 62002305 and No. 62272401), and the Natural Science Foundation of Fujian Province of China (No.2021J01002, No.2022J06001).
[59] Florian Tramér, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel. Ensemble adversarial training: Attacks and defenses. In ICLR, 2018.
[60] Jonathan Uesato, Brendan O'donoghue, Pushmeet Kohli, and Aaron Oord. Adversarial risk and the dangers of evaluating against weak attacks. In ICML, pages 5025-5034. PMLR, 2018.
[61] Ashish Vavwani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[62] Xiaolong Wang, Yufei Ye, and Abhinav Gupta. Zero-shot recognition via semantic embeddings and knowledge graphs. In CVPR, pages 6857-6866, 2018.
[63] Zekai Wang, Tianyu Pang, Chao Du, Min Lin, Weiwei Liu, and Shuicheng Yan. Better diffusion models further improve adversarial training. ICML, 2023.
[64] Zifeng Wang, Tong Jian, Aria Masoomi, Stratis Ioannidis, and Jennifer Dy. Revisiting hilbert-schmidt information bottleneck for adversarial robustness. Advances in Neural Information Processing Systems, 34, 2021.
[65] Eric Wong, Leslie Rice, and J Zico Kolter. Fast is better than free: Revisiting adversarial training. In ICLR, 2020.
[66] Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust generalization. In NeurIPS, 2020.
[67] Lei Wu, Zhanxing Zhu, Cheng Tai, et al. Understanding and enhancing the transferability of adversarial examples. arXiv preprint arXiv:1802.09707, 2018.
[68] Mengting Xu, Tao Zhang, Zhongnian Li, and Daoqiang Zhang. Infoat: Improving adversarial training using the information bottleneck principle. IEEE Transactions on Neural Networks and Learning Systems, 2022.
[69] Yuzhe Yang, Guo Zhang, Dina Katabi, and Zhi Xu. Me-net: Towards effective adversarial robustness with matrix estimation. In ICML, 2019.
[70] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.
[71] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P Xing, Laurent El Ghaoui, and Michael I Jordan. Theoretically principled trade-off between robustness and accuracy. In ICML, 2019.
[72] Jingfeng Zhang, Xilie Xu, Bo Han, Gang Niu, Lizhen Cui, Masashi Sugiyama, and Mohan Kankanhalli. Attacks which do not kill training make adversarial learning stronger. In ICML, pages 11278-11287. PMLR, 2020.
[73] Jianing Zhu, Jiangchao Yao, Bo Han, Jingfeng Zhang, Tongliang Liu, Gang Niu, Jingren Zhou, Jianliang Xu, and Hongxia Yang. Reliable adversarial distillation with unreliable teachers. arXiv preprint arXiv:2106.04928, 2021.
[74] Bojia Zi, Shihao Zhao, Xingjun Ma, and Yu-Gang Jiang. Revisiting adversarial robustness distillation: Robust soft labels make student better. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16443-16452, 2021.
A Variational Information Bottleneck
We show the derivation process of Variational Information Bottleneck (VIB), which is provided by [2, 35]. The information bottleneck (IB) [57] expresses a trade-off in intermediate features  $Z$  between information useful for output prediction  $Y$  and information retained about the input  $X$ . The objective of IB can be formulated as follows:
$$
I B (\theta) = \max  I (Z; Y) - \beta I (Z; X), \tag {17}
$$
where  $I$  denotes mutual information and  $\beta$  controls the trade-off between the two terms.
The first term can be expressed as follows:
$$
\begin{array}{l} I (Z; Y) = \int p (y, z) \log \frac {p (y , z)}{p (y) p (z)} d y d z \tag {18} \\ = \int p (y, z) \log \frac {p (y | z)}{p (y)} d y d z \\ \end{array}
$$
To apply it to a deep neural network  $h_\theta = (g * f)$ , The  $q(y|z)$  is modelled by the classifier  $g$ . Based on variational inference, it is a closed form for a true likelihood  $p(y|z)$ . This approximation is formulated by KL divergence and then the following inequality is constructed as follows:
$$
K L \left(p (Y | Z) \| q (Y | Z)\right) \geq 0 \quad \Rightarrow \quad \int p (y | z) \log p (y | z) d y \geq \int p (y | z) \log q (y | z) d y \tag {19}
$$
which helps to get the objective of IB to be tractable. With this equality, the first term in Eq. (17) can be represented to a lower bound as:
$$
\begin{array}{l} I (Z; Y) \geq \int p (y, z) \log \frac {q (y | z)}{p (y)} d y d z \\ = \int p (y, z) \log q (y | z) d y d z - \int p (y, z) \log p (y) d y d z \\ = \int p (y, z) \log q (y | z) d y d z - \int p (y) \log p (y) d y \tag {20} \\ = \int p (y, z) \log q (y | z) d y d z + H (Y) \\ \geq \int p (y, z) \log q (y | z) d y d z = \mathbb {E} _ {p (x, y) p (z | x)} [ \log q (y | z) ] \\ \end{array}
$$
where a positive constant  $H(Y)$  denotes the Shannon entropy of target labels.
The second term in Eq. (17) is described as :
$$
\begin{array}{l} I (Z; X) = \int p (z, x) \log \frac {p (z , x)}{p (x) p (z)} d z d x \tag {21} \\ = \int p (z, x) \log \frac {p (z \mid x)}{p (z)} d z d x \\ \end{array}
$$
where a dataset probability  $p(x)$  is erased on the fraction. Here, an approximate feature probability  $q(Z)$  is introduced to appropriate the true feature probability  $p(Z)$ . As similar to Eq.(19), the relationship between  $q(Z)$  and  $p(Z)$  can be written and then it builds the following equality:
$$
K L \left(p (Z) \| q (Z)\right) \geq 0 \Rightarrow \int p (z) \log p (z) d z \geq \int p (z) \log q (z) d z \tag {22}
$$
By using it, the second term is constructed with an upper bound as follows:
$$
\begin{array}{l} I (Z; X) \leq \int p (z, x) \log \frac {p (z \mid x)}{q (z)} d z d x \\ = \int p (x) p (z \mid x) \log \frac {p (z \mid x)}{q (z)} d z d x \tag {23} \\ = \mathbb {E} _ {p (x, y) p (z | x)} \left[ \log \frac {p (z | x)}{q (z)} \right] \\ \end{array}
$$
where a feature likelihood is denoted by  $p(z|x)$ . To sum it up, the objective of IB can be re-formulated with a lower bound as follows:
$$
I (Z; Y) - \beta I (Z; X) \geq \mathbb {E} _ {p (x, y) p (z | x)} \left[ \log q (y | z) - \beta \log \frac {p (z | x)}{q (z)} \right], \tag {24}
$$
B Variational Bound on IBD
In the IBD setting, there is a target model and a pre-trained teacher model. We would like to learn features  $Z$  of  $X$  that will be useful for predicting  $Y$  for the target model.  $T$  is the intermediate features extracted by the adversarially pre-trained teacher model. We can represent this problem setting with the Markov chain:  $Z \gets X \rightarrow T$ . Given the conditional independence  $Z \perp T|X$  in our Markov chain and the conditional mutual information is always non-negative [20], so learning intermediate features  $Z$  of  $X$  is equivalent to minimizing  $I(X;Z|T)$ . Using the chain rule of mutual information, we have:
$$
I (X; Z \mid T) = I (X, T; Z) - I (T; Z) = I (X; Z) - I (T; Z) \tag {25}
$$
The objective of IBD can be formulated as:
$$
I B D = I (Z; Y) - \beta I (Z; X | T) \tag {26}
$$
$$
= H (Y) + H (Y | Z) - \beta \left(H (Z) - H (Z | X) - H (Z) + H (Z | T)\right) \tag {27}
$$
$$
= H (Y | Z) - \beta \left(H (Z | T) - H (Z | X)\right) \tag {28}
$$
We will variationally lower bound the first term of Eq. (26) and upper bound the second term using three distributions:  $p(z|x)$  is the feature distribution;  $q(z|t)$  is an approximation of  $p(z|y)$ ; and  $q(y|z)$  is an approximation of  $p(y|z)$ .
The first term of Eq. (26):
$$
\begin{array}{l} I (Z; Y) = H (Y) - H (Y | Z) \\ = \underset {p (x, y) p (z | x)} {\mathbb {E}} \left[ \log q (y | z) + K L \left(p (y | z) \| q (y | z)\right) \right] \tag {29} \\ \geq \underset {p (x, y) p (z | x)} {\mathbb {E}} \left[ \log q (y | z) \right]. \\ \end{array}
$$
The second term of Eq. (26) (for convenience, we set  $\beta$  to 1):
$$
\begin{array}{l} I (X; Z | T) = H (Z | T) - H (Z | X) \\ = \underset {p (x) p (z | x)} {\mathbb {E}} \left[ \log p (z | x) - \log q (z | t) - K L \left(p (z | y) \mid \mid q (z | t)\right) \right] \tag {30} \\ \leq \underset {p (x) p (z | x)} {\mathbb {E}} \left[ \log \frac {p (z | x)}{q (z | t)} \right]. \\ \end{array}
$$