SlowGuess/NeurIPS_2023_OCR huggingface dataset:
So far, we get the variational bound for IBD:
$$
I B D = I (Z; Y) - \beta I (X; Z | T) \geq \underset {p (x, y) p (z | x)} {\mathbb {E}} \left[ \log q (y | z) - \beta \log \frac {p (z | x)}{q (z | t)} \right], \tag {31}
$$
C More Experimental Details and Results
C.1 The impact of the teacher.
Table 8: How would the performance of teachers affect that of student models?
Teacher	Natural	AA	Student	Natural	AA	Student	Natural	AA
Resnet-18	84.09	48.71	Resnet-18	83.74	50.52	WRN-34-10	84.41	53.94
Resnet-34	85.94	50.57	Resnet-18	84.92	49.84	WRN-34-10	85.79	54.17
WRN-34-10	84.92	53.08	Resnet-18	83.17	52.11	WRN-34-10	84.21	55.65
WRN-34-20	85.65	56.82	Resnet-18	82.82	51.64	WRN-34-10	84.73	55.71
WRN-76-10	88.54	64.25	Resnet-18	85.28	51.96	WRN-34-10	86.61	57.12
(a)
Figure 2: The impact of  $\alpha$  and  $\beta$ .
C.2 The impact of  $\alpha$  and  $\beta$ .
C.3 Different attack budgets and steps
We plot the results of testing robust accuracy over epochs and evaluate adversarial accuracy against PGD attacks under different attack budgets with a fixed attack step of 10, and we also conduct experiments using PGD attacks with different attack iterations with a fixed attack budget of 8/255. The results are shown in Figure 3. Our IBD is better than standard AT, TRADES and RSLAD at larger budgets, besides, our IBD is stable against large iterations attacks, e.g., PGD attack with 500 step iterations. Therefore, the results demonstrate the effectiveness of our proposed IBD.
Figure 3: The test robust accuracy under PGD attack with different attack budgets and attack iterations, respectively. All these experiments were conducted on CIFAR-10 using ResNet-18 architecture. (Best view in color)
C.4 Visualization of attention matrix
We visualize the attention matrices when distilling between different network architectures, as shown in Figure 4. We can see that models of the same architecture can achieve mutual correspondence between feature layers, but for models of different architectures, the shallow layers of the student model will receive a lot of attention, and the high-level features will correspond to each other.
D Limitations.
One limitation of our method is that the accuracy of our method on natural samples does not improve significantly. This may be because there exists a trade-off between robustness and natural accuracy [71]. Despite this, our natural accuracy is still over  $83 + \%$  on CIFAR10, we think this is acceptable. Another limitation is that our method requires a robust pre-trained teacher model, which may increase training costs and training time. At present, our IBD has only verified its effect on the adversarial
Figure 4: The visualization of attention matrix.
robustness of DNNs. It is not yet known whether it can achieve the same effect in other applications of IB. We are further studying its generalization ability.
E Broader Impacts.
We propose an adversarial defense method to enhance the robustness of the model, but we still need to be aware of the potential negative societal impacts it might result in. For example, the attacker can get our model and design a special attack algorithm for it. At present, we can not guarantee that our model can defend against stronger attack algorithms that may appear in the future. Thus, we encourage our machine learning community to further establish more reliable adversarial robustness checking routines for machine learning models deployed in safety-critical applications.