SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\tilde {P} _ {i} = \left\{ \begin{array}{l l} \mathbf {e} _ {k} & k = \arg \max  _ {j} P _ {i, j} ^ {\prime}, \quad P _ {i, k} ^ {\prime} > \alpha \\ P _ {i} ^ {\prime} & o. w. \end{array} \right. \tag {5}
$$
where  $\mathbf{e}_k\in \{0,1\} ^C$  is the one-hot vector and the  $k$  -th element is 1.
The proposed intra-modal proxy learning method (InMaP) is summarized in Alg. 1. Pre-trained CLIP models are only applied to extract features as in Step 2, which is the inevitable cost of conventional zero-shot learning. After obtaining features, the computational overhead of our proposed optimizations in Steps 3-5 is mild as shown in the analysis of running time.
To evaluate the proposed method, we follow the common practice and conduct experiments on ImageNet [28] and 13 diverse downstream vision tasks for zero-shot transfer. Text prompts are
important for obtaining appropriate text proxies. We have the selected 7 prompts in [33] as templates for ensemble, which shows better performance than a single prompt or multiple prompts in CLIP. For the proposed method, the temperature for obtaining pseudo labels with the text proxy  $\tau_{T}$  is set to 0.01 as obtained by CLIP. The temperature for recovering the visual proxy  $\tau_{I}$  is set to 0.04 for all experiments. The intra-modal proxy is learned by standard projected gradient descent, where the initial learning rate is 10 and the number of iterations is 2,000 for sufficient training. The learning rate will be decayed by 2 when the norm of gradient increases. Sinkhorn distance is optimized by 20 iterations for refining pseudo labels. All experiments are conducted on a single V100 GPU.
First, we have the ablation experiments on ImageNet to demonstrate the efficacy of our method InMaP. Pre-trained ResNet-50 [13] is adopted as the vision encoder and the vanilla zero-shot method in CLIP is denoted as "Baseline".
Temperature  $\tau_{I}$ $\tau_{I}$  is used to recover the proxy in the vision space as in Eqn. 2. According to our analysis in Proposition 3,  $\tau_{I}$  should be larger than  $\tau_{T}$  in CLIP to leverage the overlap between the text and vision space. Table 1 shows the result when varying  $\tau_{I}$  in  $\{0.01,0.02,0.03,0.04,0.05\}$ . "Sim" computes the mean similarity between each example and its nearest proxy, depicting the gap (lower the similarly, higher the gap).
First, when  $\tau_{I} = 0.01$ , the gain is mainly from learning the vision proxy with refined labels and is already  $2.59\%$  better than the baseline with text proxy. By increasing  $\tau_{I}$ , the similarity between examples and proxies also increases, which confirms our analysis in Proposition 1. With an appropriate  $\tau_{I}$ , an additional gain of  $0.83\%$  can be observed with the similarity of 0.39. According to Proposition 3, a large  $\tau_{I}$  is crucial to recover the optimal vision proxy with the guidance from the text proxy. Thus, we fix  $\tau_{I} = 0.04$  for the main comparison.
Table 1: Effect of  ${\tau }_{I}$  . Table 2: Effect of  $\alpha$  .
τI	Acc%	Sim
Baseline	60.32	0.26
0.01	62.91	0.31
0.02	63.22	0.32
0.03	63.66	0.35
0.04	63.74	0.39
0.05	63.29	0.43
α	Acc%
1	63.49
0.9	63.53
0.7	63.70
0.6	63.74
0.5	63.72
0	63.14
Table 3: Effect of  $\gamma$  . Table 4: Components in
γ	Acc%
1	60.95
0.5	62.98
0.1	63.75
0.05	63.73
0.01	63.74
0	63.74
methods	Acc%
Baseline	60.32
InMaP0.25	60.83
InMaP0.5	60.95
Sinkhorn	62.53
InMaP	63.74
Pseudo label threshold  $\alpha$  Since the SoftMax operator can incur the over-smoothing issue for output logits, converting the soft label with high confidence to one-hot is an effective strategy in semi-supervised learning. We adopt and evaluate the performance with the threshold  $\alpha$  in Eqn. 5.
Table 2 summarizes the results with different thresholds. When  $\alpha = 1$ , the soft label after Sinkhorn distance optimization is adopted for optimization, which already surpasses the baseline by a clear margin of  $3.17\%$ . When reducing  $\alpha$ , more soft labels will be converted to one-hot labels and the performance can be further improved by eliminating noise from the SoftMax operator. When letting  $\alpha = 0$ , all pseudo labels become one-hot, which amplifies the noise from low-confidence prediction and degenerates the performance. We find that our method is insensitive to the parameter and fix it as 0.6 in the remaining experiments if not specified.
Pseudo label smoothing  $\gamma$  While the validation set of ImageNet is well-balanced with 50 examples for each class, we conduct the experiments with different  $\gamma$  in Eqn. 4 for demonstration.
Table 3 shows the accuracy on ImageNet with varying  $\gamma$ .  $\gamma = 1$  follows the distribution from the text prediction and degenerates to the original pseudo label without Sinkhorn distance refinement. By balancing the distribution of the class assignment with a small  $\gamma$ , the accuracy can be increased with refined pseudo labels. When  $\gamma$  is sufficiently small, the constraint in Sinkhorn distance degenerates to the balanced constraint and shows the desired performance due to the consistency with the ground-truth distribution. We will have the balanced prior distribution as  $\gamma = 0$  for other data sets if not specified.
Components in InMaP Finally, we show the gain from two components in InMaP in Table 4. Let "InMaP $^{0.25}$ " denote the proxy learning without refined labels and "InMaP $^{0.5}$ " be the variant optimized by solely applying the threshold  $\alpha$  to pseudo labels from the text proxy. "Sinkhorn" denotes the pseudo labels refined by Sinkhorn distance without proxy learning. We can observe that with the labels predicted from the text proxy, the recovered vision proxy of InMaP $^{0.25}$  can improve the baseline by  $0.51\%$  with the reduction in the modality gap. Then, simply thresholding the prediction from the text proxy can further improve the performance by  $0.12\%$  as in InMaP $^{0.5}$ . On the other hand, refining the labels via Sinkhorn distance can substantially outperform the baseline by  $2.21\%$ . Learning with better pseudo labels, InMaP shows an additional gain of  $1.21\%$  over refined labels. It confirms our analysis in Theorem 2 that the proposed intra-modal proxy learning can benefit from appropriate pseudo labels.
Number of text prompts An ensemble of text prompts shows the superior performance over the single prompt on obtaining text proxy for zero-shot classification [26]. We compare these two strategies on our method and summarize the accuracy on ImageNet in Table 5.
Table 5: Comparison of accuracy(%) on ImageNet with different text prompts. "Single" denotes the application of the single prompt of "a photo of a ().", while "ensemble" contains 7 prompts as suggested by [33].
Vision encoder	Baseline_single	Baseline_ensemble	InMaPsingle	InMaP_ensemble
ResNet-50	58.15	60.32	63.15	63.74
ViT-B/16	66.72	68.75	71.93	72.55
First, we can observe that the ensemble improves the performance of baseline with a clear margin of  $2\%$  with different vision encoders. It shows that the vanilla zero-shot transfer strategy is sensitive to the selection of text prompts and confirms the motivation of prompt learning methods. Second, our method outperforms the baseline with the single or ensemble of text prompts for text proxy, which demonstrates the effectiveness of our proposed intra-modal vision proxy. Moreover, the gap between the single and ensemble prompts shrinks on InMaP to less than  $1\%$ , which can help reduce the efforts of text prompt tuning.
Running time InMaP consists of two optimization problems, that is, pseudo label optimization in Eqn. 3 and proxy learning in Eqn. 2. The former one can be formulated as optimizing the Sinkhorn distance that can be solved efficiently by the standard iterative method in [8]. With 20 iterations, the running time is only 0.002 second, which is negligible. The latter problem is convex and defined with only extracted features. Unlike prompt learning, the expensive encoders are not included in optimization. Even with 2,000 iterations for gradient descent, the running time is only about 30 seconds on a single GPU and the cost can be further reduced to 6 seconds with FP16. In addition, many efficient methods are available for solving convex problems as in Eqn. 2 and can be explored if needed [3].
4.2 Comparison on ImageNet
In this subsection, we compare InMaP to state-of-the-art methods on ImageNet with different vision encoders provided by CLIP. Table 6 summarizes the results. Besides the vanilla zero-shot prediction in CLIP, TPT [30], SuS-X [33] and VisDesc [20] are included in the comparison, where TPT leverages test data as ours but for prompt learning and the other two employ external large models that are marked in grey color. Results for these methods are directly borrowed from their original papers.
First, we can observe that our method  $\mathrm{InMaP^{0.5}}$  outperforms the baseline and TPT with both ResNet-50 and ViT-B/16 as the vision encoder. Unlike the baseline and TPT that optimizes text prompts for images, we propose to reconstruct the proxy in vision space, which reduces the influence from the modality gap and obtains an appropriate intra-modal classifier as demonstrated in Theorem 2. In addition, the running time of proxy learning mainly depends on the dimension of extracted features and that with features from ViT-L/14@336 runs faster than ResNet-50, which guarantees the efficiency with different encoders. By further improving pseudo labels with Sinkhorn distance, InMaP shows the best performance among all methods and even surpasses those ones with additional large models, which confirms the effectiveness of our method. Moreover, InMaP has a steady improvement over  $\mathrm{InMaP^{0.5}}$  using different encoders. The observation indicates that label refinement
Table 6: Comparison of accuracy (%) on ImageNet with different vision encoders in CLIP. † in grey indicates the application of external large models. The overall best performance is in bold, while the best performance without any additional model is underlined. “-” denotes that the result is unavailable in their original papers.
Vision encoder	Baseline [26]	TPT [30]	InMaP0.5	InMaP	VisDesc† [20]	SuS-X-LC† [33]	SuS-X-SD† [33]
ResNet-50	60.32	60.74	60.95	63.74	59.68	61.89	61.84
ViT-B/32	63.77	-	64.82	67.29	62.97	64.73	64.71
ViT-B/16	68.75	68.98	70.17	72.55	68.03	70.00	69.88
ViT-L/14	75.96	-	77.14	79.29	75.00	-	-
ViT-L/14@336	77.02	-	78.27	80.21	76.16	-	-
is complementary to the intra-modal proxy learning, which is consistent with Theorem 2. Finally, with the largest backbone provided by CLIP, InMaP achieves  $80.21\%$  accuracy on ImageNet, which shows the potential of large pre-trained vision-language models.
After the optimization with the target data set, we demonstrate the performance of InMaP when the target data arrives in a streaming manner but an unlabeled set of relevant data is available. Concretely, the vision proxy is constructed from the unlabeled training set in lieu of the target validation set while the learned proxy is evaluated on the validation set. Table 7 shows the results of our method learned with unlabeled training set ("InMaPtrain") or validation set ("InMaPval") of ImageNet. For InMaPtrain,  $\tau_{I}$  and  $\alpha$  are reduced to 0.03 and 0.4, respectively, while other parameters remain unchanged.
Table 7: Comparison of accuracy (%) on ImageNet. For InMaP, "train" and "val" denote that the vision proxy is obtained from the unlabeled training set and validation set, respectively.
Vision encoder	ResNet-50	ViT-B/32	ViT-B/16	ViT-L/14	ViT-L/14@336
InMaP0.5val	60.95	64.82	70.17	77.14	78.27
InMaPval	63.74	67.29	72.55	79.29	80.21
InMaP0.5train	61.38	65.12	70.37	77.69	78.68
InMaPtrain	63.98	67.37	72.59	79.21	80.13
We can observe that even without the target data for recovering the vision proxy, an appropriate unlabeled data set is sufficient to obtain effective proxy for zero-shot classification. By thresholding the original pseudo labels,  $\mathrm{InMaP}_{\mathrm{train}}^{0.5}$  outperforms  $\mathrm{InMaP}_{\mathrm{val}}^{0.5}$  over all vision encoders, where the benefit may come from the large size of the training set. Then, if refining pseudo labels with Sinkhorn distance, the performance of proxies obtained with different data becomes similar and is better than the variant without optimized pseudo labels, which confirms the efficacy of our strategy for improving labels. Finally, this experiment demonstrates that the proposed method is applicable for different scenarios in real-world applications.
4.3 Comparison on 13 Downstream Tasks
Then, we evaluate the proposed method on 13 diverse downstream tasks, including Aircraft [19], Caltech101 [10], Stanford Cars [16], CIFAR-10 [17], CIFAR-100 [17], CUB200-2011 [35], Describable Textures Dataset (DTD) [7], EuroSAT [14], Flowers [21], Food101 [2], Oxford-IIIT Pet (Pets) [24], Sun397 [39], and UCF101 [32]. These data sets cover a large range of classification tasks such as fine-grained visual categorization, texture recognition, scene categorization, classification with low-resolution images, etc. The same parameters used for ImageNet are applied for most of them except certain parameters for refining labels. Concretely, Caltech and Flowers can benefit from the appropriate reference distribution for generating pseudo labels and we set  $\gamma$  in Sinkhorn distance as  $\gamma = 0.9$  for ResNet and  $\gamma = \{1,0.5\}$  for ViT, respectively for the two data sets. Moreover, the threshold  $\alpha$  is reduced to 0.3 on EuroSAT. Other parameters remain unchanged.
Table 8 summarizes the comparison. First, we observe that with the same parameters for all data sets,  $\mathrm{InMaP^{0.5}}$  shows the better performance than the baseline consistently and obtains  $1\%$  improvement on average by ResNet-50. It demonstrates that recovering the proxy in vision space can benefit different tasks for zero-shot transfer with pre-trained CLIP models. Second, by incorporating refined labels from optimizing Sinkhorn distance, InMaP can further improve  $\mathrm{InMaP^{0.5}}$  by more than  $4\%$  on average. The superior performance confirms that appropriate labels can facilitate the intra-modal
Table 8: Comparison of accuracy (%) on 13 diverse downstream tasks with ResNet-50 and ViT-B/16. † in grey indicates the application of external large models. The overall best performance is in bold, while the best performance without any additional model is underlined. “-” denotes that the result is unavailable in their original papers.
Aircraft	Caltech	Cars	Cifar10	Cifar100	CUB	DTD	EuroSAT	Flowers	Food	Pets	SUN	UCF101	Avg.
ResNet-50:
Baseline	16.62	86.00	56.31	73.15	40.60	41.37	41.13	26.90	63.13	74.10	81.85	59.25	55.56	55.07
TPT	17.58	87.02	58.46	-	-	-	40.84	28.33	62.69	74.88	84.49	61.46	60.82	-
InMap0.5	16.95	86.41	58.11	74.23	40.72	41.82	42.14	27.52	66.29	74.95	82.94	59.95	57.49	56.12
InMap	18.96	86.73	63.30	78.84	49.26	49.17	44.86	35.88	66.68	78.36	87.93	63.82	63.36	60.55
VisDesc†	16.26	88.11	54.76	73.22	39.69	48.31	41.96	37.60	65.37	76.80	82.39	59.84	58.47	57.14
SuS-X-LC†	21.09	89.57	57.17	74.95	44.48	48.86	49.23	44.23	67.07	77.62	86.59	63.01	61.49	60.41
SuS-X-SD†	19.47	89.53	57.27	74.69	44.63	49.12	50.59	45.57	67.72	77.58	85.34	62.95	61.54	60.46
ViT-B/16:
Baseline	23.13	93.51	66.29	91.09	67.29	49.36	45.09	50.17	67.64	84.43	87.00	65.68	65.21	65.84
TPT	24.78	94.16	66.87	-	-	-	47.75	42.44	68.98	84.67	87.79	65.50	68.04	-
InMap0.5	23.55	93.59	67.96	92.48	68.37	51.54	46.28	53.64	69.64	85.76	88.17	67.18	67.25	67.34
InMap	28.35	93.59	73.00	93.27	72.51	57.87	50.77	63.98	71.28	87.76	92.94	70.85	72.06	71.40
VisDesc†	-	-	-	-	-	57.75	45.59	48.82	-	88.50	86.92	-	-	-
SuS-X-LC†	30.51	93.91	65.90	90.94	68.66	56.96	55.32	58.06	73.08	86.08	91.58	67.85	66.72	69.66
SuS-X-SD†	28.68	93.96	66.13	89.88	68.47	57.11	54.55	57.49	73.81	86.08	90.57	67.73	66.59	69.31
proxy learning to obtain better proxy reconstruction in the target space. In addition, our method is the best among all methods on 8 out of 13 data sets with ResNet-50, which shows that leveraging unlabeled target data can be more effective than external models for zero-shot learning. Moreover, the same parameters are shared by different tasks for InMaP, while fine-tuning them by a validation set for each data set can further improve the performance.
With ViT-B as the vision backbone, the accuracy of the baseline increases by  $10\%$ , which shows the potential of large models. The proposed method can also benefit from the large models and achieve the accuracy of  $71.4\%$  on average, which is more than  $5\%$  better than the baseline. Compared with SuS-X with external models, InMaP outperforms it by  $1.74\%$  that confirms the effectiveness of intra-modal proxy learning. Moreover, for the data set with a high zero-shot accuracy as Caltech, the predicted distribution can approximate the ground-truth well and the label refinement with Sinkhorn distance can be skipped by letting  $\gamma = 1$ . Finally, both  $\mathrm{InMaP^{0.5}}$  and InMaP improves over the baseline on diverse tasks with ResNet and ViT, which illustrates that the proposed method is applicable for different vision encoders.
In this work, we focus on improving the zero-shot transfer with CLIP. First, our theoretical analysis indicates that the modality gap between the text and vision space obtained by CLIP will be preserved, which can degenerate the performance of zero-shot transfer. To mitigate the challenge, we propose to recover the proxy of each class in the vision space with the help from the text proxy. Concretely, by leveraging the unlabeled target data and refining the prediction from the text proxy, we can obtain the vision proxy via learning with unlabeled data and the corresponding pseudo label. Experiments on diverse data sets demonstrate that our method can improve the zero-shot accuracy consistently with different vision encoders. Exploring our method with other pre-trained models rather than CLIP can be our future work. Besides, recent work [33] shows that other large pre-trained models, i.e., GPT-3, can help CLIP for zero-shot prediction. Combining our method with existing large models is also an interesting future direction. Finally, our analysis for the efficacy of the text proxy is from the perspective of approximating the supervised learning for images as in Eqn. 1. Text information can be complementary to the knowledge in images [36] and investigating the potential benefit of text for vision tasks can help understand vision-language models better.
While our analysis is for the modality gap in contrastive pre-training with the architecture of dual encoders, it cannot be applied directly for other pre-training paradigms with different architectures. For example, BEiT-3 [37] has masked data modeling for pre-training with the shared backbone for different modalities and CoCa [40] has encoder-decoder captioning for pre-training. The analysis for the modality gap in these methods has been less investigated and can be inspired by our results as a future direction.
Proof. According to the definition, we have
$$
(1 - \delta) \exp (\mathbf {x} _ {i} ^ {\top} \mathbf {t} _ {i} / \tau) = \delta \sum_ {j: j \neq i} ^ {m} \exp (\mathbf {x} _ {i} ^ {\top} \mathbf {t} _ {j} / \tau)
$$
$$
\exp (\mathbf {x} _ {i} ^ {\top} \mathbf {t} _ {i} / \tau) = \frac {\delta}{1 - \delta} \sum_ {j: j \neq i} ^ {m} \exp (\mathbf {x} _ {i} ^ {\top} \mathbf {t} _ {j} / \tau) \leq \frac {\delta (m - 1)}{1 - \delta} \exp (\mathbf {x} _ {i} ^ {\top} \mathbf {t} _ {k} / \tau)
$$
$$
\exp (\mathbf {x} _ {i} ^ {\top} \mathbf {t} _ {i} / \tau) \geq \frac {\delta}{1 - \delta} \exp (\mathbf {x} _ {i} ^ {\top} \mathbf {t} _ {k} / \tau)
$$
where  $k = \arg \max_{j:j\neq i}\mathbf{x}_i^\top \mathbf{t}_j$ . Since logarithm function is monotone, the similarity between positive pair can be bounded by
where  $c_{1} = \log \left(\frac{\delta}{1 - \delta}\right)$  and  $c_{2} = \log \left(\frac{\delta(m - 1)}{1 - \delta}\right)$ .
$$
p _ {i, j} ^ {\prime} = \frac {\exp (\mathbf {x} _ {i} ^ {\top} \mathbf {z} _ {j} / \tau_ {T})}{\sum_ {k} \exp (\mathbf {x} _ {i} ^ {\top} \mathbf {z} _ {k} / \tau_ {T})} = \frac {\exp (\sqrt {a} \mathbf {x} _ {i} ^ {\top} \mathbf {z} _ {j} ^ {x} / \tau_ {T})}{\sum_ {k} \exp (\sqrt {a} \mathbf {x} _ {i} ^ {\top} \mathbf {z} _ {k} ^ {x} / \tau_ {T})} = \frac {\exp (\sqrt {a} \mathbf {x} _ {i} ^ {\top} \mathbf {w} _ {j} ^ {*} / \tau_ {T})}{\sum_ {k} \exp (\sqrt {a} \mathbf {x} _ {i} ^ {\top} \mathbf {w} _ {j} ^ {*} / \tau_ {T})}
$$
If  $\sqrt{a} /\tau_T = 1 / \tau_I$ , the predictions are equivalent.
$$
\left\| Z - W ^ {*} \right\| _ {F} ^ {2} = 2 C - 2 \sqrt {a} \langle Z ^ {x}, W ^ {*} \rangle = 2 C (1 - \sqrt {a}) + \sqrt {a} \| Z ^ {x} - W ^ {*} \| _ {F} ^ {2} \tag {6}
$$
To bound the similarity between  $Z^x$  and  $W^*$ , we assume  $Z^x = U_rA^\top$  where  $U_r \in \mathcal{R}^{d \times r}$  is a subset of  $U \in \mathcal{R}^{d \times d'}$ . By solving the problem  $\min_A \|U_rA - W^*\|_F^2$ , we have  $Z^x = U_rA = U_r(U_r^\top U_r)^{-1}U_r^\top W^* = U_rU_r^\top W^*$  that is projecting  $W^*$  to the subspace spanned by  $U_r$ . Since  $U_r$  is a subset from  $U$ , we have  $U_rU_r^\top = \sum_i^{d'} r_i u_i u_i^\top$  with  $\forall i, r_i \in \{0, 1\}$  and  $\sum_i r_i = r$ . Then, we have
$$
\| U _ {r} U _ {r} ^ {\top} W ^ {*} - W ^ {*} \| _ {F} ^ {2} = \| \sum_ {i} (r _ {i} - 1) s _ {i} u _ {i} v _ {i} ^ {\top} \| _ {F} ^ {2} = \sum_ {i} (r _ {i} - 1) ^ {2} s _ {i} ^ {2}
$$
$$
\| Z ^ {x} - W ^ {*} \| _ {F} ^ {2} \geq \| U _ {r} U _ {r} ^ {\top} W ^ {*} - W ^ {*} \| _ {F} ^ {2} \geq \sum_ {i = r + 1} ^ {d ^ {\prime}} s _ {i} ^ {2}
$$
where the last inequality is from keeping the largest singular values for minimizing the approximation loss. The target result is obtained by taking this inequality to Eqn. 6.
Proof. First, we note that  $L(P, W)$  is a convex function in  $W$ . We assume that it is  $\mu$ -strongly convex in  $W$  such that for the arbitrary  $(W_1, W_2)$ , we have
$$
L (P, W _ {1}) \geq L (P, W _ {2}) + \langle \nabla_ {W _ {2}} L (P, W _ {2}), W _ {1} - W _ {2} \rangle + \frac {\mu}{2} \| W _ {1} - W _ {2} \| _ {F} ^ {2}
$$
According to the optimality of  $W^{\prime *}$  , we have
$$
L \left(P ^ {\prime}, W ^ {\prime *}\right) - L (Y, W ^ {*}) \leq L \left(P ^ {\prime}, W ^ {*}\right) - L (Y, W ^ {*}) = \langle P ^ {\prime} - Y, - \log \left(P _ {W ^ {*}}\right) \rangle \tag {7}
$$
The lower-bound can be obtained from the strongly convexity of  $W$  as
$$
\begin{array}{l} L \left(P ^ {\prime}, W ^ {\prime *}\right) - L (Y, W ^ {*}) = L (Y, W ^ {*}) - L (Y, W ^ {*}) + \langle P ^ {\prime} - Y, - \log \left(P _ {W ^ {\prime *}}\right) \rangle \\ \geq \frac {\mu}{2} \| W ^ {\prime *} - W ^ {*} \| _ {F} ^ {2} + \left\langle P ^ {\prime} - Y, - \log \left(P _ {W ^ {\prime *}}\right) \right\rangle \tag {8} \\ \end{array}
$$
Combining inequalities in Eqns. 7 and 8, we have
$$
\| W ^ {\prime *} - W ^ {*} \| _ {F} ^ {2} \leq \frac {2}{\mu} \langle P ^ {\prime} - Y, \log \left(P _ {W ^ {\prime *}}\right) - \log \left(P _ {W ^ {*}}\right) \rangle \leq \frac {2}{\mu} \| P ^ {\prime} - Y \| _ {F} \| \log \left(P _ {W ^ {\prime *}}\right) - \log \left(P _ {W ^ {*}}\right) \| _ {F}
$$