SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Convergence Analysis. We will prove the convergence property of the function value. Indeed, we prove a stronger condition that the function value decreases after each round of iterations on  $w$  and  $t$ . From the monotone convergence theorem, the convergence can thus be derived. We first present the decreasing result of inner loop of  $t$  by Theorem 1 and the same property holds for the update of  $s$ . Then the convergence of the whole algorithm can be derived by Theorem 2. The detailed proofs are placed in the Appendix Sec.A.
Theorem 1. Suppose  $s(w,t) = \frac{1}{2}\| Xw - Zt\| _F^2$  where  $X\in \mathbb{R}^{N\times D}$ ,  $Z\in \mathbb{R}^{N\times K}$ ,  $w\in \mathbb{R}^{D\times 1}$  and  $t\in \triangle^{K - 1}$ , the inner loop of  $t$  in Algorithm 1 lines 7 - 10 decreases after each iteration. Specifically, denote  $\beta = 1 / \| 2Z^T Z\|$  and  $t^{+} = \Pi_{\triangle^{K - 1}}(t - \beta \nabla s(w,t))$ . For any  $t\in \triangle^{K - 1}$ ,  $s(w,t^{+}) - s(w,t)\leq -\frac{1}{2\beta}\| t - t^{+}\|^{2}\leq 0$ .
Theorem 2. Suppose  $s(w,t) = \frac{1}{2}\| Xw - Zt\| _2^2$  where  $X\in \mathbb{R}^{N\times D}$ ,  $Z\in \mathbb{R}^{N\times K}$ ,  $w\in \mathbb{R}^{D\times 1}$  and  $t\in \triangle^{K - 1}$ , the function value in Algorithm 1 will converge. Specifically, denote  $w^{\star},t^{\star}$  as the result after one iteration of  $w,t$  respectively, we have  $0\leq s(w^{\star},t^{\star})\leq s(w^{\star},t)\leq s(w,t)$ .
Computational Speedup. Although this algorithm 1 guarantees convergence, it is a bit time-consuming due to the two-level loop, we optimized this part and achieved similar results in very little time. Since the least squares solution is extremely fast, we performs least squares on  $w$  and  $t$ , and then replace projection onto simplex with explicit Sparsemax transformation [45, 46], iteratively. The fast solver is illustrated in Algorithm 2. we experimentally verify its convergence and find that the approach achieves impressive speedup.
This section evaluates our method EMMS on different downstream tasks, including image classification, image caption, visual question answering, text question answering and referring expression comprehension. We put more experiments details in Appendix Sec.B. Moreover, we conduct a detailed ablation study to analyze our EMMS in Appendix Sec.C
5.1 Training Details
Benchmark. For image classification, We adopt 11 classification benchmarks, including FGVC Aircraft [47], Caltech-101 [48], Stanford Cars [49], CIFAR-10 [50], CIFAR-100 [50], DTD [51], Oxford 102 Flowers [52], Food-101 [53], Oxford-IIIT Pets [54], SUN397 [55], and VOC2007 [56]. For image caption, We use Flickr8k [57], Flickr30k [58], FlickrStyle10K-Humor [59], FlickrStyle10K-Romantic [59] and RSICD [60]. For visual question answer, We apply COCOQA [61], DAQUAR [62] and CLEVR [63]. For text question answer and referring expression comprehension, we separately use SQuAD1.1 [64], SQuAD2.0 [65] and RefCOCO [66], RefCOCO+ [66], RefCOCOg [67].
Ground truth. In order to obtain the ground truth, we finetune all pre-trained models on all target datasets with a grid search of hyper-parameters. Details of target datasets and fine-tuning schemes are described in Appendix Sec.B.
Evaluation protocol. To assess how well a model selector predict the transferability of pre-trained models, we calculate the rank correlation between  $\{T_m\}_{m=1}^M$  and  $\{G_m\}_{m=1}^M$ . Following the common practice [13, 31], we use weighted Kendall's  $\tau_w$ . The larger  $\tau_w$  indicates a better correlation and better transferability metric. For computation complexity, we record the runtime of executing algorithm
Table 1: Comparison of different transferability metrics on ViT models regarding  $\tau_w$  and the wall-clock time where EMMS(One) denotes EMMS with the one-hot label. Our proposed EMMS achieves the best transfer-ability assessment over 11 target tasks and exhibits higher efficiency than NLEEP.
Aircraft	Caltech	Cars	CF-10	CF-100	DTD	Flowers	Food	Pets	SUN	VOC	Avg.
Weighted Kendall's tau τw
LogME	0.299	0.382	0.633	0.741	0.727	0.569	0.512	0.580	0.528	0.619	0.591	0.561
NLEEP	-0.282	0.027	0.693	0.674	0.538	0.123	-0.262	0.105	0.40	0.268	0.109	0.218
TransRate	0.244	0.412	0.487	0.260	0.702	0.533	0.655	0.542	0.707	0.612	0.651	0.527
EMMS(One)	0.412	0.444	0.565	0.740	0.736	0.621	0.562	0.579	0.740	0.592	0.730	0.611
EMMS	0.481	0.444	0.706	0.718	0.745	0.621	0.562	0.673	0.740	0.619	0.730	0.639
Wall-Clock Time (s)
LogME	8.93	10.89	30.28	53.07	62.13	4.78	9.27	104.92	6.28	425.43	7.42	65.76
NLEEP	553.7	716.8	1.1e3	8.0e3	1.2e4	183.7	819.2	3.4e4	256.4	2.7e4	288.3	7719.8
TransRate	19.43	19.21	36.9	61.73	63.82	8.73	18.26	110.79	15.51	89.92	5.11	40.85
EMMS(One)	4.12	4.45	8.07	19.45	26.18	2.65	4.03	39.72	3.50	24.84	4.07	12.82
EMMS	21.31	17.23	28.06	154.61	182.11	13.87	15.95	265.99	17.93	63.86	16.63	72.55
over all models given the feature and label on a target task and analyzed the computational complexity of EMMS as well as LogME. (Details can be found in Appendix Sec.C)
Baseline. For the image classification task, we choose NLEEP [31], TransRate [23], and LogME [13] as the baseline; for other multimodal tasks, we choose LogME with F-Label as the baseline; in addition, for the VQA task, we additionally compare PACTran [24]. Details of baselines and why we choose them are described in Appendix Sec.B.
5.2 Image Classification with ViT Models
Vision transformer [9] (ViT) models have been increasingly used for a variety of tasks and have achieved better results than CNN models. The architecture of ViT models are more complex than CNN models. Hence, how to do the model selection on ViT models is a more challenging and rewarding task. Details of pre-trained models are described in Appendix Sec.B.
Performance and wall-clock time comparison. As shown in Table.1, our EMMS achieve the best average  $\tau_w$  on 11 target datasets and the best  $\tau_w$  on 9 target datasets with relatively short time. For example, EMMS outperforms LogME by 0.182 and 0.139 rank correlation  $\tau_w$  on Aircraft, and VOC2007, respectively, showing the effectiveness of our EMMS in measuring the transfer-ability of pre-trained ViT models. On the other hand, for the remaining 2 target datasets (i.e. CF-10, DTD), our EMMS still has a marginal gap compared to the best-performing transferability metric. Besides, we find that the effect of model selection of EMMS in ViT models selection has an improvement compared to CNN models selection, we guess F-Label has spatial similarity with the model feature of ViT-base model because the foundation models are mostly transformer-based, which can model the relationship between model feature from ViT-base models and F-Labels more accurately.
5.3 Image Captioning
Here we treat image caption as a vocab-based classification task. That is we use a vocabulary and classify the caption into the index of some words in the vocabulary. Afterward, training is done according to the classification task criteria. Here we calculate the average  $\tau_w$  and time of LogME with  $K$  single F-label from  $K$  foundation models we use respectively. We wants to select the best combination of image encoder and language encoder. Details of pre-trained models and the model architecture are described in Appendix Sec.B.
Performance and wall-clock time comparison. As shown in Table.2, EMMS is significantly ahead of baseline in both time and effect for each dataset. For example, EMMS outperforms LogME with the relative improvements of  $39\%$  and  $37\%$  in rank correlation  $\tau_w$  on Flickr8k and Flickr30k, respectively. In addition, the time of EMMS is reduced by  $83.7\%$  and  $79.8\%$  relative to LogME on these two datasets, which shows the efficiency of our algorithm. The average rank correlation  $\tau_w$  alone the five datasets is 0.64, which denotes EMMS has sufficient confidence.
Table 2: Comparison of different transferability metrics on image caption models in rank correlation  $\tau_w$  with the ground truth and the wall-clock time. The LogME denotes using LogME with F-Label. Our proposed EMMS achieves the best transfer-ability assessment on each target task with much less time compared to LogME.
F8k	F30k	RSD	F10k-H	F10k-R	F8k	F30k	RSD	F10k-H	F10k-R
Weighted Kendall's tau τw	Wall-Clock Time (s)
LogME	0.483	0.368	0.501	0.780	0.654	425.67	1594.16	973.22	60.35	63.79
EMMS	0.660	0.504	0.704	0.802	0.678	69.01	321.32	88.77	16.56	14.59
5.4 Text Question Answering
For natural language understanding, we consider Text Question Answering (TQA) as a reading comprehension task, where the response to each question is a text segment extracted directly from the affiliated reading passage, or the question may indeed be deemed unanswerable. Details of pre-trained models and how to finetune are described in Appendix Sec.B.
Performance and wall-clock time comparison. In Table 3, the performance improvement of EMMS on the TQA is consistent with the enhancements observed in the earlier mentioned computer vision tasks. More specifically, our EMMS attains accuracies of  $60.3\%$  and  $46.3\%$  on the Stanford Question Answering Dataset (SQuAD) versions 1.1 and 2.0 respectively, using rank correlation  $\tau_w$  as an evaluation metric. This represents a significant relative increment of  $11.2\%$  and  $13.2\%$  compared to the performance of LogME.
Table 3: Comparison of different transferability Table 4: Comparison of different transferability metrics on TQA models in rank correlation  ${\tau }_{w}$  metrics on referring expression models in rank with the ground truth and the wall-clock time.correlation  ${\tau }_{w}$  with ground truth and the time. The LogME denotes using LogME with F-Label. The LogME denotes using LogME with F-Label.
SQu1.1	SQu2.0	SQu1.1	SQu2.0
Weighted Kendall's tau τw	Wall-Clock Time (s)
LogME	0.542	0.409	3587.22	3596.23
EMMS	0.603	0.463	571.23	589.78
Ref	Ref+	Refg	Ref	Ref+	Refg
Weighted Kendall's tau τw	Wall-Clock Time (s)
LogME	0.423	0.389	0.398	2457.87	2478.90	2298.76
EMMS	0.458	0.549	0.521	454.26	467.92	356.94
5.5 Referring Expression Comprehension
Referring expression comprehension (REC) is a widely challenging task because it requires precise alignment between linguistic concepts and image features. To address this, the objects in each image are represented as a sequence of discrete tokens, while their bounding box corner coordinates are turned into integer location tokens. This allows for a unified F-Label to be extracted using various language models. More details about the pre-trained models can be found in Appendix Sec.B.
Performance and wall-clock time comparison. As shown in Table 4, our EMMS continues to exhibit its superiority in the enhancement of performance on the REC task, an instance-level cross-modal localization task. Specifically, the proposed EMMS produces accuracies of  $45.8\%$ ,  $54.9\%$ , and  $52.1\%$  on the RefCOCO, RefCOCO+, and RefCOCOg datasets respectively. This significantly surpasses its counterpart, LogME, in terms of margins when evaluated with rank correlation  $\tau_w$ .
5.6 Ablation Analysis
Comparison with different number of F-Label Here we denote the number of F-Label is  $K$  and choose the image caption task to illustrate the impact of  $K$  on our solution. As shown in Table 6. We find that increasing  $K$  in a certain range brings a gain in effectiveness to our method, but when  $K$  becomes larger, the time also increases and we find that  $K = 4$  is not as effective as  $K = 3$ . We believe that the increase in  $K$  brings difficulties in fitting the true Label, resulting in a loss of effectiveness. Therefore, we use  $K = 3$  for the sake of effect and time.
Performance on F-Label using small model On the one hand, using foundation model can extract the joint embedding compared to the small model, which allows EMMS to be extended to tasks with multiple forms of labels. On the other hand, the foundation model can handle many types of tasks,
Table 6: EMMS under different number of F-Label of transferability assessment on image caption task. The improvement of  $K$  in a certain range brought an increase in rank correlation  ${\tau }_{w}$  .
F8k	F30k	RSD	F10k-H	F10k-R	F8k	F30k	RSD	F10k-H	F10k-R
Weighted Kendall's tau τw	Weighted Kendall's tau τw
K=1	0.490	0.386	0.527	0.772	0.668	K=2	0.574	0.454	0.553	0.762	0.646
K=3	0.660	0.504	0.704	0.802	0.678	K=4	0.660	0.504	0.704	0.802	0.644
so we can use the foundation model for different tasks for label embedding. As shown in Table 5, we experimentally demonstrate that the use of the foundation model leads to more accurate F-Label extraction and thus to an improvement in the performance of the method.
The effect of using a single foundation model We investigate how EMMS is influenced when only a single foundation model is provided. We conduct experiments on image classification and image captioning. We consider EMMS with the single foundation model including language foundation model (1) GPT-2 [27], (2) BERT [43], (3) RoBerta [68], and multimodal foundation model (4) CLIP [26], (5) FLAVA [69], and (6) AltCLIP [70]. For comparison, we include the result of our EMMS with default setting  $(\mathrm{K} = 3$  , i.e. CLIP, BERT, and GPT-2) and the result of previous state-ofthe-art methods obtained from LogME, NLEEP and TransRate. The results are reported in Table 20 and Table 5.
We have several observations. (1) Different downstream tasks prefer F-Labels obtained from different foundation models. No single foundation model is dominant in all target tasks. In particular, CLIP is not the best model for extracting F-Labels. (2) For image captioning, multimodal foundation models are more appropriate for extracting F-Labels than language foundation models. (3) Our EMMS can achieve the best results by combining F-Labels obtained from multiple foundation models.
Table 5: The effect of the single foundation model on EMMS. The results are obtained on image captioning regarding  $\tau_w$ .
F8k	F30k	RSD	F10kH	F10kR	Avg	SOTA/All
Weighted Kendall's tau τw
LogME(Clip)	0.530	0.393	0.618	0.764	0.634	0.588	0/5
(1) Gpt2	0.566	0.393	0.431	0.715	0.618	0.545	0/5
(2) Bert	0.395	0.319	0.448	0.802	0.711	0.535	2/5
(3) RoBerta	0.346	0.111	0.587	0.571	0.566	0.436	0/5
(4) CLIPB	0.453	0.393	0.704	0.802	0.634	0.533	2/5
(5) CLIPL	0.510	0.448	0.704	0.802	0.678	0.628	2/5
(6) FLAVA	0.463	0.382	0.693	0.704	0.678	0.584	0/5
(7) AltCLIP	0.453	0.448	0.623	0.802	0.678	0.601	1/5
EMMS	0.660	0.504	0.704	0.802	0.678	0.670	4/5
How to select a pre-trained model for different tasks quickly and effectively is an important issue in the field of transfer learning. This paper proposes an efficient multi-task model selector(EMMS) that can be applied to many types of tasks. EMMS uses foundation model for Label embedding in order to transform diverse label formats of different tasks into the same form and see them as noisy labels. To estimate a model's transferability, EMMS model this problem as a simple weighted linear regression, which can be solved using an alternating minimization algorithm. Compared with existing methods, EMMS achieves the first model selection in multi-task scenarios, including image caption, referring segmentation, etc., with high speed and great results. For the limitations of the method, if the foundation model generalize very poor on downstream tasks, it may lead to low-quality label embedding, which is a drawback of our method. Moreover, building a holistic benchmark of various label embeddings would be useful in many applications such as multi-modal adaptation [71]. We leave it as a future work.
This paper is partially supported by the National Key RD Program of China No.2022ZD0161000, National Key RD Program of China(NO.2022ZD0160100) and the General Research Fund of Hong
Kong No.17200622. Besides, thanks Ruimao Zhang from CUHK(SZ) for thoughtful discussion and Prof. Anthony Man-Cho So from CUHK for his valuable discussion about solving the WLSR in the paper. At last, this work was done during the internship at Shanghai Artificial Intelligence Laboratory.
[83] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904-6913, 2017.
[84] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. XLnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.
[85] Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654, 2020.
[86] Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electro-style pre-training with gradient-disentangled embedding sharing. arXiv preprint arXiv:2111.09543, 2021.
[87] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pages 12888-12900. PMLR, 2022.
[88] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694-9705, 2021.
[89] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In International Conference on Machine Learning, pages 23318-23340. PMLR, 2022.
[90] Hao Li, Pratik Chaudhari, Hao Yang, Michael Lam, Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Rethinking the hyperparameters for fine-tuning. arXiv preprint arXiv:2002.11770, 2020.
Here we derive in detail the regression with Unified Noisy Label Embeddings that appear in the method section of the text in Sec.A.1 and give complete proof of the convergence of the method in Sec.A.2.
A.1 Regression with Unified Noisy Label Embeddings
Setup. we assume that label embedding  $z$  is a linear mapping of the model feature with additive Gaussian noise with a variance of  $\sigma_0^2$ , as given by  $z = z_{0} + \epsilon = w^{T}\hat{x} +\epsilon$  and  $\epsilon \sim N(0,\sigma_0^2 I_L)$  where  $z_{0} = w^{T}\hat{x}$  is the regression prediction,  $w\in \mathbb{R}^{D\times L}$  and  $\epsilon$  are regression weights and regression error, respectively, and  $I_{L}$  is a L-by-L identity matrix.
We assume that F-labels  $\{z_k\}_{k=1}^K$  obtained from different foundation models are oracles that independently provide noisy estimates of the label embedding  $z$ . Formally, we have  $P(z_k|z) = N(z, \sigma_k^2 I_L)$ . Without loss of generality, we assume that  $L = 1$
Then the joint probability over noisy labels for a fixed  $n$ , That is, for given  $x^n$ , we have:
$$
P \left(z _ {1} ^ {n}, \dots , z _ {K} ^ {n} \mid x ^ {n}, w\right) = \int P \left(z _ {1} ^ {n}, \dots , z _ {K} ^ {n} \mid z, x ^ {n}, w\right) P \left(z \mid x ^ {n}, w\right) d z \tag {4}
$$
Due to the independence between  $z_{k}$  and  $x$ , using the real label  $z$ , we can rewrite it as:
$$
P \left(z _ {1} ^ {n}, \dots , z _ {K} ^ {n} \mid x ^ {n}, w\right) = \int P \left(z _ {1} ^ {n}, \dots , z _ {K} ^ {n} \mid z, w\right) P \left(z \mid x ^ {n}, w\right) d z \tag {5}
$$
And using the independencies among  $z_{k}$ , we have:
$$
P \left(z _ {1} ^ {n}, \dots , z _ {K} ^ {n} \mid z, w\right) = \prod_ {k = 1} ^ {K} P \left(z _ {K} ^ {n} \mid z, \sigma_ {1} ^ {2}, \dots , \sigma_ {k} ^ {2}\right) = \frac {1}{\left(2 \pi\right) ^ {\frac {K}{2}} \prod_ {k = 1} ^ {K} \sigma_ {k}} \exp^ {- \sum_ {k = 1} ^ {K} \frac {\left(z _ {k} ^ {n} - z\right) ^ {2}}{2 \sigma_ {k} ^ {2}}} \tag {6}
$$
Due to  $P(z_{k}|z) = N(z,\sigma_{k}^{2}I_{L})$ , we can rewrite it as:
$$
P \left(z _ {1} ^ {n}, \dots , z _ {K} ^ {n} \mid x ^ {n}, w\right) = \int \frac {1}{(2 \pi) ^ {\frac {K + 1}{2}} \prod_ {k = 0} ^ {K} \sigma_ {k}} \exp^ {- \sum_ {k = 1} ^ {K} \frac {\left(z _ {K} ^ {n} - z\right) ^ {2}}{2 \sigma_ {k} ^ {2}} - \frac {\left(z - z _ {0}\right) ^ {2}}{2 \sigma_ {0} ^ {2}}} d y \tag {7}
$$
which can be calculated as :
$$
P \left(z _ {1} ^ {n}, \dots , z _ {K} ^ {n} \mid x ^ {n}, w\right) = A _ {1} \int e ^ {- A _ {2} y ^ {2} + A _ {3} ^ {n} y - A _ {4} ^ {n}} d z = A _ {1} \sqrt {\frac {\pi}{A _ {2}}} e ^ {\frac {\left(A _ {3} ^ {n}\right) ^ {2}}{4 A _ {2}} - A _ {4} ^ {n}} \tag {8}
$$
where  $A_{1} = \prod_{k = 0}^{K}1 / \sigma_{k}, A_{2} = \sum_{k = 0}^{K}1 / 2\sigma_{k}^{2}, A_{3}^{n} = \sum_{k = 0}^{K}z_{k}^{n} / \sigma_{k}^{2}$ , and  $A_{4}^{n} = \sum_{k = 0}^{K}(z_{k}^{n})^{2} / 2\sigma_{k}^{2}$ .
Consider the joint probability over all  $N$  instances, we have:
$$
P \left(z _ {1} ^ {n}, \dots , z _ {K} ^ {n} \mid X, w\right) = \prod_ {i = 1} ^ {N} A _ {1} \sqrt {\frac {\pi}{A _ {2}}} e ^ {\frac {\left(A _ {3} ^ {n}\right) ^ {2}}{4 A _ {2}} - A _ {4} ^ {n}} \tag {9}
$$
where  $X \in R^{N \times D}$  denotes the feature matrix,  $N$  is the number of data points and  $D$  is the number of features.
Then given  $N$  data points, the negative log-likelihood is given by
$$
- \mathcal {L} = \underbrace {- N \log A _ {1} + \frac {N}{2} \log A _ {2}} _ {\mathcal {L} _ {1}} + \frac {1}{2} \underbrace {\sum_ {n = 1} ^ {N} \left(A _ {4} ^ {n} - \frac {\left(A _ {3} ^ {n}\right) ^ {2}}{4 A _ {2}}\right)} _ {\mathcal {L} _ {2}} + \text {c o n s t} \tag {10}
$$
where  $\mathcal{L}_1$  and  $\mathcal{L}_2$  are given by
$$
\mathcal {L} _ {1} = \frac {N}{2} \log \sum_ {k = 0} ^ {K} \frac {1}{2 \sigma_ {k} ^ {2}} + N \sum_ {k = 0} ^ {K} \log \sigma_ {k}, \quad \mathcal {L} _ {2} = \sum_ {n = 1} ^ {N} \left\{\sum_ {k = 0} ^ {K} \frac {\left(z _ {k} ^ {n}\right) ^ {2}}{\sigma_ {k} ^ {2}} - \frac {\left(\sum_ {k = 0} ^ {K} z _ {k} ^ {n} / \sigma_ {k} ^ {2}\right) ^ {2}}{\sum_ {k = 1} ^ {K} 1 / \sigma_ {k} ^ {2}} \right\} \tag {11}
$$
Since  $\mathcal{L}_1$  is independent of input data, we focus on  $\mathcal{L}_2$ . To simplify the notation, we re-denote  $\gamma_k = 1 / \sigma_k^2$  and  $\Gamma = \sum_{k=1}^{K} \gamma_k$ . Using this notation,  $\mathcal{L}_2$  can be rearranged as:
$$
\begin{array}{l} \mathcal {L} _ {2} = \sum_ {n = 1} ^ {N} \left\{\gamma_ {0} z _ {0} ^ {2} + \sum_ {k = 1} ^ {K} \gamma_ {k} \left(z _ {k} ^ {n}\right) ^ {2} - \frac {\left(\sum_ {k = 1} ^ {K} \gamma_ {k} z _ {k} ^ {n} + \gamma_ {0} z _ {0}\right) ^ {2}}{\Gamma + \gamma_ {0}} \right\} (12) \\ = \sum_ {n = 1} ^ {N} \left\{\left(\gamma_ {0} - \frac {\gamma_ {0} ^ {2}}{\Gamma + \gamma_ {0}}\right) z _ {0} ^ {2} - \left(\frac {2 \Gamma \gamma_ {0}}{\Gamma + \gamma_ {0}} \sum_ {k = 1} ^ {K} \frac {\gamma_ {0}}{\Gamma} z _ {k} ^ {n}\right) z _ {0} + \sum_ {k = 1} ^ {K} \gamma_ {k} \left(z _ {k} ^ {n}\right) ^ {2} - \left(\sum_ {k = 1} ^ {K} \gamma_ {k} z _ {k} ^ {n}\right) ^ {2} \right\} (13) \\ = \sum_ {n = 1} ^ {N} \left\{\frac {\Gamma \gamma_ {0}}{\Gamma + \gamma_ {0}} \left(z _ {0} - \sum_ {k = 1} ^ {K} \frac {\gamma_ {k}}{\Gamma} z _ {k} ^ {n}\right) ^ {2} + \sum_ {k = 1} ^ {K} \gamma_ {k} \left(z _ {k} ^ {n}\right) ^ {2} - \left(1 + \frac {\gamma_ {0}}{\Gamma (\Gamma + \gamma_ {0})} \left(\sum_ {k = 1} ^ {K} \gamma_ {k} z _ {k} ^ {n}\right) ^ {2} \right. \right\} (14) \\ = \sum_ {n = 1} ^ {N} \left\{\frac {\Gamma \gamma_ {0}}{\Gamma + \gamma_ {0}} \left(w ^ {T} \hat {x} ^ {n} - \sum_ {k = 1} ^ {K} \frac {\gamma_ {k}}{\Gamma} z _ {k} ^ {n}\right) ^ {2} + \sum_ {k = 1} ^ {K} \gamma_ {k} \left(z _ {k} ^ {n}\right) ^ {2} - \left(1 + \frac {\gamma_ {0}}{\Gamma (\Gamma + \gamma_ {0})} \left(\sum_ {k = 1} ^ {K} \gamma_ {k} z _ {k} ^ {n}\right) ^ {2} \right. \right\} (15) \\ \end{array}
$$
Hence, the negative likelihood in Eqn.(10 can be written as
$$
- \mathcal {L} = \frac {\Gamma \gamma_ {0}}{\Gamma + \gamma_ {0}} \left\{\underbrace {\frac {1}{2} \sum_ {i = 1} ^ {N} \left(w ^ {T} \hat {x} ^ {n} - \sum_ {k = 1} ^ {K} \frac {\gamma_ {k}}{\Gamma} z _ {k} ^ {n}\right) ^ {2}} _ {s (w, t)} \right\} + \mathcal {R} \left(\gamma_ {k}\right) \tag {16}
$$