SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\operatorname {t r} (\mathbf {G P}) = \operatorname {t r} \left(\mathbf {G} \boldsymbol {\Lambda} \mathbf {S} (\mathbf {V}) ^ {\mathrm {T}}\right) = \operatorname {t r} \left(\mathbf {S} (\mathbf {V}) ^ {\mathrm {T}} \mathbf {G} \boldsymbol {\Lambda}\right) = \operatorname {t r} (\mathbf {S H}) = \sum_ {i} s _ {i i} h _ {i i}, \tag {17}
$$
where  $\mathbf{H} = (\mathbf{V})^{\mathrm{T}}\mathbf{G}\mathbf{A}$ ,  $s_{ii}$  and  $h_{ii}$  are the  $(i,i)$  elements of  $\mathbf{S}$  and  $\mathbf{H}$ , respectively. It can be easily verified that  $\mathbf{H}(\mathbf{H})^{\mathrm{T}} = \mathbf{I}$ , where  $\mathbf{I}$  is an identity matrix. Therefore  $-1 \leqslant h_{ii} \leqslant 1$  and  $s_{ii} \geqslant 0$ . Thus we have:
$$
\operatorname {t r} (\mathbf {G P}) = \sum_ {i} s _ {i i} h _ {i i} \leqslant \sum_ {i} s _ {i i}. \tag {18}
$$
The equality holds when  $\mathbf{H}$  is an identity matrix.  $\mathrm{tr}(\mathbf{GP})$  reaches the maximum when  $\mathbf{H} = [\mathbf{I},\mathbf{0}]$
So the solution of (15) is:
$$
\overline {{\boldsymbol {\mathcal {H}}}} ^ {(v)} = \overline {{\boldsymbol {\Lambda}}} ^ {(v)} \left(\overline {{\boldsymbol {V}}} ^ {(v)}\right) ^ {\mathrm {T}} \tag {19}
$$
where  $\overline{\boldsymbol{\Lambda}}^{(v)}$  and  $\overline{\boldsymbol{V}}^{(v)}$  can be obtained by SVD  $\overline{\pmb{B}}^{(v)} = \overline{\boldsymbol{\Lambda}}^{(v)}\mathbf{X}(\overline{\boldsymbol{V}}^{(v)})^{\mathrm{T}}$
- Solve  $\mathcal{Q}$  with fixed  $\mathcal{H},\mathcal{G},\mathcal{J}$ . (7) becomes:
$$
\left. \min  _ {\mathcal {Q} \geqslant 0} \frac {\mu}{2} \right\| \mathcal {H} - \mathcal {Q} + \frac {\mathcal {Y} _ {1}}{\mu} \Bigg \| _ {F} ^ {2} \tag {20}
$$
(20) is obviously equivalent to:
$$
\left. \min  _ {\boldsymbol {Q} \geqslant 0} \frac {\mu}{2} \right\| \boldsymbol {Q} - \left(\boldsymbol {H} + \frac {\boldsymbol {y} _ {1}}{\mu}\right) \| _ {F} ^ {2} \tag {21}
$$
According to [37], the solution of (21) is:
$$
\boldsymbol {Q} = \left(\boldsymbol {\mathcal {H}} + \frac {\boldsymbol {y} _ {1}}{\mu}\right) _ {+} \tag {22}
$$
- Solve  $\mathcal{J}$  with fixed  $\mathcal{Q},\mathcal{H},\mathcal{G}$ . (7) becomes:
$$
\min  \lambda \| \mathcal {J} \| _ {\mathscr {S} ^ {p}} ^ {p} + \frac {\rho}{2} \left\| \boldsymbol {\mathcal {H}} - \mathcal {J} + \frac {\mathcal {Y} _ {2}}{\rho} \right\| _ {F} ^ {2}, \tag {23}
$$
after completing the square regarding  $\mathcal{I}$ , we can deduce
$$
\mathcal {J} ^ {*} = \arg \min  \frac {1}{2} \left\| \mathcal {H} + \frac {\mathcal {Y} _ {2}}{\rho} - \mathcal {J} \right\| _ {F} ^ {2} + \frac {\lambda}{\rho} \| \mathcal {J} \| _ {\mathbb {S} ^ {p}} ^ {p}, \tag {24}
$$
which has a closed-form solution as Lemma 2 [12]:
Lemma 2. Let  $\mathcal{Z} \in \mathbb{R}^{n_1 \times n_2 \times n_3}$  have a  $t$ -SVD  $\mathcal{Z} = \mathcal{U} * \mathcal{S} * \mathcal{V}^{\mathrm{T}}$ , then the optimal solution for
$$
\min  _ {\mathcal {X}} \frac {1}{2} \| \mathcal {X} - \mathcal {Z} \| _ {F} ^ {2} + \tau \| \mathcal {X} \| _ {\mathscr {S D}} ^ {p}. \tag {25}
$$
is  $\mathcal{X}^* = \Gamma_{\tau}(\mathcal{Z}) = \mathcal{U} * \mathrm{iff}(P_{\tau}(\overline{\mathcal{Z}})) * \mathcal{V}^{\mathrm{T}}$ , where  $P_{\tau}(\overline{\mathcal{Z}})$  is an  $f$ -diagonal 3rd-order tensor, whose diagonal elements can be found by using the GST algorithm introduced in [12].
Now the solution of (24) is:
$$
\boldsymbol {\mathcal {J}} ^ {*} = \Gamma_ {\frac {\lambda}{\rho}} (\boldsymbol {\mathcal {H}} + \frac {\boldsymbol {\mathcal {Y}} _ {2}}{\rho}). \tag {26}
$$
Finally, the optimization procedure for Multi-View Clustering via Orthogonal non-negative Tensor Factorization (Orth-NTF) is outlined in Algorithm 1.
Algorithm 1 Multi-View Clustering via Orthogonal non-negative Tensor Factorization (Orth-NTF)
Input: Data matrices  $\{\mathbf{X}^{(v)}\}_{v = 1}^{V}\in \mathbb{R}^{N\times dv}$  ; anchors numbers  $m$  ; cluster number  $K$    
Output: Cluster labels  $\mathbf{Y}$  of each data points.   
Initialize:  $\mu = 10^{-5},\rho = 10^{-5},\eta = 1.6,\pmb {\gamma}_1 = 0,\pmb {\gamma}_2 = 0,\overline{\mathbf{Q}}^{(v)}$  is identity matrix;   
1: Compute graph matrix  $\mathbf{S}^{(v)}$  of each views;   
2: while not condition do   
3: Update  $\overline{\mathcal{G}}^{(v)}$  by solving (11);   
4: Update  $\overline{\mathcal{H}}^{(v)}$  by solving (19);   
5: Update  $\overline{\mathcal{Q}}^{(v)}$  by solving (22);   
6: Update  $\mathcal{T}$  by using (24);   
7: Update  $\pmb {y}_1,\pmb {y}_2,\pmb {\mu}$  and  $\rho$  ..  $\pmb {y}_1 = \pmb {y}_1 + \mu (\pmb {H} - \pmb {\mathcal{Q}}),\pmb {y}_2 = \pmb {y}_2 + \mu (\pmb {H} - \pmb {\mathcal{I}}),\mu = \min (\eta \mu ,10^{13}),$ $\rho = \min (\eta \rho ,10^{13})$  .   
8: end while   
9: Calculate the  $K$  clusters by using   
 $\mathbf{H} = \sum_{v = 1}^{V}\mathbf{H}^{(v)} / V;$    
10: return Clustering result (The position of the largest element in each row of the indicator matrix is the label of the corresponding sample).
Theorem 2. [Convergence Analysis of Algorithm 1] Let  $\mathcal{P}_k = \{\mathcal{Q}_k, \mathcal{H}_k, \mathcal{G}_k, \mathcal{J}_k, \mathcal{V}_{2,k}, \mathcal{V}_{1,k}\}$ ,  $1 \leq k < \infty$  in (7) be a sequence generated by Algorithm 1, then
1.  $\mathcal{P}_k$  is bounded with the assumption  $\lim_{k\to 0}\max \{\mu_k,\rho_k\} (\bar{\mathcal{H}}_{k + 1}^{(v)} - \bar{\mathcal{H}}_k^{(v)}) = 0;$
2. Any accumulation point of  $\mathcal{P}_k$  is a stationary KKT point of (7).
The proof will be provided in the appendix and we need to mention that the KKT conditions can be used to determine the stop conditions for Algorithm 1, which are  $\| \mathcal{Q}_k - \mathcal{H}_k\|_{\infty}\leq \varepsilon$ $\| \mathcal{Q}_k - \mathcal{I}_k\|_{\infty}\leq \varepsilon$
Table 1: Multi-view datasets used in our experiments
#Dataset	#Samples	#View	#Class	#Feature
MSRC	210	5	7	24,576,512,256,254
HandWritten4	2000	4	10	76,216,47,6
Mnist4	4000	3	4	30,9,30
Reuters	18758	5	6	21531,24892,34251,15506,11547
Noisy MNIST	50000	2	10	784,784
For Orth-NTF, the storage requirements for  $\mathcal{G},\mathcal{H},\mathcal{Q},\mathcal{J},\mathcal{Y}_1$  and  $\mathcal{V}_{2}$  have complexities of  $\mathcal{O}(V(m + k)n),\mathcal{O}(V(n + k)k),\mathcal{O}(Vnk),\mathcal{O}(Vnk),\mathcal{O}(Vnk)$  and  $\mathcal{O}(Vnk)$ , respectively. Combining these, the total storage complexity for Orth-NTF is  $\mathcal{O}(Vnm + vk^2 +6Vnk)$ .
For the computational complexity, the process of constructing  $\mathcal{S}$  has a computational complexity of  $\mathcal{O}(Vnm + Vnm\log (m))$ . When updating the four variables,  $\mathcal{G},\mathcal{H},\mathcal{Q}$  and  $\mathcal{J}$ , their respective computational complexities are  $\mathcal{O}(Vnm + Vnm\log (m))$ ,  $\mathcal{O}(Vm^2 k + Vmk^2)$ ,  $\mathcal{O}(Vnk)$  and  $\mathcal{O}(2Vnk\log (Vk) + V^2 kn)$ . Given that  $m$ ,  $n$ ,  $k$  and  $V$  are relatively small constants, the primary computational cost associated with updating the variables stands at  $\mathcal{O}(Vnkm + Vm^2 k)$ . Summing it all up, the overall computational complexity of our proposed method is  $\mathcal{O}(Vnm + Vm^2 k)$ .
(a) MSRC
(b) HandWritten4
(c)Mnist4
Figure 3: Convergence experiments on MSRC, HandWritten4, Mnist4 and AWA.
(d) AWA
In this section, we demonstrate the performance of our proposed method through extensive experiments. It is compared with plenty of state-of-art multi-view clustering algorithms on some multi-view datasets. We evaluate the clustering performance by applying 7 metrics used widely, i.e., 1) ACC; 2) NMI; 3) Purity; 4) PRE; 5) REC; 6) F-score; and 7) ARI. The higher the value the better the clustering results for all metrics mentioned above. Detailed experimental configurations and hyper-parameters on each dataset are in the appendix.
5.1 Datasets and Compared Baselines Methods
The following multi-view datasets are selected to examine our proposed method. The details of the datasets are shown in Table 1. MSRC [34]; HandWritten4 [10]; Mnist4 [6]; AWA [11]; Reuters [2]; Noisy MNIST [32]; We choose the following 8 state-of-art multi-view clustering algorithms to compare with our proposed methods: AMGL [25]; MVGL [40]; CSMSC [24]; GMC [30]; LMVSC [17]; SMSC [16]; SFMC [21] FMCNOF[37]; FPMVS-CAG [31]; ETLMSC [35]; MSC-BG [38];
5.2 Experiments Result
The clustering performances are listed in Table 2 and Table 3. They contain four medium-scale datasets and two large-scale datasets. The corresponding experimental configurations and descriptions are included in the appendix. It is clear that our algorithm outperforms the other baseline algorithms on most of the datasets.
This advantage may stem from the fact that our model directly factorizes the tensorized anchor graph—comprised of anchor graphs from various views—into the product of two non-negative tensors, one being an index tensor. As a result, our model effectively captures both the spatial structural information and the complementary data present in the anchor graphs from different
perspectives. Additionally, with orthogonal and non-negative constraints in place, our model offers clear interpretability for clustering. This means that each row of the indicator matrix for every view contains a single non-zero element, with its position indicating the label of the associated sample. Consequently, our model can immediately provide the label without necessitating any post-processing, a step which other methods still require.
5.3 Ablation Experiments
We do some ablation experiments on orthogonal constraint and Schatten p-norm on four datasets as shown in Table 4. It can be found that, tensor Schatten p-norm regularization is overall superior to orthogonal constraint. The reason is that tensor Schatten p-norm regularization effectively characterizes both the complementary information and spatial structure information of index matrices of different views. Compared to orthogonal and tensor Schatten p-norm constraints, Joint constraints have great contribution for clustering.
5.4 Experiments of convergence
We optimize the objective function iteratively by introducing two auxiliary variables  $\mathcal{Q}$  and  $\mathcal{I}$ . We test the convergence of our algorithm by checking the difference between  $\mathcal{H}$  and the two auxiliary
Table 3: Clustering results and running time (sec.) on Reuters and NoisyMNIST. ("OM" means out of memory and "-" means the algorithm takes more than three hours to calculate.)
Dataset	Reuters	Dataset	Noisy MNIST
Metrics	ACC	NMI	Purity	Time	Metrics	ACC	NMI	Purity	Time
AMGL [25] (JCAI' 16)	OM	OM	OM	OM	AMGL [25] (JCAI' 16)	OM	OM	OM	OM	OM
MVGL [40] (TCYB' 17)	OM	OM	OM	OM	MVGL [40] (TCYB' 17)	OM	OM	OM	OM	OM
CSMSC [24] (AAAI' 18)	OM	OM	OM	OM	CSMSC [24] (AAAI' 18)	OM	OM	OM	OM	OM
GMC [30] (TKDE' 19)	-	-	-	-	GMC [30] (TKDE' 19)	-	-	-	-	-
LMVSC [17] (AAAI' 20)	0.587±0.000	0.335±0.000	0.616±0.000	150.51	LMVSC [17] (AAAI' 20)	0.388±0.000	0.344±0.000	0.434±0.000	151.14
SMSC [16] (Inf Fusion' 20)	OM	OM	OM	OM	SMSC [16] (Inf Fusion' 20)	OM	OM	OM	OM
FCMNOF [37] (TIP' 21)	0.343±0.007	0.125±0.037	0.358±0.052	186.45	FCMNOF [37] (TIP' 21)	0.333±0.038	0.237±0.032	0.340±0.032	192.60
FPMVS-CAG [31] (TIP' 21)	0.576±0.000	0.359±0.000	0.637±0.000	2252.20	FPMVS-CAG [31] (TIP' 21)	0.554±0.000	0.513±0.000	0.567±0.000	2258.06
SFMC [21] (TPAMI' 22)	0.602±0.000	0.354±0.000	0.552±0.000	494.68	SFMC [21] (TPAMI' 22)	0.699±0.000	0.681±0.000	0.727±0.000	495.90
ETLMSC [35] (TIP' 19)	OM	OM	OM	OM	ETLMSC [35] (TIP' 19)	OM	OM	OM	OM
MSC-BG [38] (TIP' 22)	0.640±0.000	0.484±0.000	0.686±0.000	462.33	MSC-BG [38] (TIP' 22)	OM	OM	OM	OM
ours	0.694±0.000	0.686±0.000	0.809±0.000	557.09	ours	0.701±0.000	0.729±0.000	0.747±0.000	563.66
Table 4: Ablation experiments on MSRC, HandWritten4, Mnist4 and AWA. ( $\mathcal{C}_{\text{orth}}$  and  $\mathcal{C}_{Sp}$  represent the orthogonal constraint and the tensor Schatten  $p$ -norm regularization, respectively.)
Situations	MSRC	HandWritten4
Corth	CSp	ACC	NMI	Purity	ACC	NMI	Purity
X	X	0.776	0.653	0.776	0.559	0.572	0.581
✓	X	0.785	0.659	0.785	0.594	0.621	0.615
X	✓	0.886	0.819	0.886	0.725	0.741	0.725
✓	✓	0.990	0.987	0.990	0.985	0.969	0.985
Situations	Mnist4	AWA
Corth	CSp	ACC	NMI	Purity	ACC	NMI	Purity
X	X	0.898	0.750	0.898	0.020	0.020	0.020
✓	X	0.905	0.759	0.905	0.092	0.147	0.097
X	✓	0.912	0.789	0.912	0.212	0.125	0.213
✓	✓	0.977	0.926	0.977	0.646	0.815	0.670
variables. The result of the experiment is shown in Fig 7. It is evident that when the iteration reaches around 80, the difference decreases significantly until it is about zero finally. When the number of iteration increases, the clustering metric (such as ACC) overall improves gradually and tends to be constant with the convergence of the algorithm. It also indicates that our method has good clustering performances.
More experiments can be found in the appendix.
We are concerned in this paper with multi-view clustering based on semi-non-negative tensor factorization (Orth-NTF) with one-side orthogonal constraint. Our proposed model extends NMF to Orth-NTF so that the spatial structure information of the multi-view data can be utilized to improve the clustering performance. In addition, the complementary information embedded in different views is fully leveraged by imposing the tensor Schatten  $p$ -norm composed of cluster indicator matrices. To diminish the computational complexity, we adopt anchor graphs instead of the original multi-view data. Also, we provide an optimization algorithm for the proposed method and validate the effectiveness of this approach in extensive experiments on different datasets.
The authors would like to thank the anonymous reviewers, AC and PC for their constructive comments and suggestions. This work was supported in part by National NSFC under Grants 62176203, in part by the Natural Science Foundation of Shandong Province under Grant ZR202102180986, in part by the State Key Laboratory of Multimodal Artificial Intelligence Systems under Grant 202200035, in part by Guangxi Key Laboratory of Digital Infrastructure under Grant GXDIOP2023010, in part by the Fundamental Research Funds for the Central Universities, the Innovation Fund of Xidian University.
A Proof of Convergence
A.1 Proof of the 1st part
Lemma 3 (Proposition 6.2 of [20]). Suppose  $F: \mathbb{R}^{n_1 \times n_2} \to \mathbb{R}$  is represented as  $F(X) = f \circ \sigma(X)$ , where  $X \in \mathbb{R}^{n_1 \times n_2}$  with SVD  $X = U\mathrm{diag}(\sigma_1, \ldots, \sigma_n)V^{\mathrm{T}}$ ,  $n = \min(n_1, n_2)$ , and  $f$  is differentiable. The gradient of  $F(X)$  at  $X$  is
$$
\frac {\partial F (X)}{\partial X} = U \operatorname {d i a g} (\theta) V ^ {\mathrm {T}}, \tag {27}
$$
where  $\theta = \frac{\partial f(y)}{\partial y}\big|_{y = \sigma (X)}$
To minimize  $\bar{\mathcal{Q}}^{(v)}$  at step  $k + 1$  in (21) in the main body, the optimal  $\bar{\mathcal{Q}}_{k + 1}^{(v)}$  needs to satisfy the first-order optimal condition
$$
\bar {\mathcal {Q}} _ {k + 1} ^ {(v)} = \bar {\mathcal {H}} _ {k} ^ {(v)} + \frac {\bar {\mathcal {V}} _ {1 , k}}{\mu_ {k}}.
$$
By using the updating rule  $\bar{\mathcal{V}}_{1,k + 1}^{(v)} = \bar{\mathcal{V}}_{1,k}^{(v)} + \mu_k(\bar{\mathcal{H}}_k^{(v)} - \bar{\mathcal{Q}}_k^{(v)})$ , we have
$$
\frac {\bar {\mathcal {Y}} _ {1 , k + 1} ^ {(v)}}{\mu_ {k}} + \left(\bar {\mathcal {Q}} _ {k} ^ {(v)} - \bar {\mathcal {Q}} _ {k + 1} ^ {(v)}\right) = 0.
$$
According to our assumption  $\lim_{k\to 0}\mu_k(\bar{\mathcal{Q}}_{k + 1}^{(v)} - \bar{\mathcal{Q}}_k^{(v)}) = 0$ , we know  $\pmb{\nu}_{1,k + 1}$  is bounded.
To minimize  $\mathcal{J}$  at step  $k + 1$  in (24) in the main body, the optimal  $\mathcal{J}_{k + 1}$  needs to satisfy the first-order optimal condition
$$
\lambda \nabla_ {\mathcal {J}} \| \mathcal {J} _ {k + 1} \| _ {\widehat {\mathscr {S}}} ^ {p} + \rho_ {k} \left(\mathcal {J} _ {k + 1} - \mathcal {H} _ {k + 1} - \frac {1}{\rho_ {k}} \mathcal {Y} _ {2, k}\right) = 0.
$$
Recall that when  $0 < p < 1$ , in order to overcome the singularity of  $(|\eta|^p)' = p\eta / |\eta|^{2 - p}$  near  $\eta = 0$ , we consider for  $0 < \epsilon \ll 1$  the approximation
$$
\partial | \eta | ^ {p} \approx \frac {p \eta}{\max  \left\{\epsilon^ {2 - p} , | \eta | ^ {2 - p} \right\}}.
$$
Letting  $\overline{\mathcal{J}}^{(i)} = \overline{\mathcal{U}}^{(i)}\mathrm{diag}\left(\sigma_j(\overline{\mathcal{J}}^{(i)})\right)\overline{\mathcal{V}}^{(i)\mathrm{H}}$ , then it follows from Lemma 3 that
$$
\frac {\partial \left\| \overline {{\boldsymbol {\mathcal {J}}}} ^ {(i)} \right\| _ {\mathfrak {S} ^ {\flat}} ^ {p}}{\partial \overline {{\boldsymbol {\mathcal {J}}}} ^ {(i)}} = \overline {{\boldsymbol {\mathcal {U}}}} ^ {(i)} \operatorname {d i a g} \left(\frac {p \sigma_ {j} (\overline {{\boldsymbol {\mathcal {J}}}} ^ {(i)})}{\max  \left\{\epsilon^ {2 - p} , \left| \sigma_ {j} (\overline {{\boldsymbol {\mathcal {J}}}} ^ {(i)}) \right| ^ {2 - p} \right\}}\right) \overline {{\boldsymbol {\mathcal {V}}}} ^ {(i) \mathrm {H}}.
$$
And then one can obtain
$$
\begin{array}{l} \frac {p \sigma_ {j} (\overline {{\mathcal {J}}} ^ {(i)})}{\max  \{\epsilon^ {2 - p} , | \sigma_ {j} (\overline {{\mathcal {J}}} ^ {(i)}) | ^ {2 - p} \}} \leq \frac {p}{\epsilon^ {1 - p}} \\ \Longrightarrow \left\| \frac {\partial \| \overline {{\boldsymbol {\mathcal {J}}}} ^ {(i)} \| _ {\widehat {\mathscr {S} p}} ^ {p}}{\partial \overline {{\boldsymbol {\mathcal {J}}}} ^ {(i)}} \right\| _ {F} ^ {2} \leq \sum_ {i = 1} ^ {n} \frac {p ^ {2}}{\epsilon^ {2 (1 - p)}}. \\ \end{array}
$$
So  $\frac{\partial\|\overline{\mathcal{T}}\|_{\widehat{\mathcal{S}^p}}^p}{\partial\overline{\mathcal{T}}}$  is bounded.
Let us denote  $\widetilde{\mathbf{F}}_V = \frac{1}{\sqrt{V}}\mathbf{F}_V$ ,  $\mathbf{F}_V$  is the discrete Fourier transform matrix of size  $V\times V$ ,  $\mathbf{F}_V^{\mathrm{H}}$  denotes its conjugate transpose. For  $\mathcal{J} = \overline{\mathcal{J}}\times_{3}\widetilde{\mathbf{F}}_{V}$  and using the chain rule in matrix calculus, one can obtain that
$$
\nabla_ {\mathcal {I}} \| \mathcal {J} \| _ {\mathfrak {S} ^ {\mathcal {P}}} ^ {p} = \frac {\partial \| \mathcal {J} \| _ {\mathfrak {S} ^ {\mathcal {P}}} ^ {p}}{\partial \overline {{\mathcal {J}}}} \times_ {3} \widetilde {\mathbf {F}} _ {V} ^ {\mathrm {H}}
$$
is bounded.
$$
\begin{array}{l} \mathcal {Y} _ {1, k + 1} = \mathcal {Y} _ {2, k} + \rho_ {k} \left(\mathcal {H} _ {k + 1} - \mathcal {J} _ {k + 1}\right) \\ \Rightarrow \lambda \nabla_ {\mathcal {I}} \| \mathcal {I} _ {k + 1} \| _ {\mathscr {S} ^ {p}} ^ {p} = \mathcal {Y} _ {2, k + 1}, \\ \end{array}
$$
thus  $\mathcal{V}_{2,k + 1}$  appears to be bounded.
Moreover, by using the updating rule  $\mathcal{V}_1 = \mathcal{V}_1 + \mu (\mathcal{H} - \mathcal{Q})$ ,  $\mathcal{V}_2 = \mathcal{V}_2 + \rho (\mathcal{H} - \mathcal{J})$ , we can deduce  $(i = 1,2)$
$$
\begin{array}{l} \mathcal {L} \left(\mathcal {Q} _ {k + 1}, \mathcal {G} _ {k + 1}, \mathcal {H} _ {k + 1}, \mathcal {J} _ {k + 1}; \mathcal {Y} _ {i, k}\right) \tag {28} \\ \leq \mathcal {L} \left(\mathcal {Q} _ {k}, \mathcal {G} _ {k}, \mathcal {H} _ {k}, \mathcal {J} _ {k}; \mathcal {Y} _ {i, k}\right) \\ = \mathcal {L} \left(\mathcal {Q} _ {k}, \mathcal {G} _ {k}, \mathcal {H} _ {k}, \mathcal {J} _ {k}; \mathcal {Y} _ {i, k - 1}\right) \\ + \frac {\rho_ {k} + \rho_ {k - 1}}{2 \rho_ {k - 1} ^ {2}} \| \mathcal {Y} _ {2, k} - \mathcal {Y} _ {2, k - 1} \| _ {F} ^ {2} + \frac {\| \mathcal {Y} _ {2 , k} \| _ {F} ^ {2}}{2 \rho_ {k}} - \frac {\| \mathcal {Y} _ {2 , k - 1} \| _ {F} ^ {2}}{2 \rho_ {k - 1}} \\ + \frac {\mu_ {k} + \mu_ {k - 1}}{2 \mu_ {k - 1} ^ {2}} \| \boldsymbol {y} _ {1, k} - \boldsymbol {y} _ {1, k - 1} \| _ {F} ^ {2} + \frac {\| \boldsymbol {y} _ {1 , k} \| _ {F} ^ {2}}{2 \mu_ {k}} - \frac {\| \boldsymbol {y} _ {1 , k - 1} \| _ {F} ^ {2}}{2 \mu_ {k - 1}}. \\ \end{array}
$$
Thus, summing two sides of (28) from  $k = 1$  to  $n$ , we have
$$
\begin{array}{l} \mathcal {L} \left(\mathcal {Q} _ {n + 1}, \mathcal {G} _ {n + 1}, \mathcal {H} _ {n + 1}, \mathcal {J} _ {n + 1}; \mathcal {Y} _ {i, n}\right) \\ \leq \mathcal {L} \left(\mathcal {Q} _ {1}, \mathcal {G} _ {1}, \mathcal {H} _ {1}, \mathcal {J} _ {1}; \mathcal {Y} _ {i, 0}\right) \\ + \frac {\left\| \boldsymbol {y} _ {2 , n} \right\| _ {F} ^ {2}}{2 \rho_ {n}} - \frac {\left\| \boldsymbol {y} _ {2 , 0} \right\| _ {F} ^ {2}}{2 \rho_ {0}} + \sum_ {k = 1} ^ {n} \left(\frac {\rho_ {k} + \rho_ {k - 1}}{2 \rho_ {k - 1} ^ {2}} \left\| \boldsymbol {y} _ {2, k} - \boldsymbol {y} _ {2, k - 1} \right\| _ {F} ^ {2}\right) \tag {29} \\ + \frac {\| \boldsymbol {\mathcal {Y}} _ {1 , n} \| _ {F} ^ {2}}{2 \mu_ {n}} - \frac {\| \boldsymbol {\mathcal {Y}} _ {1 , 0} \| _ {F} ^ {2}}{2 \mu_ {0}} + \sum_ {k = 1} ^ {n} \left(\frac {\mu_ {k} + \mu_ {k - 1}}{2 \mu_ {k - 1} ^ {2}} \| \boldsymbol {\mathcal {Y}} _ {1, k} - \boldsymbol {\mathcal {Y}} _ {1, k - 1} \| _ {F} ^ {2}\right). \\ \end{array}
$$
$$
\sum_ {k = 1} ^ {\infty} \frac {\rho_ {k} + \rho_ {k - 1}}{2 \rho_ {k - 1} ^ {2}} <   \infty , \sum_ {k = 1} ^ {\infty} \frac {\mu_ {k} + \mu_ {k - 1}}{2 \mu_ {k - 1} ^ {2}} <   \infty ,
$$
we have the right-hand side of (29) is finite and thus  $\mathcal{L}(\mathcal{Q}_{n + 1},\mathcal{G}_{n + 1},\mathcal{H}_{n + 1},\mathcal{I}_{n + 1};\mathcal{Y}_{i,n})$  is bounded. Notice from (7) in the main body
$$
\begin{array}{l} \mathcal {L} \left(\mathcal {Q} _ {n + 1}, \mathcal {G} _ {n + 1}, \mathcal {H} _ {n + 1}, \mathcal {J} _ {n + 1}; \mathcal {Y} _ {i, n}\right) \\ = \sum_ {v = 1} ^ {V} \left\| \bar {\boldsymbol {S}} ^ {(v)} - \bar {\mathcal {H}} _ {n + 1} ^ {(v)} \left(\bar {\boldsymbol {\mathcal {G}}} _ {n + 1} ^ {(v)}\right) ^ {T} \right\| _ {F} ^ {2} \\ + \lambda \| \mathcal {J} _ {n + 1} \| _ {\mathscr {S} ^ {\circ}} ^ {p} + \frac {\rho_ {n}}{2} \| \mathcal {H} _ {n + 1} - \mathcal {J} _ {n + 1} + \frac {\mathcal {Y} _ {2 , n}}{\rho_ {n}} \| _ {F} ^ {2} \\ + \frac {\mu_ {n}}{2} \sum_ {v = 1} ^ {V} \| \bar {\mathcal {H}} _ {n + 1} ^ {(v)} - \bar {\mathcal {Q}} _ {n + 1} ^ {(v)} + \frac {\bar {\mathcal {Y}} _ {1 , n + 1} ^ {(v)}}{\mu_ {n}} \| _ {F} ^ {2}, \tag {30} \\ \end{array}
$$
and each term of (30) is nonnegative, following from the boundedness of  $\mathcal{L}(\mathcal{Q}_{n+1}, \mathcal{G}_{n+1}, \mathcal{H}_{n+1}, \mathcal{J}_{n+1}; \mathcal{Y}_{i,n})$ , we can deduce each term of (30) is bounded. And  $\| \mathcal{J}_{n+1} \|_{\mathcal{S}}^p$  being bounded implies that all singular values of  $\mathcal{J}_{n+1}$  are bounded and hence  $\| \mathcal{J}_{n+1} \|_F^2$  (the sum of squares of singular values) is bounded. Therefore, the sequence  $\{\mathcal{J}_k\}$  is bounded.
$$
\mathcal {Y} _ {1, k + 1} = \mathcal {Y} _ {1, k} + \mu_ {k} (\mathcal {Q} _ {k} - \mathcal {H} _ {k}) \Longrightarrow \mathcal {H} _ {k} = \mathcal {Q} _ {k} + \frac {\mathcal {Y} _ {1 , k + 1} - \mathcal {Y} _ {1 , k}}{\mu_ {k}},
$$
and in light of the boundedness of  $\mathcal{Q}_k, \mathcal{V}_{1,k}$ , it is clear that  $\mathcal{H}_k$  is also bounded.
And from (8) in the main body, it is evident that  $\| \bar{\pmb{G}}_k^{(v)}\| _F^2\leq \| (\bar{\pmb{S}}^{(v)})^\mathrm{T}\| _F^2\| \bar{\pmb{H}}_k^{(v)}\| _F^2$  so  $\bar{\pmb{G}}_k^{(v)}$  is also bounded. So  $\pmb{G}_k$  is bounded.
A.2 Proof of the 2nd part
From Weierstrass-Bolzano theorem, there exists at least one accumulation point of the sequence  $\mathcal{P}_k$ . We denote one of the points  $\mathcal{P}^* = \{\mathcal{H}^*, \mathcal{Q}^*, \mathcal{G}^*, \mathcal{J}^*, \mathcal{V}_1^*, \mathcal{V}_2^*\}$ . Without loss of generality, we assume  $\{\mathcal{P}_k\}_{k=1}^{+\infty}$  converge to  $P^*$ .
Note that from the updating rule for  $\pmb{\nu}_{1}$ , we have
$$
\mathcal {Y} _ {1, k + 1} = \mathcal {Y} _ {1, k} + \mu_ {k} (\mathcal {H} _ {k} - \mathcal {Q} _ {k}) \Longrightarrow \mathcal {Q} ^ {*} = \mathcal {H} ^ {*}.
$$
Note that from the updating rule for  $\pmb{\nu}_{2}$ , we have
$$
\mathcal {Y} _ {2, k + 1} = \mathcal {Y} _ {2, k} + \rho_ {k} (\mathcal {H} _ {k} - \mathcal {J} _ {k}) \Longrightarrow \mathcal {J} ^ {*} = \mathcal {H} ^ {*}.
$$
In the  $\bar{\pmb{G}}^{(v)}$  -subproblem (8) in the main body, we have
$$
\bar {\mathbf {g}} _ {k} ^ {(v)} = (\bar {\mathbf {S}} ^ {(v)}) ^ {\mathrm {T}} \bar {\mathbf {H}} _ {k} ^ {(v)} \Longrightarrow \bar {\mathbf {g}} ^ {(v) *} = (\bar {\mathbf {S}} ^ {(v)}) ^ {\mathrm {T}} \bar {\mathbf {H}} ^ {(v) *}.
$$
In the  $\mathcal{J}$ -subproblem (24) in the main body, we have