SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jing Li
Xi'an, Shaanxi, China
jing1xd@stu.xidian.edu.cn
Quanxue Gao *
Xidian University i'an, Shaanxi, China gao@xidian.edu.cn
Qianqian Wang
Xidian University Xi'an, Shaanxi, China qqwang@xidian.edu.cn
Harbin, Heilongjiang, China
yangmingmath@gmail.com
Wei Xia
xdweixia@gmail.com
Multi-view clustering (MVC) based on non-negative matrix factorization (NMF) and its variants have attracted much attention due to their advantages in clustering interpretability. However, existing NMF-based multi-view clustering methods perform NMF on each view respectively and ignore the impact of between-view. Thus, they can't well exploit the within-view spatial structure and between-view complementary information. To resolve this issue, we present orthogonal non-negative tensor factorization (Orth-NTF) and develop a novel multi-view clustering based on Orth-NTF with one-side orthogonal constraint. Our model directly performs Orth-NTF on the 3rd-order tensor which is composed of anchor graphs of views. Thus, our model directly considers the between-view relationship. Moreover, we use the tensor Schatten  $p$ -norm regularization as a rank approximation of the 3rd-order tensor which characterizes the cluster structure of multi-view data and exploits the between-view complementary information. In addition, we provide an optimization algorithm for the proposed method and prove mathematically that the algorithm always converges to the stationary KKT point. Extensive experiments on various benchmark datasets indicate that our proposed method is able to achieve satisfactory clustering performance.
As one of the most typical methods in unsupervised learning, clustering has a wide scope of application [26; 4; 1] to assign data to different clusters according to the information describing the objects and their relationships. Non-negative matrix factorization (NMF) [19] is one of the representative methods of clustering, which is proved to be equivalent to K-means clustering [7]. Despite the widespread use of NMF, there are some drawbacks that have prompted some variants of NMF [8; 5; 27; 9; 3].
In particular, the one-side G-orthogonal NMF [8] can guarantee the uniqueness of the solution of matrix factorization and has excellent clustering interpretation. Also, Ding et al. proposed the semi-NMF [9]. The data matrix and one of the factor matrices are unconstrained, which allows semi-NMF to be more suitable for applications where the input data is mixed with positive and negative numbers.
Although the above methods can achieve outstanding clustering performance, they are all single-view clustering methods and cannot be adopted straightforwardly for multi-view clustering.
Multi-view clustering tends to achieve superior performance compared to traditional single-view clustering owing to the capability to leverage the complementary information embedded in the different views. Considering the superiority of MVC and NMF, lots of NMF-based multi-view clustering methods have been proposed [13; 28; 23; 33; 15; 29; 14; 37]. The NMF-based multi-view clustering methods can save time and space because it is unnecessary to construct affinity graphs while graph-based methods have to. However, usually, they decompose the original data matrix directly, which leads to a dramatic reduction in the efficiency of the algorithm when the dimension of the original data is huge.
Inspired by the idea of anchor graph, the above issues can be solved by carrying out NMF on the anchor graph [37]. Due to the fact that the dimension of the anchor graph is considerably smaller than the original affinity graph, it follows that the clustering efficiency can be improved. However, as is well-known, there exist two ways of NMF-based multi-view clustering methods. One is to integrate different views first and then implement the NMF on the integrated matrix; the other is to perform the NMF on different views separately and then integrate the result from each view. Both ways are essentially applications of NMF on a single view, and both need to reduce the multi-view data into matrices in the process, which causes the loss of the original spatial information.
To fix the aforesaid issues, we proposed a novel multi-view clustering based on Orth-NTF with one-side orthogonal constraint. Specifically, Non-negative Matrix Factorization (NMF) is tailored primarily for second-order matrices. When processing third-order tensors, there's a need to first transform the tensor into a matrix before applying NMF. This step can lead to a loss of inherent spatial structural information from the third-order tensor. In contrast, Non-negative Tensor Factorization (NTF) sidesteps this issue. NTF directly decomposes third-order tensors. This ensures that the NTF not only acknowledges the relationships between the views but also harnesses the complementary information they offer. Fig 1 delineates the distinction between traditional NMF-based clustering techniques and our NTF-based approach. Furthermore, by incorporating an orthogonal constraint, our model offers distinct physical interpretability for clustering. This suggests that each row of the indicator matrix contains a single non-zero element, and the position of this element directly corresponds to the label of the respective sample. A large number of experiments have shown that our methods have excellent clustering performance.
The main contributions are summarized below:
- We introduce orthogonal non-negative tensor factorization, which considers the between-view relationship directly. Also, we use tensor Schatten  $p$ -norm regularization to characterize the cluster structure of multi-view data and can exploit the complementary information of between views.
- We regard the anchor graph obtained from the original data as the input of the non-negative matrix factorization, which reduces the complexity of our proposed algorithm considerably.
- We provide an optimization algorithm for the proposed method and prove it always converges to the KKT stationary point mathematically. The effectiveness of its application on tensorial G-orthogonal non-negative matrix factorization is demonstrated by extensive experiments.
In recent years, multi-view clustering (MVC) has received increasing attention due to its excellent clustering performance. Also, non-negative matrix factorization (NMF) is an efficient technique in single-view clustering, which can generate excellent clustering results that are easy to interpret, and many NMF-based variants have been proposed. Therefore, multi-view clustering-based NMF and its variants have attracted tremendous interest recently.
As the first investigation of the multi-view clustering method based on joint NMF, multiNMF [23] implements NMF at each view and pushes the different clustering results of each view to a consensus. It provides a new viewpoint for the subsequent NMF-based MVC methods. Influenced by multiNMF, He et al. proposed a multi-view clustering method combining NMF with similarity [15]. It implements NMF on each view as in multiNMF. In addition, it sets a weight for a different view and introduces
Figure 1: Comparison between traditional NMF-based multi-view clustering method (left) and Orth-NTF (right).
Figure 2: 3rd-order tensor construction process.
a similarity matrix of data points to extract consistent information from different views. To better detect the geometric structure of inner-view space, Wang et al. [33] introduced graph regularization into the NMF-based multi-view clustering method to improve clustering performance. Considering the above work, Wang et al. [29] proposed a graph regularization multi-view clustering method based on concept factorization (CF). CF is a variant of NMF and it is suitable for handling data containing negative.
As the size of data grows, lots of methods to accelerate matrix factorization are presented. Wang et al. [28] proposed a fast non-negative matrix triple factorization method. It constrains the factor matrix of NMF to a clustering indicator matrix, thereby avoiding the post-processing of the factor matrix. Inspired by the work of Wang, Han et al. [14] constrained the intermediate factor matrix in the triple factorization to a diagonal matrix, reducing the number of matrix multiplications in the solution process. Another idea to deal with large-scale multi-view data is to introduce anchor graphs, since the number of anchor points is much smaller than the number of original data, multi-view clustering methods based on anchor graphs tend to reduce the computational complexity and thus are able to deal with large-scale data [22; 17; 21]. Considering that previous NMF-based multi-view clustering methods are performed directly on the original data, Yang et al. [37] introduced an anchor graph as the input of G-orthogonal NMF. The efficiency of matrix factorization is indeed improved due to the introduction of anchor graph.
Despite the fact that existing NMF-based multi-view clustering methods can perform the clustering tasks excellently, they apply NMF to each view independently. Subsequently, they combine the low-dimensional representations from different perspectives to arrive at a unified shared representation. This approach often overlooks the interrelationships between the views, which are crucial for clustering.
3 Notations
We introduce the notations used throughout this paper. We use bold calligraphy letters for 3rd-order tensors,  $\mathcal{H} \in \mathbb{R}^{n_1 \times n_2 \times n_3}$ , bold upper case letters for matrices,  $\mathbf{H}$ , bold lower case letters for vectors,  $\mathbf{h}$ , and lower case letters such as  $h_{ijk}$  for the entries of  $\mathcal{H}$ . Moreover, the  $i$ -th frontal slice of  $\mathcal{H}$  is  $\mathcal{H}^{(i)}$ .  $\overline{\mathcal{H}}$  is the discrete Fourier transform (DFT) of  $\mathcal{H}$  along the third dimension,  $\overline{\mathcal{H}} = \mathrm{fft}(\mathcal{H}, [], 3)$ . Thus,  $\mathcal{H} = \mathrm{ifft}(\overline{\mathcal{H}}, [], 3)$ . The trace and transpose of matrix  $\mathbf{H}$  are expressed as  $\mathrm{tr}(\mathbf{H})$  and  $\mathbf{H}^{\mathrm{T}}$ . The F-norm of  $\mathcal{H}$  is denoted by  $\| \mathcal{H} \|_F$ .
Definition 1 (t-product [18]). Suppose  $\mathcal{A} \in \mathbb{R}^{n_1 \times m \times n_3}$  and  $\mathcal{B} \in \mathbb{R}^{m \times n_2 \times n_3}$ , the t-product  $\mathcal{A} * \mathcal{B} \in \mathbb{R}^{n_1 \times n_2 \times n_3}$  is given by
$$
\boldsymbol {\mathcal {A}} * \boldsymbol {\mathcal {B}} = \operatorname {i f t} (\mathrm {b d i a g} (\overline {{\mathbf {A B}}}), [, 3),
$$
where  $\overline{\mathbf{A}} = \mathrm{bdiag}(\overline{\mathcal{A}})$  and it denotes the block diagonal matrix. The blocks of  $\overline{\mathbf{A}}$  are frontal slices of  $\overline{\mathcal{A}}$ .
Definition 2. [12] Given  $\mathcal{H} \in \mathbb{R}^{n_1 \times n_2 \times n_3}$ ,  $h = \min(n_1, n_2)$ , the tensor Schatten p-norm of  $\mathcal{H}$  is defined as
$$
\left\| \boldsymbol {\mathcal {H}} \right\| _ {\mathscr {S} ^ {p}} = \left(\sum_ {i = 1} ^ {n _ {3}} \left\| \overline {{\boldsymbol {\mathcal {H}}}} ^ {(i)} \right\| _ {\mathscr {S} ^ {p}} ^ {p}\right) ^ {\frac {1}{p}} = \left(\sum_ {i = 1} ^ {n _ {3}} \sum_ {j = 1} ^ {h} \sigma_ {j} \left(\overline {{\boldsymbol {\mathcal {H}}}} ^ {(i)}\right) ^ {p}\right) ^ {\frac {1}{p}}, \tag {1}
$$
where,  $0 < p \leqslant 1$ ,  $\sigma_{j}(\overline{\mathcal{H}}^{(i)})$  denotes the  $j$ -th singular value of  $\overline{\mathcal{H}}^{(i)}$ .
It should be pointed out that for  $0 < p \leqslant 1$  when  $p$  is appropriately chosen, the Schatten  $p$ -norm provides quite effective improvements for a tighter approximation of the rank function [39; 36].
4.1 Motivation and Objective
Non-negative matrix factorization (NMF) was initially presented as a dimensionality reduction method, and it is commonly employed as an efficient latent feature learning technique recently. Generally speaking, given a non-negative matrix  $\mathbf{X}$ , the target of NMF is to decompose  $\mathbf{X}$  into two non-negative matrices,
$$
\mathbf {X} \approx \mathbf {H G} ^ {\mathrm {T}} \tag {2}
$$
where  $\mathbf{X} \in \mathbb{R}_+^{n \times p}$ ,  $\mathbf{H} \in \mathbb{R}_+^{n \times k}$  and  $\mathbf{G} \in \mathbb{R}_+^{p \times k}$ .  $\mathbb{R}_+^{n \times p}$  means  $n$ -by- $p$  matrices with elements are all nonnegative.  $n$  and  $k$  means the number of samples and the number of clusters, respectively.
In order to approximate the matrix before and after factorization,  $\ell_2$ -norm and F-norm are frequently adopted as the objective function for the NMF. Considering that F-norm can make the model optimization easier, we use F-norm to construct the objective function.
With the extensive use of NMF, more and more variants of NMF have emerged, among which are G-orthogonal NMF [8] and Semi-NMF [9]. By imposing an orthogonality constraint on one of the factor matrices in NMF, we obtain the objective function of the one-side orthogonal NMF,
$$
\min  _ {\mathbf {H} \geqslant 0, \mathbf {G} \geqslant 0} \left\| \mathbf {X} - \mathbf {H} \mathbf {G} ^ {\mathrm {T}} \right\| _ {F} ^ {2}, \quad \text {s . t .} \quad \mathbf {H} ^ {\mathrm {T}} \mathbf {H} = \mathbf {I}. \tag {3}
$$
If we relax the nonnegative constraint on one of the factor matrices in the NMF and the input matrix  $\mathbf{X}$  can also be mixed positive and negative, then we can get Semi-NMF. Semi-NMF can be adapted to process input data that has mixed symbols. For G-orthogonal NMF and Semi-NMF, Ding et al. [8] presented the following lemma:
Lemma 1.  $G$ -orthogonal NMF and Semi-NMF are all relaxation of  $K$ -means clustering, and the main advantages of  $G$ -orthogonal NMF are (1) Uniqueness of the solution; (2) Excellent clustering interpretability.
Taking into account the one-side orthogonal NMF, we relax the nonnegative constraints on  $\mathbf{X}$  and  $\mathbf{G}$ . Moreover, inspired by FMCNOF [37], we construct the anchor graph  $\mathbf{S}$  obtained from the original data  $\mathbf{X}$  as the input of matrix factorization. Compared with the original data, the number of anchors is much smaller, therefore, by adopting the anchor graph constructed by anchors and original data points as the input of matrix factorization, we can reduce the computational complexity of the algorithm effectively.
$$
\min  _ {\mathbf {H} \geqslant 0} \left\| \mathbf {S} - \mathbf {H} \mathbf {G} ^ {\mathrm {T}} \right\| _ {F} ^ {2}, \quad \text {s . t .} \quad \mathbf {H} ^ {\mathrm {T}} \mathbf {H} = \mathbf {I}, \tag {4}
$$
where  $\mathbf{S} \in \mathbb{R}^{n \times m}$ ,  $\mathbf{H} \in \mathbb{R}^{n \times k}$  and  $\mathbf{G} \in \mathbb{R}^{m \times k}$ ,  $m$  is the number of anchors and we consider  $\mathbf{H}$  as the cluster indicator matrix for clustering rows as described in [8]. We will introduce the details of anchor selection and the construction of the anchor graph in the appendix.
As described in the previous section, the existing NMF-based multi-view clustering methods are essentially a matrix factorization on a single view combined with the integration of multiple views. It causes the loss of the original spatial structure of the multi-view data. We extend NMF to the 3rd-order tensor, which can process the multi-view data directly and can also take full advantage of the original spatial structure of the multi-view data. The objective function of tensorial one-side orthogonal non-negative matrix factorization is written in the following form:
$$
\min  _ {\mathcal {H} \geqslant 0} \left\| \boldsymbol {S} - \boldsymbol {\mathcal {H}} * \boldsymbol {\mathcal {G}} ^ {\mathrm {T}} \right\| _ {F} ^ {2}, \quad \text {s . t .} \quad \boldsymbol {\mathcal {H}} ^ {\mathrm {T}} * \boldsymbol {\mathcal {H}} = \boldsymbol {\mathcal {I}}, \tag {5}
$$
The 3rd-order tensor construction process is illustrated in Fig 2.
In order to better exploit the complementary information and spatial structure between different views, we get inspiration from the excellent performance of the tensor Schatten p-norm [12; 38]. We introduce tensor Schatten p-norm regularization on the tensorial form of the cluster indicator matrix. Our objective function is formulated as follows:
$$
\min  \left\| \boldsymbol {S} - \boldsymbol {\mathcal {H}} * \boldsymbol {\mathcal {G}} ^ {\mathrm {T}} \right\| _ {F} ^ {2} + \lambda \| \boldsymbol {\mathcal {H}} \| _ {\mathscr {Q}} ^ {p} \quad s. t. \quad \boldsymbol {\mathcal {H}} \geqslant 0, \boldsymbol {\mathcal {H}} ^ {\mathrm {T}} * \boldsymbol {\mathcal {H}} = \boldsymbol {\mathcal {I}} \tag {6}
$$
where  $0 < p \leqslant 1$ ,  $\lambda$  is the hyperparameter of the Schatten  $p$ -norm term.
Remark 1. The regularizer in the proposed objective (6) is used to explore the complementary information embedded in inter-views cluster assignment matrices  $\pmb{H}^{(v)}$  ( $v = 1,2,\dots ,V$ ). Fig. 2 shows the construction of tensor  $\mathcal{H}$ , it can be seen that the  $k$ -th frontal slice  $\Delta^{(k)}$  describes the similarity between  $N$  sample points and the  $k$ -th cluster in different views. The idea cluster assignment matrix  $\pmb{H}^{(v)}$  should satisfy that the relationship between  $N$  data points and the  $k$ -th cluster is consistent in different views. Since different views usually show different cluster structures, we impose tensor Schatten  $p$ -norm minimization [12] constraint on  $\mathcal{H}$ , which can make sure each  $\Delta^{(k)}$  has spatial low-rank structure. Thus  $\Delta^{(k)}$  can well characterize the complementary information embedded in inter-views.
4.2 Optimization
Inspired by Augmented Lagrange Multiplier (ALM), we introduce two auxiliary variables  $\mathcal{Q}$  and  $\mathcal{J}$  and let  $\mathcal{H} = \mathcal{Q}$ ,  $\mathcal{H} = \mathcal{J}$ , respectively, where  $\mathcal{Q} \geqslant 0$ . Then, we rewrite the model as the following unconstrained problem:
min  $\mathcal{L}(\mathcal{Q},\mathcal{H},\mathcal{G},\mathcal{I})$
$$
= \min  _ {\mathcal {Q} \geqslant 0, \mathcal {H} ^ {\mathrm {T}} * \mathcal {H} = \mathcal {I}} \left\| \boldsymbol {S} - \boldsymbol {\mathcal {H}} * \boldsymbol {\mathcal {G}} ^ {\mathrm {T}} \right\| _ {F} ^ {2} + \lambda \| \boldsymbol {\mathcal {J}} \| _ {\mathbb {S} ^ {\mathcal {D}}} ^ {p} + \frac {\mu}{2} \left\| \boldsymbol {\mathcal {H}} - \boldsymbol {\mathcal {Q}} + \frac {\boldsymbol {\mathcal {Y}} _ {1}}{\mu} \right\| _ {F} ^ {2} + \frac {\rho}{2} \left\| \boldsymbol {\mathcal {H}} - \boldsymbol {\mathcal {J}} + \frac {\boldsymbol {\mathcal {Y}} _ {2}}{\rho} \right\| _ {F} ^ {2}, \tag {7}
$$
where  $\mathcal{V}_1, \mathcal{V}_2$  represent Lagrange multipliers and  $\mu, \rho$  are the penalty parameters. The optimization process can therefore be separated into four steps:
- Solve  $\mathcal{G}$  with fixed  $\mathcal{Q},\mathcal{H},\mathcal{J}$ . (7) becomes:
$$
\min  \left\| \boldsymbol {S} - \boldsymbol {\mathcal {H}} * \boldsymbol {\mathcal {G}} ^ {\mathrm {T}} \right\| _ {F} ^ {2} \tag {8}
$$
After being implemented with discrete Fourier transform (DFT) along the third dimension, the equivalent representation of (8) in the frequency domain becomes:
$$
\min  \sum_ {v = 1} ^ {V} \left\| \overline {{\boldsymbol {S}}} ^ {(v)} - \overline {{\boldsymbol {\mathcal {H}}}} ^ {(v)} \left(\overline {{\boldsymbol {\mathcal {G}}}} ^ {(v)}\right) ^ {\mathrm {T}} \right\| _ {F} ^ {2}, \tag {9}
$$
where  $\overline{\mathcal{G}} = \mathrm{fft}(\mathcal{G},[],3)$ , and the others in the same way.
Let  $\Phi = \left\| \overline{\pmb{S}}^{(v)} - \overline{\pmb{\mathcal{H}}}^{(v)}(\overline{\pmb{G}}^{(v)})^{\mathrm{T}}\right\|_{F}^{2}$ , we can obviously get the following equation:
$$
\Phi = \operatorname {t r} \left((\overline {{\boldsymbol {S}}} ^ {(v)}) ^ {\mathrm {T}} \overline {{\boldsymbol {S}}} ^ {(v)}\right) - 2 \operatorname {t r} \left((\overline {{\boldsymbol {\mathcal {H}}}} ^ {(v)}) ^ {\mathrm {T}} \overline {{\boldsymbol {S}}} ^ {(v)} \overline {{\boldsymbol {\mathcal {G}}}} ^ {(v)}\right) + \operatorname {t r} \left((\overline {{\boldsymbol {\mathcal {G}}}} ^ {(v)}) ^ {\mathrm {T}} \overline {{\boldsymbol {\mathcal {G}}}} ^ {(v)}\right). \tag {10}
$$
Setting the derivative  $\partial \Phi / \partial \overline{\pmb{G}}^{(v)} = 0$  gives  $2\overline{\pmb{G}}^{(v)} - 2(\overline{\pmb{S}}^{(v)})^{\mathrm{T}}\overline{\pmb{H}}^{(v)} = 0$ . So the solution of (9) is:
$$
\overline {{\boldsymbol {\mathcal {G}}}} ^ {(v)} = \left(\overline {{\boldsymbol {\mathcal {S}}}} ^ {(v)}\right) ^ {\mathrm {T}} \overline {{\boldsymbol {\mathcal {H}}}} ^ {(v)} \tag {11}
$$
- Solve  $\mathcal{H}$  with fixed  $\mathcal{Q},\mathcal{G},\mathcal{J}$ . (7) becomes:
$$
\min  _ {\mathcal {H} ^ {\mathrm {T}} * \mathcal {H} = \mathcal {I}} \left\| \boldsymbol {S} - \boldsymbol {\mathcal {H}} * \boldsymbol {\mathcal {G}} ^ {\mathrm {T}} \right\| _ {F} ^ {2} + \frac {\mu}{2} \left\| \boldsymbol {\mathcal {H}} - \boldsymbol {\mathcal {Q}} + \frac {\boldsymbol {\mathcal {Y}} _ {1}}{\mu} \right\| _ {F} ^ {2} + \frac {\rho}{2} \left\| \boldsymbol {\mathcal {H}} - \boldsymbol {\mathcal {J}} + \frac {\boldsymbol {\mathcal {Y}} _ {2}}{\rho} \right\| _ {F} ^ {2} \tag {12}
$$
And (12) is equivalent to the following in the frequency domain:
$$
\begin{array}{l} \min  _ {(\overline {{\boldsymbol {\mathcal {H}}}} ^ {(v)}) ^ {\mathrm {T}} \overline {{\boldsymbol {\mathcal {H}}}} ^ {(v)} = \mathbf {I} _ {v = 1}} \sum_ {v = 1} ^ {V} \left\| \overline {{\boldsymbol {\mathcal {S}}}} ^ {(v)} - \overline {{\boldsymbol {\mathcal {H}}}} ^ {(v)} (\overline {{\boldsymbol {\mathcal {G}}}} ^ {(v)}) ^ {\mathrm {T}} \right\| _ {F} ^ {2} \\ + \sum_ {v = 1} ^ {V} \frac {\mu}{2} \left\| \overline {{\boldsymbol {\mathcal {H}}}} ^ {(v)} - \overline {{\boldsymbol {\mathcal {Q}}}} ^ {(v)} + \frac {\overline {{\boldsymbol {\mathcal {Y}}}} _ {1} ^ {(v)}}{\mu} \right\| _ {F} ^ {2} + \sum_ {v = 1} ^ {V} \frac {\rho}{2} \left\| \overline {{\boldsymbol {\mathcal {H}}}} ^ {(v)} - \overline {{\boldsymbol {\mathcal {J}}}} ^ {(v)} + \frac {\overline {{\boldsymbol {\mathcal {Y}}}} _ {2} ^ {(v)}}{\rho} \right\| _ {F} ^ {2}, \tag {13} \\ \end{array}
$$
where  $\overline{\mathcal{H}} = \mathrm{fft}(\mathcal{H},[],3)$ , and the others in the same way.
And (13) can be reduced to:
$$
\min  _ {(\overline {{\boldsymbol {\mathcal {H}}}} ^ {(v)}) ^ {\mathrm {T}} \overline {{\boldsymbol {\mathcal {H}}}} ^ {(v)} = \mathbf {I}} - 2 \operatorname {t r} \left(\overline {{\boldsymbol {\mathcal {G}}}} ^ {(v)} (\overline {{\boldsymbol {\mathcal {H}}}} ^ {(v)}) ^ {\mathrm {T}} \overline {{\boldsymbol {\mathcal {S}}}} ^ {(v)}\right) - \mu \operatorname {t r} \left((\overline {{\boldsymbol {\mathcal {H}}}} ^ {(v)}) ^ {\mathrm {T}} \overline {{\boldsymbol {\mathcal {W}}}} _ {1} ^ {(v)}\right) - \rho \operatorname {t r} \left((\overline {{\boldsymbol {\mathcal {H}}}} ^ {(v)}) ^ {\mathrm {T}} \overline {{\boldsymbol {\mathcal {W}}}} _ {2} ^ {(v)}\right) \tag {14}
$$
where  $\overline{\mathcal{W}}_1^{(v)} = \overline{\mathcal{Q}}^{(v)} - \frac{\overline{\mathbf{y}}_1^{(v)}}{\mu}$  and  $\overline{\mathcal{W}}_2^{(v)} = \overline{\mathcal{J}}^{(v)} - \frac{\overline{\mathbf{y}}_2^{(v)}}{\rho}$ .
and it also can be reduced to:
$$
\max  _ {(\overline {{\boldsymbol {\mathcal {H}}}} ^ {(v)}) ^ {\mathrm {T}} \overline {{\boldsymbol {\mathcal {H}}}} ^ {(v)} = \mathbf {I}} \operatorname {t r} \left((\overline {{\boldsymbol {\mathcal {H}}}} ^ {(v)}) ^ {\mathrm {T}} \overline {{\boldsymbol {\mathcal {B}}}} ^ {(v)}\right) \tag {15}
$$
where  $\overline{\pmb{B}}^{(v)} = 2\overline{\pmb{S}}^{(v)}\overline{\pmb{G}}^{(v)} + \mu \overline{\mathcal{W}}_1^{(v)} + \rho \overline{\mathcal{W}}_2^{(v)}$
To solve (15), we introduce the following Theorem:
Theorem 1. Given  $\mathbf{G}$  and  $\mathbf{P}$ , where  $\mathbf{G}(\mathbf{G})^{\mathrm{T}} = \mathbf{I}$  and  $\mathbf{P}$  has the singular value decomposition  $\mathbf{P} = \boldsymbol{\Lambda}\mathbf{S}(\mathbf{V})^{\mathrm{T}}$ , then the optimal solution of
$$
\max  _ {\mathbf {G} (\mathbf {G}) ^ {\mathrm {T}} = \mathbf {I}} \operatorname {t r} (\mathbf {G P}) \tag {16}
$$
$$
i s \mathbf {G} ^ {*} = \mathbf {V} [ \mathbf {I}, \mathbf {0} ] (\boldsymbol {\Lambda}) ^ {\mathrm {T}}.
$$
Proof. From the SVD  $\mathbf{P} = \boldsymbol{\Lambda}\mathbf{S}(\mathbf{V})^{\mathrm{T}}$  and together with (16), it is evident that