SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\sigma_{min}$  denotes the minimum eigenvalue of  $A$ .
Proof. It is equivalent to prove that,
$$
\operatorname {T r} \left(\left(\mathbf {A} - \sigma_ {\min } (\mathbf {A}) \mathbf {I}\right) \mathbf {B}\right) \geq 0. \tag {19}
$$
Let  $\mathbf{C},\mathbf{D}$  be matrices such that  $\mathbf{A} - \sigma_{min}(\mathbf{A})\mathbf{I} = \mathbf{C}^{\top}\mathbf{C},\mathbf{B} = \mathbf{D}^{\top}\mathbf{D}$ , then
$$
\begin{array}{l} \mathbf {T r} ((\mathbf {A} - \sigma_ {\min } (\mathbf {A}) \mathbf {I}) \mathbf {B}) = \mathbf {T r} (\mathbf {C} ^ {\intercal} \mathbf {C} \mathbf {D} ^ {\intercal} \mathbf {D}) \\ = \operatorname {T r} \left(\mathbf {C D} ^ {\top} \mathbf {D C} ^ {\top}\right) \tag {20} \\ = \operatorname {T r} \left(\left(\mathbf {D C} ^ {\intercal}\right) ^ {\intercal} \left(\mathbf {D C} ^ {\intercal}\right)\right) \geq 0. \\ \end{array}
$$
Theorem A.4. Suppose the forward of decomposed convolution layer for the  $u$ -th model is  $\mathbf{Z}_u = \alpha \mathbf{X}\mathbf{D}_u$ .  $\mathbf{Z}_u, \mathbf{Z}_v$  nearly have zero-mean since  $\mathbf{X}_p$  is preprocessed to be normalized. CCA coefficient is defined as  $S(\mathbf{Z}_u, \mathbf{Z}_v) = \sqrt{\frac{1}{c} \sum_{i=1}^{c} \sigma_i^2}$ , where  $\sigma_i^2$  denotes the  $i$ -th eigenvalue of  $\Lambda_{u,v} = Q_u^\top Q_v$ ,  $Q_u = \mathbf{Z}_u (\mathbf{Z}_u^\top \mathbf{Z}_u)^{-\frac{1}{2}}$ . Then  $S(\mathbf{Z}_u, \mathbf{Z}_v)$  is upper bounded,
$$
\mathcal {S} \left(\mathbf {Z} _ {u}, \mathbf {Z} _ {v}\right) \leq \frac {c ^ {\frac {3}{2}} \mathcal {T}}{\mathcal {C}} \cos \left(\mathbf {D} _ {u}, \mathbf {D} _ {v}\right), \tag {21}
$$
where  $\mathcal{T} = \operatorname{Tr}(\mathbf{X}^{\intercal}\pmb{\alpha}^{\intercal}\pmb{\alpha}\mathbf{X}),\mathcal{C} = \sigma_{min}(\mathbf{X}^{\intercal}\pmb{\alpha}^{\intercal}\pmb{\alpha}\mathbf{X})$
Proof. Consider  $\mathcal{S}^2 = \frac{1}{c}\sum_{i = 1}^{c}\sigma_i^2$
$$
\mathcal {S} ^ {2} = \frac {1}{c} \sum_ {i = 1} ^ {c} \sigma_ {i} ^ {2} = \frac {1}{c} \operatorname {T r} \left(\Lambda_ {u, v} \Lambda_ {u, v} ^ {\intercal}\right). \tag {22}
$$
$$
\mathbf {T r} \left(\Lambda_ {u, v} \Lambda_ {u, v} ^ {\intercal}\right) = \mathbf {T r} \left(Q _ {u} ^ {\intercal} Q _ {v} Q _ {v} ^ {\intercal} Q _ {u}\right) = \mathbf {T r} \left(Q _ {v} Q _ {v} ^ {\intercal} Q _ {u} Q _ {u} ^ {\intercal}\right). \tag {23}
$$
As defined above, we have
$$
Q _ {u} Q _ {u} ^ {\intercal} = \mathbf {Z} _ {u} \left(\mathbf {Z} _ {u} ^ {\intercal} \mathbf {Z} _ {u}\right) ^ {- \frac {1}{2}} \left(\mathbf {Z} _ {u} ^ {\intercal} \mathbf {Z} _ {u}\right) ^ {- \frac {1}{2}} \mathbf {Z} _ {u} ^ {\intercal} = \mathbf {Z} _ {u} \left(\mathbf {Z} _ {u} ^ {\intercal} \mathbf {Z} _ {u}\right) ^ {- 1} \mathbf {Z} _ {u} ^ {\intercal} \tag {24}
$$
$$
Q _ {v} Q _ {v} ^ {\intercal} = \mathbf {Z} _ {v} \left(\mathbf {Z} _ {v} ^ {\intercal} \mathbf {Z} _ {v}\right) ^ {- \frac {1}{2}} \left(\mathbf {Z} _ {v} ^ {\intercal} \mathbf {Z} _ {v}\right) ^ {- \frac {1}{2}} \mathbf {Z} _ {v} ^ {\intercal} = \mathbf {Z} _ {v} \left(\mathbf {Z} _ {v} ^ {\intercal} \mathbf {Z} _ {v}\right) ^ {- 1} \mathbf {Z} _ {v} ^ {\intercal}.
$$
Then Equation 23 becomes,
$$
\mathbf {T r} \left(\Lambda_ {u, v} \Lambda_ {u, v} ^ {\intercal}\right) = \mathbf {T r} \left(\mathbf {Z} _ {u} \left(\mathbf {Z} _ {u} ^ {\intercal} \mathbf {Z} _ {u}\right) ^ {- 1} \mathbf {Z} _ {u} ^ {\intercal} \mathbf {Z} _ {v} \left(\mathbf {Z} _ {v} ^ {\intercal} \mathbf {Z} _ {v}\right) ^ {- 1} \mathbf {Z} _ {v} ^ {\intercal}\right) \tag {25}
$$
$$
= \mathbf {T r} ((\mathbf {Z} _ {u} ^ {\intercal} \mathbf {Z} _ {u}) ^ {- 1} \mathbf {Z} _ {u} ^ {\intercal} \mathbf {Z} _ {v} (\mathbf {Z} _ {v} ^ {\intercal} \mathbf {Z} _ {v}) ^ {- 1} \mathbf {Z} _ {v} ^ {\intercal} \mathbf {Z} _ {u}).
$$
By Cauchy-Schwartz Inequality,
$$
\mathbf {T r} \left(\Lambda_ {u, v} \Lambda_ {u, v} ^ {\intercal}\right) \leq \mathbf {T r} \left(\left(\mathbf {Z} _ {u} ^ {\intercal} \mathbf {Z} _ {u}\right) ^ {- 1}\right) \mathbf {T r} \left(\left(\mathbf {Z} _ {v} ^ {\intercal} \mathbf {Z} _ {v}\right) ^ {- 1}\right) \mathbf {T r} \left(\mathbf {Z} _ {u} ^ {\intercal} \mathbf {Z} _ {v}\right) ^ {2}. \tag {26}
$$
Then we analyze these terms individually,
$$
\begin{array}{l} \begin{array}{l} \operatorname {T r} \left(\mathbf {Z} _ {u} ^ {\top} \mathbf {Z} _ {v}\right) = \operatorname {T r} \left(\mathbf {D} _ {u} ^ {\top} \mathbf {X} ^ {\top} \boldsymbol {\alpha} ^ {\top} \boldsymbol {\alpha} \mathbf {X D} _ {v}\right) = \operatorname {T r} \left(\mathbf {X} ^ {\top} \boldsymbol {\alpha} ^ {\top} \boldsymbol {\alpha} \mathbf {X D} _ {v} \mathbf {D} _ {u} ^ {\top}\right) \\ <   \operatorname {T r} (\mathbf {X I} _ {u} \cdot \operatorname {I} _ {u} \cdot \mathbf {X}) \operatorname {T r} (\mathbf {D I D} _ {u}) <   \mathcal {T} _ {u} \cdot \operatorname {T r} (\mathbf {D I D} _ {u}) \end{array} \tag {27} \\ \leq \operatorname {T r} \left(\mathbf {X} ^ {\intercal} \boldsymbol {\alpha} ^ {\intercal} \boldsymbol {\alpha} \mathbf {X}\right) \operatorname {T r} \left(\mathbf {D} _ {u} ^ {\intercal} \mathbf {D} _ {v}\right) \leq \mathcal {T} \cdot \operatorname {T r} \left(\mathbf {D} _ {u} ^ {\intercal} \mathbf {D} _ {v}\right) \\ \end{array}
$$
As for  $\mathbf{Tr}((\mathbf{Z}_u^\intercal \mathbf{Z}_u)^{-1})$ , let  $\lambda_1, \lambda_2, \dots, \lambda_c$  be eigenvalues for  $\mathbf{Z}_u^\intercal \mathbf{Z}_u$  listed in descending order ( $\lambda_1 \geq \lambda_2 \geq \dots \geq \lambda_c$ ), and assume the condition number of  $\mathbf{Z}_u^\intercal \mathbf{Z}_u$  and  $\mathbf{Z}_v^\intercal \mathbf{Z}_v$  satisfy  $\lambda_{max} / \lambda_{min} \leq \gamma$ , then,
$$
\operatorname {T r} \left(\left(\mathbf {Z} _ {u} ^ {\intercal} \mathbf {Z} _ {u}\right) ^ {- 1}\right) = \sum_ {i = 1} ^ {c} \frac {1}{\lambda_ {i}} \leq c \cdot \frac {1}{\lambda_ {c}} \leq \frac {\gamma c}{\lambda_ {1}}, \tag {28}
$$
where  $\lambda_{1} = ||\mathbf{Z}_{u}^{\mathrm{T}}\mathbf{Z}_{u}||_{2},||\cdot ||_{2}$  denotes the operator norm induced by the vector  $L_{2}$ -norm. With the norm inequalities of any positive semidefinite matrix  $A$
$$
\left| \left| A \right| \right| _ {2} \geq \frac {1}{\sqrt {c}} \left| \left| A \right| \right| _ {F} \geq \frac {1}{c} \left| \left| A \right| \right| _ {*} \geq \frac {1}{c} \mathbf {T r} (A), \tag {29}
$$
where  $||\cdot ||_F,||\cdot ||_*$  denote the Frobenius norm and the nuclear norm, respectively.
Equation (30) then becomes,
$$
\operatorname {T r} \left(\left(\mathbf {Z} _ {u} ^ {\intercal} \mathbf {Z} _ {u}\right) ^ {- 1}\right) \leq c \cdot \frac {1}{\left\| \mathbf {Z} _ {u} ^ {\intercal} \mathbf {Z} _ {u} \right\| _ {2}} \leq \frac {\gamma c ^ {2}}{\operatorname {T r} \left(\mathbf {Z} _ {u} ^ {\intercal} \mathbf {Z} _ {u}\right)}. \tag {30}
$$
By Lemma A.3,
$$
\begin{array}{l} \operatorname {T r} \left(\mathbf {Z} _ {u} ^ {\intercal} \mathbf {Z} _ {u}\right) = \operatorname {T r} \left(\mathbf {D} _ {u} ^ {\intercal} \mathbf {X} ^ {\intercal} \boldsymbol {\alpha} ^ {\intercal} \boldsymbol {\alpha} \mathbf {X D} _ {u}\right) \\ = \operatorname {T r} \left(\mathbf {X} ^ {\top} \boldsymbol {\alpha} ^ {\top} \boldsymbol {\alpha} ^ {\top} \mathbf {X D} _ {u} \mathbf {D} _ {u} ^ {\top}\right) \\ \geq \sigma_ {\min } \left(\mathbf {X} ^ {\top} \boldsymbol {\alpha} ^ {\top} \boldsymbol {\alpha} ^ {\top} \mathbf {X}\right) \operatorname {T r} \left(\mathbf {D} _ {u} ^ {\top} \mathbf {D} _ {u}\right) \tag {31} \\ \geq \mathcal {C} \cdot \operatorname {T r} \left(\mathbf {D} _ {u} ^ {\mathsf {T}} \mathbf {D} _ {u}\right) \\ \geq \mathcal {C} \cdot | | v e c (\mathbf {D} _ {u}) | | _ {2} ^ {2}, \\ \end{array}
$$
where  $vec(\cdot)$  denotes vectorization of a matrix.
Then Equation 30 is further derived as,
$$
\operatorname {T r} \left(\left(\mathbf {Z} _ {u} ^ {\mathrm {T}} \mathbf {Z} _ {u}\right) ^ {- 1}\right) \leq \frac {\gamma c ^ {2}}{\mathcal {C} \cdot \left\| v e c \left(\mathbf {D} _ {u}\right) \right\| _ {2} ^ {2}}. \tag {32}
$$
$$
\operatorname {T r} \left(\left(\mathbf {Z} _ {v} ^ {\mathrm {T}} \mathbf {Z} _ {v}\right) ^ {- 1}\right) \leq \frac {\gamma c ^ {2}}{\mathcal {C} \cdot \left\| v e c \left(\mathbf {D} _ {v}\right) \right\| _ {2} ^ {2}}. \tag {33}
$$
Finally, with  $\mathbf{Tr}(\mathbf{D}_u^\intercal \mathbf{D}_v) = <  vec(\mathbf{D}_u),vec(\mathbf{D}_v)>$  , we have
$$
\begin{array}{l} \mathbf {T r} \left(\Lambda_ {u, v} \Lambda_ {u, v} ^ {\intercal}\right) \leq \frac {\gamma^ {2} \mathcal {T} ^ {2} c ^ {4} (<   v e c (\mathbf {D} _ {u}) , v e c (\mathbf {D} _ {v}) >) ^ {2}}{\mathcal {C} ^ {2} | | v e c (\mathbf {D} _ {u}) | | _ {2} ^ {2} \cdot | | v e c (\mathbf {D} _ {v}) | | _ {2} ^ {2}} \tag {34} \\ \leq \frac {\gamma^ {2} \mathcal {T} ^ {2} c ^ {4}}{\mathcal {C} ^ {2}} \cdot \cos^ {2} (\mathbf {D} _ {u}, \mathbf {D} _ {v}), \\ \end{array}
$$
$$
\begin{array}{l} \mathcal {S} \left(\mathbf {Z} _ {u}, \mathbf {Z} _ {v}\right) = \sqrt {\frac {1}{c} \operatorname {T r} \left(\Lambda_ {u , v} \Lambda_ {u , v} ^ {\intercal}\right)} \tag {35} \\ \leq \frac {\gamma \mathcal {T} c ^ {\frac {3}{2}}}{\mathcal {C}} \cdot \cos (\mathbf {D} _ {u}, \mathbf {D} _ {v}). \\ \end{array}
$$
Then the claimed theorem is proved.
Lemma A.5. For two matrices  $\mathbf{A}$ ,  $\mathbf{B}$ , their Frobenius norm satisfies,
$$
\left\| \mathbf {A} \mathbf {B} \right\| _ {F} = \left\| \mathbf {A} \right\| _ {F} \left\| \mathbf {B} \right\| _ {F} \sqrt {1 - \frac {\Delta_ {1}}{\left\| \mathbf {A} \right\| _ {F} ^ {2} \left\| \mathbf {B} \right\| _ {F} ^ {2}}}, \tag {36}
$$
where  $\Delta_{1} = \sum_{ij}(\sum_{k}A_{ik}^{2})(\sum_{k}B_{kj}^{2})\cdot \sin^{2}(\langle A_{i:},B_{:j}\rangle)$
Proof. According to the definition of Frobenius norm  $\| \mathbf{A} \|_F = \sqrt{\sum_{ij} |A_{ij}|^2}$  we have,
$$
\left\| \mathbf {A B} \right\| _ {F} = \sqrt {\sum_ {i j} \left(\sum_ {k} A _ {i k} B _ {k j}\right) ^ {2}}. \tag {37}
$$
Note that  $(\sum_{i}x_{i}y_{i})^{2} = (\sum_{i}x_{i}^{2})(\sum_{i}y_{i}^{2})\cdot \cos^{2}(\langle x,y\rangle) = (\sum_{i}x_{i}^{2})(\sum_{i}y_{i}^{2}) - (\sum_{i}x_{i}^{2})(\sum_{i}y_{i}^{2})\cdot \sin^{2}(\langle x,y\rangle)$ , where  $\langle x,y\rangle$  is the angle of two vectors  $x$  and  $y$ . We have,
$$
\begin{array}{l} \sqrt {\sum_ {i j} (\sum_ {k} A _ {i k} B _ {k j}) ^ {2}} \\ = \sqrt {\sum_ {i j} \left[ (\sum_ {k} A _ {i k} ^ {2}) (\sum_ {k} B _ {k j} ^ {2}) - (\sum_ {k} A _ {i k} ^ {2}) (\sum_ {k} B _ {k j} ^ {2}) \cdot \sin^ {2} (\langle A _ {i :} , B _ {: j} \rangle) \right]} \\ = \sqrt {\sum_ {i k} A _ {i k} ^ {2}} \sqrt {\sum_ {k j} B _ {k j} ^ {2}} \sqrt {1 - \frac {\sum_ {i j} \left(\sum_ {k} A _ {i k} ^ {2}\right) \left(\sum_ {k} B _ {k j} ^ {2}\right) \cdot \sin^ {2} \left(\left\langle A _ {i :} , B _ {: j} \right\rangle\right)}{\sum_ {i k} A _ {i k} ^ {2} \sum_ {k j} B _ {k j} ^ {2}}} \tag {38} \\ = \| \mathbf {A} \| _ {F} \| \mathbf {B} \| _ {F} \sqrt {1 - \frac {\sum_ {i j} (\sum_ {k} A _ {i k} ^ {2}) (\sum_ {k} B _ {k j} ^ {2}) \cdot \sin^ {2} (\langle A _ {i :} , B _ {: j} \rangle)}{\| \mathbf {A} \| _ {F} ^ {2} \| \mathbf {B} \| _ {F} ^ {2}}} \\ = \| \mathbf {A} \| _ {F} \| \mathbf {B} \| _ {F} \sqrt {1 - \frac {\Delta_ {1}}{\| \mathbf {A} \| _ {F} ^ {2} \| \mathbf {B} \| _ {F} ^ {2}}}, \\ \end{array}
$$
where  $A_{i}$  is the  $i$ -th row of  $\mathbf{A}$  and  $B_{:j}$  is the  $j$ -th column of  $\mathbf{B}$ ,  $\Delta_1 = \sum_{ij} (\sum_k A_{ik}^2) (\sum_k B_{kj}^2) \cdot \sin^2 (\langle A_{i,:}, B_{:j} \rangle)$ . As  $A_{i}$  and  $B_{:j}$  are more correlated,  $\langle A_{i,:}, B_{:j} \rangle \to 0$ , thus,  $\Delta_1 \ll \| \mathbf{A} \|_F^2 \| \mathbf{B} \|_F^2$ .
Lemma A.6.
$$
\left\| \mathbf {A} ^ {1 / 2} \right\| _ {F} = \left\| \mathbf {A} \right\| _ {F} ^ {1 / 2} \left(1 + \frac {\Delta_ {1} \mathbf {A} ^ {1 / 2}}{\left\| \mathbf {A} \right\| _ {F} ^ {2}}\right) ^ {1 / 4}. \tag {39}
$$
Proof. According to Lemma A.5, we have,
$$
\left\| \mathbf {A} \right\| _ {F} ^ {2} = \left\| \mathbf {A} ^ {1 / 2} \right\| _ {F} ^ {4} - \Delta_ {1}. \tag {40}
$$
$$
\left\| \mathbf {A} ^ {1 / 2} \right\| _ {F} = \left\| \mathbf {A} \right\| _ {F} ^ {1 / 2} \left(1 + \frac {\Delta_ {1 A ^ {1 / 2}}}{\left\| \mathbf {A} \right\| _ {F} ^ {2}}\right) ^ {1 / 4}, \tag {41}
$$
where  $\Delta_{1\mathbf{A}^{1/2}} = \sum_{ij} (\sum_{k} (A^{1/2})_{ik}^2) (\sum_{k} (A^{1/2})_{kj}^2) \cdot \sin^2 \left( \langle (A^{1/2})_{i}, (A^{1/2})_{:j} \rangle \right)$ . As  $(A^{1/2})_i$  and  $(A^{1/2})_{:j}$  are more correlated,  $\langle (A^{1/2})_{i}, (A^{1/2})_{:j} \rangle \to 0$ , thus,  $\Delta_{1A^{1/2}} \ll \| \mathbf{A} \|_F^2$ .
Lemma A.7. For three matrices  $\mathbf{A}$ ,  $\mathbf{B}$ , and  $\mathbf{C}$ , their Frobenius norm satisfies,
$$
\left\| \mathbf {A} \right\| _ {F} = \left\| \mathbf {A} \right\| _ {F} \left\| \mathbf {B} \right\| _ {F} \left\| \mathbf {C} \right\| _ {F} \sqrt {1 - \frac {\Delta_ {2} + \Delta_ {3}}{\left\| \mathbf {A} \right\| _ {F} ^ {2} \left\| \mathbf {B} \right\| _ {F} ^ {2} \left\| \mathbf {C} \right\| _ {F} ^ {2}}}, \tag {42}
$$
where  $\Delta_2 = \frac{1}{2} [\| \mathbf{A}\| _F^2\sum_{kj}(\sum_lB_{kl}^2)(\sum_lC_{lj}^2)\cdot \sin^2 (\langle B_{k:},C_{:j}\rangle) + \| \mathbf{C}\| _F^2\sum_{il}(\sum_kA_{ik}^2)(\sum_kB_{kl}^2)\cdot$ $\sin^2 (\langle A_{i:},B_{:l}\rangle)]$  and  $\Delta_3 = \frac{1}{2} [\sum_{ij}(\sum_kA_{ik}^2)(\sum_k(BC)_{kj}^2)\cdot \sin^2 (\langle A_{i:},(BC)_{:j}\rangle) +$ $\sum_{ij}(\sum_l(AB)_{il}^2)(\sum_lC_{lj}^2)\cdot \sin^2 (\langle (AB)_{i:},C_{:j}\rangle)]$
Proof. Based on Lemma A.5, we have,
$$
\begin{array}{l} \| \mathbf {A B C} \| _ {F} ^ {2} \\ = \| \mathbf {A B} \| _ {F} ^ {2} \| \mathbf {C} \| _ {F} ^ {2} - \sum_ {i j} \left(\sum_ {l} (A B) _ {i l} ^ {2}\right) \left(\sum_ {l} C _ {l j} ^ {2}\right) \cdot \sin^ {2} \left(\left\langle (A B) _ {i,:}, C _ {: j} \right\rangle\right) \\ = \| \mathbf {A} \| _ {F} ^ {2} \| \mathbf {B} \| _ {F} ^ {2} \| \mathbf {C} \| _ {F} ^ {2} - \| \mathbf {C} \| _ {F} ^ {2} \sum_ {i l} \left(\sum_ {k} A _ {i k} ^ {2}\right) \left(\sum_ {k} B _ {k l} ^ {2}\right) \cdot \sin^ {2} \left(\left\langle A _ {i:}, B _ {: l} \right\rangle\right) \tag {43} \\ - \sum_ {i j} \left(\sum_ {l} (A B) _ {i l} ^ {2}\right) \left(\sum_ {l} C _ {l j} ^ {2}\right) \cdot \sin^ {2} \left(\left\langle (A B) _ {i,:}, C _ {: j} \right\rangle\right) \\ \end{array}
$$
Symmetrically, we also have,
$$
\begin{array}{l} \| \mathbf {A B C} \| _ {F} ^ {2} \\ = \| \mathbf {A} \| _ {F} ^ {2} \| \mathbf {B C} \| _ {F} ^ {2} - \sum_ {i j} (\sum_ {k} A _ {i k} ^ {2}) (\sum_ {k} (B C) _ {k j} ^ {2}) \cdot \sin^ {2} \left(\left\langle A _ {i,:}, (B C) _ {: j} \right\rangle\right) \\ = \| \mathbf {A} \| _ {F} ^ {2} \| \mathbf {B} \| _ {F} ^ {2} \| \mathbf {C} \| _ {F} ^ {2} - \| \mathbf {A} \| _ {F} ^ {2} \sum_ {k j} \left(\sum_ {l} B _ {k l} ^ {2}\right) \left(\sum_ {l} C _ {l j} ^ {2}\right) \cdot \sin^ {2} \left(\left\langle B _ {k:}, C _ {: j} \right\rangle\right) \tag {44} \\ - \sum_ {i j} (\sum_ {k} A _ {i k} ^ {2}) (\sum_ {k} (B C) _ {k j} ^ {2}) \cdot \sin^ {2} \left(\langle A _ {i,:}, (B C) _ {: j} \rangle\right) \\ \end{array}
$$
$$
\begin{array}{l} \| \mathbf {A B C} \| _ {F} ^ {2} \\ = \frac {1}{2} [ \| \mathbf {A} \| _ {F} ^ {2} \| \mathbf {B} \| _ {F} ^ {2} \| \mathbf {C} \| _ {F} ^ {2} - \| \mathbf {A} \| _ {F} ^ {2} \sum_ {k j} (\sum_ {l} B _ {k l} ^ {2}) (\sum_ {l} C _ {l j} ^ {2}) \cdot \sin^ {2} (\langle B _ {k,:}, C _ {: j} \rangle) \\ - \sum_ {i j} \left(\sum_ {k} A _ {i k} ^ {2}\right) \left(\sum_ {k} (B C) _ {k j} ^ {2}\right) \cdot \sin^ {2} \left(\left\langle A _ {i:}, (B C) _ {: j} \right\rangle\right) \tag {45} \\ + \| \mathbf {A} \| _ {F} ^ {2} \| \mathbf {B} \| _ {F} ^ {2} \| \mathbf {C} \| _ {F} ^ {2} - \| \mathbf {C} \| _ {F} ^ {2} \sum_ {i l} (\sum_ {k} A _ {i k} ^ {2}) (\sum_ {k} B _ {k l} ^ {2}) \cdot \sin^ {2} \left(\langle A _ {i:}, B _ {: l} \rangle\right) \\ \left. - \sum_ {i j} \left(\sum_ {l} (A B) _ {i l} ^ {2}\right) \left(\sum_ {l} C _ {l j} ^ {2}\right) \cdot \sin^ {2} \left(\left\langle (A B) _ {i,:}, C _ {: j} \right\rangle\right) \right] \\ = \| \mathbf {A} \| _ {F} ^ {2} \| \mathbf {B} \| _ {F} ^ {2} \| \mathbf {C} \| _ {F} ^ {2} - \Delta_ {2} - \Delta_ {3}, \\ \end{array}
$$
where  $\Delta_2 = \frac{1}{2} [ \| A \|_F^2 \sum_{kj} (\sum_l B_{kl}^2) (\sum_l C_{lj}^2) \cdot \sin^2 (\langle B_{k:}, C_{:j} \rangle) + \| C \|_F^2 \sum_{il} (\sum_k A_{ik}^2) (\sum_k B_{kl}^2) \cdot \sin^2 (\langle A_{i:}, B_{:l} \rangle)$  and  $\Delta_3 = \frac{1}{2} [\sum_{ij} (\sum_k A_{ik}^2) (\sum_k (BC)_{kj}^2) \cdot \sin^2 (\langle A_{i:}, (BC)_{:j} \rangle) + \sum_{ij} (\sum_l (AB)_{il}^2) (\sum_l C_{lj}^2) \cdot \sin^2 (\langle (AB)_{i:}, C_{:j} \rangle)]$ . Therefore,
$$
\left\| \mathbf {A} \mathbf {B} \mathbf {C} \right\| _ {F} = \left\| \mathbf {A} \right\| _ {F} \left\| \mathbf {B} \right\| _ {F} \left\| \mathbf {C} \right\| _ {F} \sqrt {1 - \frac {\Delta_ {2} + \Delta_ {3}}{\left\| \mathbf {A} \right\| _ {F} ^ {2} \left\| \mathbf {B} \right\| _ {F} ^ {2} \left\| \mathbf {C} \right\| _ {F} ^ {2}}}. \tag {46}
$$
As  $A_{i}$  and  $B_{:l}$ ,  $B_{k}$  and  $C_{:j}$  are more correlated,  $\langle A_{i,:}, B_{:l} \rangle$ ,  $\langle B_{k,:}, C_{:j} \rangle$ ,  $\langle A_{i,:}, (BC)_{:j} \rangle$ ,  $\langle (AB)_{i,:}, C_{:j} \rangle \to 0$ , thus,  $\Delta_2 \ll \| \mathbf{A} \|_F^2 \| \mathbf{B} \|_F^2 \| \mathbf{C} \|_F^2$  and  $\Delta_3 \ll \| \mathbf{A} \|_F^2 \| \mathbf{B} \|_F^2 \| \mathbf{C} \|_F^2$ .
Lemma A.8.
$$
\left\| \mathbf {A} ^ {- 1 / 2} \mathbf {B} \mathbf {C} ^ {- 1 / 2} \right\| _ {F} = \kappa_ {F} \left(\mathbf {A} ^ {1 / 2}\right) \kappa_ {F} \left(\mathbf {C} ^ {1 / 2}\right) \frac {\left\| \mathbf {B} \right\| _ {F}}{\left\| \mathbf {A} ^ {1 / 2} \right\| _ {F} \left\| \mathbf {C} ^ {1 / 2} \right\| _ {F}} \sqrt {1 - \frac {\Delta_ {2} + \Delta_ {3}}{\left\| \mathbf {A} ^ {- 1 / 2} \right\| _ {F} ^ {2} \left\| \mathbf {B} \right\| _ {F} ^ {2} \left\| \mathbf {C} ^ {- 1 / 2} \right\| _ {F} ^ {2}}}, \tag {47}
$$
where  $\kappa_F(\mathbf{A}^{1/2})$  and  $\kappa_F(\mathbf{C}^{1/2})$  are the condition number of  $\mathbf{A}^{1/2}$  and  $\mathbf{C}^{1/2}$ ,  $\kappa_F(\mathbf{A}^{1/2}) = \sqrt{(\sum \sigma_i^2(\mathbf{A}^{1/2})) (\sum \frac{1}{\sigma_i^2(\mathbf{A}^{1/2})})}$  and  $\kappa_F(\mathbf{C}^{1/2}) = \sqrt{(\sum \sigma_i^2(\mathbf{C}^{1/2})) (\sum \frac{1}{\sigma_i^2(\mathbf{C}^{1/2})})}$ ;  $\sigma_i^2(\mathbf{A}^{1/2})$  is singular value of  $\mathbf{A}^{1/2}$  and  $\sigma_i^2(\mathbf{C}^{1/2})$  are singular value of  $\mathbf{C}^{1/2}$ .
Proof. Based on Lemma A.7, we have,
$$
\left\| \mathbf {A} ^ {- 1 / 2} \mathbf {B} \mathbf {C} ^ {- 1 / 2} \right\| _ {F} = \left\| \mathbf {A} ^ {- 1 / 2} \right\| _ {F} \left\| \mathbf {B} \right\| _ {F} \left\| \mathbf {C} ^ {- 1 / 2} \right\| _ {F} \sqrt {1 - \frac {\Delta_ {2} + \Delta_ {3}}{\left\| \mathbf {A} ^ {- 1 / 2} \right\| _ {F} ^ {2} \left\| \mathbf {B} \right\| _ {F} ^ {2} \left\| \mathbf {C} ^ {- 1 / 2} \right\| _ {F} ^ {2}}}. \tag {48}
$$
By the definition of condition number  $\kappa_F(\mathbf{X}) = \| \mathbf{X}\| _F\| \mathbf{X}^{-1}\| _F = \sqrt{(\sum\sigma_i^2(\mathbf{X}))(\sum\frac{1}{\sigma_i^2(\mathbf{X})})}$ ,
$$
\left\| \mathbf {A} ^ {- 1 / 2} \mathbf {B} \mathbf {C} ^ {- 1 / 2} \right\| _ {F} = \kappa_ {F} \left(\mathbf {A} ^ {1 / 2}\right) \kappa_ {F} \left(\mathbf {C} ^ {1 / 2}\right) \frac {\left\| \mathbf {B} \right\| _ {F}}{\left\| \mathbf {A} ^ {1 / 2} \right\| _ {F} \left\| \mathbf {C} ^ {1 / 2} \right\| _ {F}} \sqrt {1 - \frac {\Delta_ {2} + \Delta_ {3}}{\left\| \mathbf {A} ^ {- 1 / 2} \right\| _ {F} ^ {2} \left\| \mathbf {B} \right\| _ {F} ^ {2} \left\| \mathbf {C} ^ {- 1 / 2} \right\| _ {F} ^ {2}}}. \tag {49}
$$
Theorem A.9. Suppose the forward of decomposed convolution layer for the  $u$ -th model is  $\mathbf{Z}_u = \alpha \mathbf{X}\mathbf{D}_u$ , CCA coefficient be  $S(\mathbf{Z}_u, \mathbf{Z}_v) = \sqrt{\frac{1}{c} \sum_{i=1}^{c} \sigma_i^2}$ , where  $\sigma_i^2$  denotes the  $i$ -th eigenvalue of  $\Lambda_{u,v} = Q_u^\top Q_v$ ,  $Q_u = \mathbf{Z}_u (\mathbf{Z}_u^\top \mathbf{Z}_u)^{-\frac{1}{2}}$ . Then  $S(\mathbf{Z}_u, \mathbf{Z}_v)$  is approximately linear to filter subspace similarity,
$$
\mathcal {S} \left(\mathbf {Z} _ {u}, \mathbf {Z} _ {v}\right) = \frac {\gamma_ {1} \gamma_ {2} \gamma_ {3}}{\sqrt {c}} \cos \left(\mathbf {D} _ {u}, \mathbf {D} _ {v}\right), \tag {50}
$$
Proof. Based on  $S(\mathbf{Z}_u, \mathbf{Z}_v) = \sqrt{\frac{1}{c} \sum_{i=1}^{c} \sigma_i^2}$  and  $\| \Lambda_{u,v} \|_F = \sqrt{\sum_{i=1}^{c} \sigma_i^2}$ , where  $\sigma_i$  are the singular value of  $\Lambda_{u,v}$ ,
$$
\mathcal {S} = \sqrt {\frac {1}{c} \sum_ {i = 1} ^ {c} \sigma_ {i} ^ {2}} = \frac {1}{\sqrt {c}} \| \Lambda_ {u, v} \| _ {F} = \frac {1}{\sqrt {c}} \| \left(\mathbf {Z} _ {u} ^ {\intercal} \mathbf {Z} _ {u}\right) ^ {- \frac {1}{2}} \mathbf {Z} _ {u} ^ {\intercal} \mathbf {Z} _ {v} \left(\mathbf {Z} _ {v} ^ {\intercal} \mathbf {Z} _ {v}\right) ^ {- \frac {1}{2}} \| _ {F}. \tag {51}
$$
According to Lemma. A.8, we have
$$
\frac {1}{\sqrt {c}} \left\| \left(\mathbf {Z} _ {u} ^ {\intercal} \mathbf {Z} _ {u}\right) ^ {- \frac {1}{2}} \mathbf {Z} _ {u} ^ {\intercal} \mathbf {Z} _ {v} \left(\mathbf {Z} _ {v} ^ {\intercal} \mathbf {Z} _ {v}\right) ^ {- \frac {1}{2}} \right\| _ {F} = \frac {\gamma_ {1} \gamma_ {2}}{\sqrt {c}} \frac {\left\| \mathbf {Z} _ {u} ^ {\intercal} \mathbf {Z} _ {v} \right\| _ {F}}{\left\| \left(\mathbf {Z} _ {u} ^ {\intercal} \mathbf {Z} _ {u}\right) ^ {\frac {1}{2}} \right\| _ {F} \left\| \left(\mathbf {Z} _ {v} ^ {\intercal} \mathbf {Z} _ {v}\right) ^ {\frac {1}{2}} \right\| _ {F}}, \tag {52}
$$
where  $\gamma_{1} = \kappa_{F}\bigl ((\mathbf{Z}_{u}^{\intercal}\mathbf{Z}_{u})^{\frac{1}{2}}\bigr)\cdot \kappa_{F}\bigl ((\mathbf{Z}_{v}^{\intercal}\mathbf{Z}_{v})^{\frac{1}{2}}\bigr)$  and  $\gamma_{2} = \sqrt{1 - \frac{\Delta_{2} + \Delta_{3}}{\|(\mathbf{Z}_{u}^{\intercal}\mathbf{Z}_{u})^{-1 / 2}\|_{F}^{2}\|\mathbf{Z}_{u}^{\intercal}\mathbf{Z}_{v}\|_{F}^{2}\|\bigl (\mathbf{Z}_{v}^{\intercal}\mathbf{Z}_{v}\bigr)^{-1 / 2}\|_{F}^{2}}}$ .
As  $\mathbf{Z}_u = \alpha \mathbf{X}\mathbf{D}_u$  and  $\mathbf{Z}_v = \alpha \mathbf{X}\mathbf{D}_v$ , we have
$$
\begin{array}{l} \frac {\gamma_ {1} \gamma_ {2}}{\sqrt {c}} \frac {\left\| \mathbf {Z} _ {u} ^ {\intercal} \mathbf {Z} _ {v} \right\| _ {F}}{\left\| \left(\mathbf {Z} _ {u} ^ {\intercal} \mathbf {Z} _ {u}\right) ^ {\frac {1}{2}} \right\| _ {F} \left\| \left(\mathbf {Z} _ {v} ^ {\intercal} \mathbf {Z} _ {v}\right) ^ {\frac {1}{2}} \right\| _ {F}} \\ = \frac {\gamma_ {1} \gamma_ {2}}{\sqrt {c}} \frac {\left\| \mathbf {D} _ {u} ^ {\intercal} \mathbf {X} ^ {\intercal} \boldsymbol {\alpha} ^ {\intercal} \boldsymbol {\alpha} \mathbf {X D} _ {v} \right\| _ {F}}{\left\| \left(\mathbf {D} _ {u} ^ {\intercal} \mathbf {X} ^ {\intercal} \boldsymbol {\alpha} ^ {\intercal} \boldsymbol {\alpha} \mathbf {X D} _ {u}\right) ^ {\frac {1}{2}} \right\| _ {F} \left\| \left(\mathbf {D} _ {v} ^ {\intercal} \mathbf {X} ^ {\intercal} \boldsymbol {\alpha} ^ {\intercal} \boldsymbol {\alpha} \mathbf {X D} _ {v}\right) ^ {\frac {1}{2}} \right\| _ {F}}. \tag {53} \\ \end{array}
$$
According to Lemma A.6,
$$
\begin{array}{l} \frac {\gamma_ {1} \gamma_ {2}}{\sqrt {c}} \frac {\| \mathbf {D} _ {u} ^ {\intercal} \mathbf {X} ^ {\intercal} \boldsymbol {\alpha} ^ {\intercal} \boldsymbol {\alpha} \mathbf {X D} _ {v} \| _ {F}}{\| (\mathbf {D} _ {u} ^ {\intercal} \mathbf {X} ^ {\intercal} \boldsymbol {\alpha} ^ {\intercal} \boldsymbol {\alpha} \mathbf {X D} _ {u}) ^ {\frac {1}{2}} \| _ {F} \| (\mathbf {D} _ {v} ^ {\intercal} \mathbf {X} ^ {\intercal} \boldsymbol {\alpha} ^ {\intercal} \boldsymbol {\alpha} \mathbf {X D} _ {v}) ^ {\frac {1}{2}} \| _ {F}} \\ = \frac {\gamma_ {1} \gamma_ {2} \gamma_ {3}}{\sqrt {c}} \frac {\left\| \mathbf {D} _ {u} ^ {\intercal} \mathbf {X} ^ {\intercal} \boldsymbol {\alpha} ^ {\intercal} \boldsymbol {\alpha} \mathbf {X D} _ {v} \right\| _ {F}}{\left\| \left(\mathbf {D} _ {u} ^ {\intercal} \mathbf {X} ^ {\intercal} \boldsymbol {\alpha} ^ {\intercal} \boldsymbol {\alpha} \mathbf {X D} _ {u}\right) \right\| _ {F} ^ {\frac {1}{2}} \left\| \left(\mathbf {D} _ {v} ^ {\intercal} \mathbf {X} ^ {\intercal} \boldsymbol {\alpha} ^ {\intercal} \boldsymbol {\alpha} \mathbf {X D} _ {v}\right) \right\| _ {F} ^ {\frac {1}{2}}}, \tag {54} \\ \end{array}
$$