SlowGuess/NeurIPS_2023_OCR huggingface dataset:
3.1 Validation Experiments
We conduct empirical validation to confirm the near-linear relationship between filter subspace similarity and probing-based similarity and explored the limitations of probing-based similarities.
Correlation of CCA and filter subspace similarity. The empirical verification of the correlation between CCA and filter subspace similarity is presented in Figure 2. In this experiment, 10 tasks are
Figure 4: Layer-wise similarity matrices that show relations of model parameters of different training time points. (a)(b)(c) are the 1st, 3rd and 5th convolutional layer of AlexNet trained on CIFAR-100. (d)(e)(f) are the 1st, 4th and 8th convolutional layer of VGG11 trained on ImageNet. We mark the epoch when the parameter reaches 0.99/0.999 similarity to its final state with white lines. For both models, we observe bottom-up learning dynamics where layers closer to the input solidify into their final states faster than very top layers, which is in accord with previous studies [37, 43].
generated from CIFAR-100 dataset [23], each consisting of 10 classes. We employ the ResNet18 model [12], training only the filter atoms while keeping the atom coefficients fixed on each task. The CCA and filter subspace similarity are calculated among 45 pairs of models. The correlation between CCA and filter subspace similarity is 0.9327, as depicted in Figure 2(b). Furthermore, the correlation between Centered Kernel Alignment (CKA) and filter subspace similarity is also reported in Table 2. These findings clearly indicate that the proposed filter subspace similarity exhibits a strong linear relationship with well-established probing-based similarities, supporting the claims made in Theorem 2.5 and Theorem 2.7.
Limitations of probing-based similarities. The consistency of probing-based similarities can vary depending on the probing data. Ideally, we anticipate a high similarity value when comparing models trained on the same dataset. To investigate this, we conduct an experiment where models are trained on the CIFAR-100 dataset. Figure 3(b) displays the distribution of model similarity with different probing data, where the y-axis represents the similarity and the x-axis represents the corresponding density of models. And with CIFAR-100 probing data, the CCA similarity between models yields a value over 0.8. However, when the probing data are derived from the other datasets including CIFAR-10, SVHN [38], CelebA [31], and etc., the CCA similarity drops to 0.59. A similar inconsistency in values is observed with the CKA similarity using different probing data. In contrast, the average of our proposed filter subspace similarity between models is 0.91, which aligns well with our expectation of high similarity. This finding demonstrates the effectiveness of our approach in capturing the inherent similarities between models trained on the same dataset, irrespective of the specific choice of probing data.
3.2 Learning Dynamics
The filter subspace similarity has various applications in analyzing NNs. It is capable of reflecting the data similarity and measuring the evolution of model similarity during the training time. We examine the training dynamics based on the heat map of filter subspace similarities. In this experiment, AlexNet [24] is trained on CIFAR-100 [23] for 150 epochs and VGG11 [50] is fine-tuned on ImageNet [47] for 20 epochs. For both models, we train and store atoms at each epoch. Figure 4 shows heat maps of similarities of the model among different training epochs.
Figure 4(a-c) are heat maps of the 1st, 3rd and 5th convolutional layers of Alexnet. We mark the epoch when the parameters of each layer reaches 0.99 similarity with the their states in the last
(a) 10th epoch
(b) 50th epoch
Figure 5: Similarity matrices that show relations among 60 users in FL with our filter subspace similarity through the training process. The labels of x-axis represent the ID's of CIFAR tasks. We can clearly see user clusters in all three figures. Specifically, the last 20 clients with SVHN data show higher similarities with themselves than the first 40 clients with CIFAR data, while every five of the first 40 clients sharing the same CIFAR task also show high similarities within themselves.
(c) 90th epoch
epoch. The first layer reaches 0.99 similarity at epoch 36 which is earlier than final layers. In Figure 4(d-f), VGG11 shows a similar behavior. Several previous works have also indicated this bottom-up learning dynamics where layers closer to the input solidify into their final states faster than very top layers [37, 43]. Our filter subspace similarity provides a highly efficient way to examine the training dynamics while showing results in accord with previous studies. Moreover, we can apply our method to calculate the similarity of a model trained on different tasks, so we can track the process of the same model interacting with different datasets. The details are shown in Appendix A.2.
3.3 Federated Learning
Federated learning (FL) aims at learning models collaboratively by leveraging the local computational power and data of all users with the concern of privacy [34]. Personalized Federated Learning (PFL) emerges to address some challenges in FL, such as poor convergence on heterogeneous data and lack of solution personalization [53].
In this setting, our framework achieves personalization by enforcing FL models with the shared atom coefficients for all users and specific filter atoms for each user. As illustrated in Figure 6, the shared coefficients preserve common knowledge, while user-specific atoms hold personalized information about each user. Then, we can assess model relationships with our filter subspace similarity without any probing data, which meets the privacy requirement of the FL scenario.
The shared atom coefficients can be achieved in different ways. With our framework, the coefficient can be obtained from a model pre-trained on a public dataset or from a global model trained by other FL approaches. We can also get the coefficients by training the model locally and evolving the coefficients at each communication round.
Measuring user similarity. With the shared atom coefficients and user-specific filter atoms, we can simply get relations of users by calculating filter subspace similarity. To be specific, we expect that users with similar data have a higher similarity. In this experiment, we distribute data of CIFAR-100 [23] and SVHN [38] to 120 clients, containing 20 SVHN clients and 100 CIFAR clients. Specifically, the SVHN dataset is randomly distributed in 20 SVHN clients. And the CIFAR-100 dataset is split into 20 subtasks with 5 classes in each subtask, and each subtask is shared by 5 CIFAR clients. The model is AlexNet [24] with 3 convolutional layers. The models share the same random initialization and filter atoms are trained independently without communication with other clients. All models are trained for  $T = 100$  communication rounds on datasets. At each round, the client executes 1 epoch of SGD with momentum to train the local model, the learning rate is 0.01 and the momentum is 0.9. The experimental details are described in Appendix A.2.
Figure 5 shows the filter subspace similarity among the last 40 clients of the CIFAR-100 task and 20 clients of the SVHN task. Specifically, a distinct cluster of the 20 SVHN clients is observed, indicating a higher similarity among these clients and dissimilarity with the CIFAR clients. Additionally, every group of 5 CIFAR clients, who share the same task, also exhibit a high similarity among themselves.
Table 1: Classification accuracy of model ensemble using different FL methods and model selection strategies: Models are selected with different similarity measures in each setting. The model ensemble using our filter subspace-based method is millions of times faster and consumes much fewer resources than probing-based methods while producing comparable performance.
FL Results	Base	+Ours	+CCA [43]	+CKA [21]
FedAvg [34]	83.78±0.08	85.82 ± 0.35	85.65 ± 0.21	85.29 ± 0.18
Ditto [27]	82.98 ± 0.13	85.49 ± 0.21	85.54 ± 0.19	85.37 ± 0.2
FedRep [8]	76.44 ± 0.06	78.35 ± 0.24	78.18 ± 0.18	77.73 ± 0.19
FedProx [29]	80.6 ± 0.1	82.95 ± 0.16	82.55 ± 0.19	82.86 ± 0.16
FedPer [2]	83.57 ± 0.07	85.21 ± 0.2	84.91 ± 0.18	84.9 ± 0.14
Pretrain	81.77 ± 0.08	85.41 ± 0.19	85.24 ± 0.13	86.33 ± 0.14
Similarity Computation Cost
GFLOPs	0.019	258,610	2,225
Time (s)	0.016	1930.4	92.6
GPU Memory (MB)	0	4915	3965
Table 2: Continual Learning Results. The model ensemble using our filter subspace similarity is significantly faster and consumes much fewer resources than probing-based methods, while maintaining comparable classification accuracy.
Method	CIFAR-100	Similarity Computation Cost
MFLOPs	Time (s)	GPU Memory (MB)
AtomCL (base)	78.11 ± 0.13	-	-	-
+CCA [43]	79.83 ± 0.04	35.2	0.26	1996
+CKA [21]	80.01 ± 0.06	111	0.3	1637
+Ours	80.19 ± 0.09	0.007	0.0008	0
This clustering capability holds great potential for facilitating efficient cluster identification in federated learning scenarios [53]. Refer to Appendix Figure 7 for the results of all 120 clients.
The computational cost of three different approaches is shown in Figure 3(a). Notably, calculating the filter subspace similarity is significantly faster (million times), requiring  $O$  GPU memory usage than probing-based methods. Note that the advantages in computational efficiency of filter subspace similarity become more prominent as the number of models increases.
Improving personalized model with ensemble of similar users. Once we get the relationships of users, we can further improve the accuracy of the current model by the ensemble of similar models, which is effective to mitigate the data heterogeneity problem in FL. The experiment is described in detail in Appendix A.2. The final results are shown in Table 1. With ensemble, the accuracies of all FL methods can be improved. Note that the results of model ensemble selected by our filter subspace similarity are comparable with probing-based methods while consuming much fewer resources.
3.4 Continual Learning
Continual learning is an open problem in machine learning in which data from multiple tasks arrive sequentially and the model is learned to adapt to new tasks while not forgetting the knowledge from the past [40]. Some of the tasks in continual learning are related, so models trained with these tasks can be benefited from aggregating knowledge from each other. We adopt the setting in [35], and apply filter subspace similarity to find related models. Specifically, we 10-Split CIFAR-100 dataset, where the 100 classes is broken down into 10 tasks with 10 classes per task. We train AlexNet including atoms and atom coefficients on the first task, and train only the atoms on the following tasks. Then, we calculate the task similarity with filter subspace similarity, and report the model ensemble result with most similar members. The accuracy and the similarity computation costs are shown in Table 2. Our method provides higher results and has faster speed compared with probing-based methods.
Model similarity. Representational similarity analysis (RSA) [22] demonstrates the method of understanding brain activities by computing similarities between brain responses in different regions. Measuring the similarity of models is beneficial for understanding neural network (NN) architectures and learning dynamics [9, 21, 37, 43]. Model similarity can be used to understand or incorporate various machine learning paradigms across different areas, including contrastive learning [13, 15], knowledge distillation [52], meta-learning [42], and transfer learning [5, 39, 44].
Multiple approaches are proposed to estimate the representational similarity of NNs. Some early works show that individual neurons can capture meaningful information [3, 4, 61, 65]. Later, gradient-based methods emerge to provide a visual explanation of deep neural networks [49]. Current popular representational similarity methods rely on features of NN. [43] proposes SVCCA to measure similarity by calculating the covariance matrix of the features of each layer after channel alignments. [21] discusses the invariance properties of similarity indices and proposes CKA with consistent correspondences between layers. probing-based similarities are data-dependent and computationally expensive. But our method measures the representational similarity only via atoms, a portion of model parameters, which is data-agnostic and much more efficient.
Learning paradigm with numerous models. Some machine learning tasks involve numerous models. For example, in Federated learning [53], thousands of models are trained across clients. In Continual learning, there are multiple models generated across time [18]. Federated learning (FL) aims to improve the performance of the system by continuously training and aggregating models from users without collecting data [20, 34, 51]. FL requires communication efficiency while thousands or even millions of clients may be involved [28]. It also required to achieve personalization [14, 53] considering data heterogeneity of different users [6, 17, 28]. Estimating user similarity can effectively address these challenges in FL. Continual learning (CL) aims at providing long-term knowledge accumulation, and the main challenge is to avoid catastrophic forgetting by learning new tasks while remembering the old ones [1, 18, 19, 26, 62]. One promising way is to store neural networks for each task [16, 30, 32, 48, 60]. As the number of tasks increases, a large number of models are generated and stored. It is important to find a way to access their relations to reuse models.
Filter atom decomposition. The research in task subspace modeling treats tasks as compositions of latent basis tasks and their linear combinations [10, 25, 33, 45, 64]. In the context of convolutional filter decomposition, DCFNet [41] introduces the filter subspace as an expansion of convolutional filters using a predetermined set of filter atoms. With the filter subspace, a group of tasks are separately modeled using neural networks, with sets of filter atoms learned for individual tasks, while a common set of atom coefficients is shared among tasks. The applications of filter subspace span among various domains, including domain adaptation [54, 57], continual learning [35], adaptive convolution [56, 59], image generation [55, 58], video comprehension [36], and graph convolution [7].
In this paper, we proposed a new paradigm for reducing representational similarity analysis in CNNs to filter subspace distance assessment, which is targeted for application scenarios where numerous models are learned. The proposed approach is targeted for application scenarios where numerous models are learned, and a computationally efficient method to assess model similarities is critical in these scenarios. We provided both theoretical and empirical evidence that the proposed filter subspace-based similarity exhibits a strong linear correlation with popular probing-based metrics while being significantly more efficient and robust in probing data. It was evaluated on both federated learning and continual learning tasks and achieves competitive performance with millions of times reduction in computational cost.
The majority of approaches in FL or CL are applied to the models with the same architecture, the proposed similarity measure with shared atom coefficient is more advantageous to be incorporated in these tasks. Our method currently assumes respective layers among compared CNNs to have coefficients with the same dimension. For our future work, we will explore the way to share atom coefficients among layers to achieve filter subspace similarity with different dimensions.
[52] Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A Alemi, and Andrew G Wilson. Does knowledge distillation really work? Advances in Neural Information Processing Systems, 2021.
[53] Alysa Ziying Tan, Han Yu, Lizhen Cui, and Qiang Yang. Toward personalized federated learning. IEEE Transactions on Neural Networks and Learning Systems, 2022.
[54] Ze Wang. Efficient Adaptation of Deep Vision Models. PhD thesis, Purdue University, 2023.
[55] Ze Wang, Xiuyuan Cheng, Guillermo Sapiro, and Qiang Qiu. Stochastic conditional generative networks with basis decomposition. In International Conference on Learning Representations, 2019.
[56] Ze Wang, Xiuyuan Cheng, Guillermo Sapiro, and Qiang Qiu. Acdc: Weight sharing in atom-coefficient decomposed convolution. arXiv preprint arXiv:2009.02386, 2020.
[57] Ze Wang, Xiuyuan Cheng, Guillermo Sapiro, and Qiang Qiu. A dictionary approach to domain-invariant learning in deep networks. Advances in neural information processing systems, 2020.
[58] Ze Wang, Seunghyun Hwang, Zichen Miao, and Qiang Qiu. Image generation using continuous filter atoms. Advances in Neural Information Processing Systems, 2021.
[59] Ze Wang, Zichen Miao, Jun Hu, and Qiang Qiu. Adaptive convolutions with per-pixel dynamic filter atom. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
[60] Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In International Conference on Learning Representations, 2018.
[61] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, 2014.
[62] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning, 2017.
[63] Xingxuan Zhang, Peng Cui, Renzhe Xu, Linjun Zhou, Yue He, and Zheyan Shen. Deep stable learning for out-of-distribution generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
[64] Yu Zhang and Qiang Yang. A survey on multi-task learning. IEEE Transactions on Knowledge and Data Engineering, 2021.
[65] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.
A.1 Theoretical Proofs
Notations of Convolutional Operations. In our paper, we express convolution operation as  $\mathbf{Z}_u = \alpha \mathbf{X}_p\mathbf{D}_u$ . More explicitly, the formulation writes as  $\mathbf{Z}_u = \alpha \mathbf{X}_p \star \mathbf{D}_u$ , where  $\star$  is the convolutional operation. By converting convolutional kernel  $\mathbf{D}_u$  into a Toeplitz matrix, we can replace the convolution operation  $\mathbf{X}_p \star \mathbf{D}_u$  with matrix multiplication  $\mathbf{X}_p\mathbf{D}_u$ . We also modify  $\alpha$  by  $I_{hw} \otimes \alpha$ , where  $\otimes$  is Kronecker product, to enable the matrix multiplication  $\alpha \mathbf{X}_p\mathbf{D}_u$ .
Proposition A.1. Suppose  $\mathbf{D}_u$  and  $\mathbf{D}_v$  are two different sets of filter atoms for a convolutional layer with the common atom coefficients  $\alpha$ , we can upper bound the changes in the corresponding features  $\mathbf{Z}_u, \mathbf{Z}_v$  with atom changes,
$$
\left\| \mathbf {Z} _ {u} - \mathbf {Z} _ {v} \right\| _ {F} \leq \left(\left\| \boldsymbol {\alpha} \right\| _ {F} \lambda\right) \sqrt {| \mathcal {B} |} \cdot \left\| \left(\mathbf {D} _ {u} - \mathbf {D} _ {v}\right) \right\| _ {F}, \quad w i t h \lambda = \sup  _ {b \in \mathcal {B}} \left\| \mathbf {X} \right\| _ {F, N _ {b}}, \tag {8}
$$
Proof. Recall the decomposed convolution can be expressed as,
$$
\mathbf {Z} = \sum_ {i = 1} ^ {m} \alpha_ {i} \left\langle \mathbf {X}, \mathbf {D} [ i ] \right\rangle_ {N _ {b}} \tag {9}
$$
$\forall b$  we have,
$$
\begin{array}{l} \left| \mathbf {Z} _ {u} (b) - \mathbf {Z} _ {v} (b) \right| = \left| \sum_ {i = 1} ^ {m} \alpha_ {i} \langle \mathbf {X}, \mathbf {D} _ {u} [ i ] \rangle_ {N _ {b}} - \sum_ {i = 1} ^ {m} \alpha_ {i} \langle \mathbf {X}, \mathbf {D} _ {v} [ i ] \rangle_ {N _ {b}} \right| \tag {10} \\ \leq | | \boldsymbol {\alpha} | | _ {F} \left(\sum_ {i = 1} ^ {m} \left| \left\langle \mathbf {X}, \left(\mathbf {D} _ {u} [ i ] - \mathbf {D} _ {v} [ i ]\right) \right\rangle_ {N _ {b}} \right| ^ {2}\right) ^ {1 / 2}. \\ \end{array}
$$
By Cauchy-Schwarz inequality,
$$
\begin{array}{l} \left| \langle \mathbf {X}, \left(\mathbf {D} _ {u} [ i ] - \mathbf {D} _ {v} [ i ]\right) \rangle_ {N _ {b}} \right| \leq \| \mathbf {X} \| _ {F, N _ {b}} \cdot \| \mathbf {D} _ {u} [ i ] - \mathbf {D} _ {v} [ i ] \| _ {F, N _ {b}} \tag {11} \\ \leq \lambda \cdot | | \mathbf {D} _ {u} [ i ] - \mathbf {D} _ {v} [ i ] | | _ {F, N _ {b}} \\ \end{array}
$$
$$
\begin{array}{l} \sum_ {b \in \mathcal {B}} | \mathbf {Z} _ {u} (b) - \mathbf {Z} _ {v} (b) | ^ {2} \leq | | \boldsymbol {\alpha} | | _ {F} ^ {2} \sum_ {b} \sum_ {i = 1} ^ {m} | \langle \mathbf {X}, (\mathbf {D} _ {u} [ i ] - \mathbf {D} _ {v} [ i ]) \rangle_ {N _ {b}} | ^ {2} \\ \leq \left\| \boldsymbol {\alpha} \right\| _ {F} ^ {2} \sum_ {b} \sum_ {i = 1} ^ {m} \left\| \mathbf {X} \right\| _ {F, N _ {b}} ^ {2} \cdot \left\| \left(\mathbf {D} _ {u} [ i ] - \mathbf {D} _ {v} [ i ]\right) \right\| _ {F, N _ {b}} ^ {2} \tag {12} \\ \leq \left(\left| \left| \boldsymbol {\alpha} \right| \right| _ {F} \lambda\right) ^ {2} \sum_ {b, i} \left| \left| \left(\mathbf {D} _ {u} [ i ] - \mathbf {D} _ {v} [ i ]\right) \right| \right| _ {F, N _ {b}} ^ {2} \\ \end{array}
$$
and observe that
$$
\sum_ {b, i} \left| \left| \left(\mathbf {D} _ {u} [ i ] - \mathbf {D} _ {v} [ i ]\right) \right| \right| _ {F, N _ {b}} ^ {2} = \sum_ {b \in \mathcal {B}} \sum_ {i = 1} ^ {m} \left| \left| \left(\mathbf {D} _ {u} [ i ] - \mathbf {D} _ {v} [ i ]\right) \right| \right| _ {F, N _ {b}} ^ {2} = | \mathcal {B} | \cdot \left| \left| \left(\mathbf {D} _ {u} - \mathbf {D} _ {v}\right) \right| \right| _ {F} ^ {2}, \tag {13}
$$
where  $|\mathcal{B}|$  is the area of the domain of  $\mathbf{X}$ . Then Eq. 12 becomes
$$
\sum_ {b \in \mathcal {B}} \left| \mathbf {Z} _ {u} (b) - \mathbf {Z} _ {v} (b) \right| ^ {2} \leq \left(\left\| \boldsymbol {\alpha} \right\| _ {F} \lambda\right) ^ {2} | \mathcal {B} | \cdot \left\| \left(\mathbf {D} _ {u} - \mathbf {D} _ {v}\right) \right\| _ {F} ^ {2}, \tag {14}
$$
which proves that  $||\mathbf{Z}_u - \mathbf{Z}_v||_F \leq (||\boldsymbol{\alpha}||_F \lambda) \sqrt{|\mathcal{B}|} \cdot ||(\mathbf{D}_u - \mathbf{D}_v)||_F$  as claimed.
Proposition A.2. Assume filter atoms  $\mathbf{D}_u, \mathbf{D}_v$  are orthogonal matrices, then  $S_{\text{Gras}} = S_{\text{Atom}}$ .
Proof. Since  $\mathbf{D}_u, \mathbf{D}_v \in \mathbb{R}^{k^2 \times m}$  are orthogonal matrices, i.e.,  $\mathbf{D}_u^T \mathbf{D}_u = \mathbf{D}_v^T \mathbf{D}_v = I$ , the Grassmann similarity can be represented as,
$$
\mathcal {S} _ {\text {G r a s}} \left(\mathcal {F} _ {u}, \mathcal {F} _ {v}\right) = \frac {1}{m} \sum_ {i} ^ {m} \cos \theta_ {i} = \frac {1}{m} \sum_ {i} ^ {m} \sigma_ {i}, \tag {15}
$$
where  $\sigma_{i} = \Sigma_{ii}, U\Sigma V = \mathbf{D}_{u}^{T}\mathbf{D}_{v}$ .
$\mathcal{S}_{Atom}$  is defined as,
$$
\mathcal {S} _ {\text {A t o m}} \left(\mathcal {F} _ {u}, \mathcal {F} _ {v}\right) = \cos \left(\mathbf {D} _ {u}, \mathbf {D} _ {v}\right) = \frac {<   v e c \left(\mathbf {D} _ {u}\right) , v e c \left(\mathbf {D} _ {v}\right) >}{\left| \left| v e c \left(\mathbf {D} _ {u}\right) \right| \right| _ {F} \cdot \left| \left| v e c \left(\mathbf {D} _ {v}\right) \right| \right| _ {F}}. \tag {16}
$$
Analyze each part separately, we have  $< \operatorname{vec}(\mathbf{D}_u), \operatorname{vec}(\mathbf{D}_v) > = \operatorname{Tr}(\mathbf{D}_u^T \mathbf{D}_v) = \sum_i^m \sigma_i$ ,  $||\operatorname{vec}(\mathbf{D}_u)||_F = \sqrt{\operatorname{Tr}(\mathbf{D}_u^T \mathbf{D}_u)} = \sqrt{\operatorname{Tr}(I)} = \sqrt{m}$ , and also  $||\operatorname{vec}(\mathbf{D}_v)||_F = \sqrt{m}$ . In total, the filter subspace similarity becomes,
$$
\mathcal {S} _ {\text {A t o m}} \left(\mathcal {F} _ {u}, \mathcal {F} _ {v}\right) = \cos \left(\mathbf {D} _ {u}, \mathbf {D} _ {v}\right) = \frac {\sum_ {i} ^ {m} \sigma_ {i}}{m}, \tag {17}
$$
which equals  $S_{\text{Gras}}$ . The claimed theorem is proved.
Lemma A.3. For two positive semidefinite matrices  $\mathbf{A},\mathbf{B}$
$$
\operatorname {T r} (\mathbf {A B}) \geq \sigma_ {\min } (\mathbf {A}) \operatorname {T r} (\mathbf {B}), \tag {18}
$$