Orthogonal Subspace Decomposition: A New Perspective of Learning Discriminative Features for Face Clustering

Abstract: Face  clustering  is  an  important  task,  due  to  its  wide  applications  in  practice. Graph-based  face  clustering  methods  have  recently  made  a  great  progress  and achieved new state-of-the-art results. Learning discriminative node features is the key to further improve the performance of graph-based face clustering.  To this end, we propose subspace learning as a new way to learn discriminative node features, which is implemented by a new orthogonal subspace decomposition (OSD) module. In graph-based face clustering, OSD leads to more discriminative node features,  which better reflect the relationship between each pair of faces, thereby boosting the accuracy of face clustering.  Extensive experiments show that OSD outperforms state-of-the-art results with a healthy margin.

Reviews and author responses:

**Novelty:**

We propose a new method for learning discriminative features, i.e., from the perspective of subspace learning and feature selection. 

For the purpose of learning discriminative representations, 
almost all previous methods in the area of deep learning aim to increase the inter-class distance or to decrease the intra-class distance, such as AM-softmax [4, 5] or center loss [3]. However, a very recent work [2] has claimed that the improvements brought by those methods in these years are marginal, and related research is stuck at a bottleneck. In our ablation studies, we also tried two very classical metric learning methods (see Table 2 in the revised paper), which do not work well. 
By contrast, our method has a very different insight, since it uses  subspace learning and feature selection to filter the noise or the redundant information of the learned features, making the learned features more compact and discriminative. 

Although there are a few previous works using subspace learning and feature selection to learn discriminative representations (see the second paragraph of ''Related Work''), none of them are integrated into neural networks, since they cannot be optimized via gradient-based algorithms. Our work is a first try to combine subspace learning and feature selection with neural networks and to train the whole framework in an end-to-end manner.

In addition, more generally speaking, our method also has verified the feasibility of using subspace learning and feature selection to help neural networks learn more discriminative features, which can be another future research direction. Researches can also focus on this direction rather than just increasing the inter-class distance or decreasing the intra-class distance.  

**New Experimental Results:**

We conducted some new experiments, and the new results have been added to the revised paper. Some tables are also shown in this part:

|-Batch Size-|----w/o OSD----|----w/ OSD----| \
|------16-------|--------56---------|--------81-------| \
|------64------ |-------183--------|-------226-------| \
|-----128------|-------429--------|-------486-------| \
Table 1: Average training time (ms) of the whole framework for a mini-batch with and without OSD, which is calculated over ten batches (the framework is tested on a single Tesla V100 SXM2).



|------Network-----|------------------------CIFAR10---------------------------|----------------------FashionMNIST---------------------| \
|-----------------------|---baseline---|---AM-Softmax---|-------OSD------|---baseline---|---AM-Softmax---|------OSD------| \
|----ResNet-32-----|------7.55------|--------7.47---------|-------**7.35**------|------6.80------|--------6.71---------|-------**6.43**------| \
|----ResNet-56-----|------6.98------|--------6.86---------|-------**6.67**------|------6.72------|--------6.36---------|-------**6.23**------| \
|---ResNet-110----|------6.41------|---------6.33---------|-------**6.13**------|------6.60------|--------6.49---------|------**6.16**------| \
Table 2: Top-1 error rates on CIFAR-10 and FashionMNIST. Note that the network architectures (i.e., ResNets) and training details strictly follow the experimental settings on CIFAR-10 in the original paper [1].


[1] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In Computer Vision and Pattern Recognition (2016). 

[2] K. Musgrave, B. Serge, and L. Ser-Nam. "A metric learning reality check." arXiv preprint arXiv:2003.08505 (2020).

[3] Y. Wen, K. Zhang, Z. Li, and Y. Qiao.   A discriminative feature learning approach for deep face recognition. In European conference on computer vision. (2016).

[4] H. Wang, Y. Wang, Z. Zhou, X. Ji, D. Gong, J. Zhou, Z. Li, and W. Liu.  CosFace:  Large margin cosine loss for deep face recognition. In Computer Vision and Pattern Recognition (2018).

[5] F. Wang, J. Cheng, W. Liu, and H. Liu.  Additive margin softmax for face verification.IEEE Signal Processing Letters (2018).

Many thanks for your comments. 

1,4: As discussed in the general comments to all reviewers above, one of the targets of OSD is to learn discriminative features from a totally new and different perspective. Learning disciminative representations can usually be of benefit to  different tasks 
such as face clustering, recognition, object classification, since the discriminative representations can better differentiate images from different categories. Therefore, we have added additional experiments in terms of object classification on CIFAR-10 and FashionMNIST to study the generality of the module, and in this way, the proposed method becomes a multiple orthogonal subspaces learning process. The new experiments can be seen in the general comments to all reviewers above and  Table 7 in the revised paper. 
When the neural networks were equipped with OSD, the performance of them was improved. The improvement brought by OSD is even slightly higher than the most common margin-based method (i.e., AM-Softmax), demonstrating the generality of OSD and its potential in terms of learning superior representations. 
Thus, the idea of subspace learning with feature selection is a new and valuable direction to be further explored for learning discriminative representations and improving the performance of deep models. 
As for contrastive learning, it also would be meaningful to further explore the proposed method or the idea, but this goes  far beyond the current submission. 

2: We first fixed $k_1=40$ and explored $k_2$ and $u$ in Table 1. After the best $k_2$ and $u$ were found, we explored $k_1$ (see Figure 2). Therefore, the best result in Table 2 is consistent with that in Figure 2 at $k_1=70$. It can also be found out that the result at $k_1=40$ in Figure 2 is consistent with the best result in Table 1 (please see the second paragraph of ''Ablation studies'' on page 7 for more details).

3: Thanks for your suggestion, but we still think that it is necessary to do the first ablation study. 
The first ablation study is to explore hyperparameters ($k_1$, $k_2$, and $u$), which are related to how input IPSs are constructed. The performance of GCNs can be affected by the input subgraphs, so Wang *et al.* [1] searched the hyper-parameters in order to find the best ones. Concerning that the most suitable hyperparameters for OSD-LP(GCN-M) might be different from the ones for LP(GCN-M); for a fair comparison, we should follow their search procedure to find the best hyperparameters as well. After the search procedure, we found out that our best IPS hyperparameters (i.e., $k_1=70$, $k_2=5$, and $u=5$) are very close to the best hyperparameters used for LP(GCN-M) [1] (i.e., $k_1=80$, $k_2=5$, and $u=5$), and therefore, we directly pick their results on the test set to compare and report. 
 
[1] Z. Wang, L. Zheng, Y. Li, and S. Wang.  Linkage-based face clustering via graph convolution net-work. In Computer Vision and Pattern Recognition (2019)


Many thanks for your comments. 

1. The novelty of this paper has been further summarized in the general comments to all reviewers above. Although graph-based clustering methods have been explored, there is still room for further improvement.  More importantly, although subspace learning has been explored, only a very small number of previous works combined it with feature selection for learning discriminative features (see the second paragraph of ''Related Work''), and among those works, none of them were combined with neural networks. Therefore, this is the first work to incorporate subspace learning and feature selection into neural networks for learning more discriminative representations, which is quite different from previous metric learning methods. 

1,3: The term ''neural networks'' refers to neural networks in general, which can be convolutional neural networks (CNN) or graph convolutional networks (GCN). In the paper, we pick GCNs and the face clustering task to verify the proposed OSD. We are sorry for the misleading representations, and please read the modified descriptions in the second and the third paragraph of ''Introduction'' in the revised paper. Note that we have also added new experiments that are conducted on CIFAR-10 and FashionMNIST for the classification task with convolutional neural networks (please see the results in the general comments to all reviewers or  Table 7 in the revised paper). The results show that the proposed OSD is effective when combined with different neural networks and can consistently improve the performance. 

3:  Please see Table 2 in our revised paper for new ablation studies on different combinations of the proposed terms.  

2,4: We have introduced the GCN-M in the paper, please carefully read the ''LP Module'' at the bottom of page 4 and ''Implementation Details'' in the appendix. To summarize, we followed [1] by using the same number of layers and parameters in order to directly verify the effectiveness of OSD. 

5. Note that our contribution is just OSD, and therefore, we should compare the training time of the whole framework with and without using OSD, rather than comparing the whole framework with other SOTA frameworks. Please refer to the additional results about the training time in the first part. 

[1] Z. Wang, L. Zheng, Y. Li, and S. Wang.  Linkage-based face clustering via graph convolution net-work. In Computer Vision and Pattern Recognition (2019)

Many thanks for your comments. 

1. Please refer to the additional results (Table 1) in the general comments to all reviewers above. We show the average time of a forward-backward propagation for a given mini-batch with and without OSD, which is calculated over ten batches on averages.

2. When building OSD, we directly followed the theorem and simply added those loss terms together. Since the performance has been improved, we did not further explore different combinations of the terms' weights. We are sorry that we cannot do a comprehensive grid search of those weights, since there are five terms in total and it is very hard to find the best combination. But we still use the variable-controlling approach and do a simple ablation study on the weight of these terms. Please refer to  Figure 4 and the last paragraph of ''Ablation Studies'' in the revised paper.
 
3. The number of clusters is automatically learned rather than preset. We define the test set as $S$, and the BCubed F-measure ($F$) can be written as:
\begin{equation}
F = \frac{2PR}{P+R},
\end{equation}
\begin{equation}
P = \frac{1}{|S|} \sum_{i \in S}\frac{\sum_{j\in S:C(j)=C(i)}Correct(i,j)}{\sum_{j\in S:C(j)=C(i)} 1}, \quad
R =\frac{1}{|S|} \sum_{i \in S}\frac{\sum_{j\in S:L(j)=L(i)}Correct(i,j)}{\sum_{j\in S:L(j)=L(i)} 1}, \\
\end{equation}

where $P$ and $R$ denote BCubed Precision and BCubed Recall.  $L(i)$ and $C(i)$ denote the ground-truth cluster label and the predicted cluster label of an instance $i$. $|S|$ is the number of instances in the test set. $Correct(i,j)$ is 1 if $L(i)=L(j)\ \text{and} \ C(i)=C(j)$, otherwise $Correct(i,j)$ is 0. 
According to the above equations, to calculate $F$, for each instance $i$,  one should just loop all other instances (indexed by $j$) to check whether the instance $i$ and other ones belong to the same cluster. The calculation of $F$ does not need to involve the number of clusters, since we only need to check the labels between each pair of samples. 
But $F$ can reflect the possible error of the cluster number. If the number of the predicted clusters and the ground-truth clusters are quite different, the numerator of $P$ and $R$ will not be large, and then $F$ will be low, since many instances are wrongly grouped.

As for the NMI, it can be defined as:
\begin{align}
NMI(\Omega, C) = \frac{2(H(\Omega) - H(\Omega|C))}{ H(\Omega) + H(C)},
\end{align}
where $\Omega$ and $C$ are the ground-truth cluster set and the predicted cluster set, respectively. $H(\cdot)$ represents the entropy.
To calculate $H(\Omega)$ or $H(C)$, one can just count the number of samples in each cluster, which is then divided by the total number of samples in the test set to get the probability $p$. The entropy can be calculated by summing $-p\,log\,p$ over all clusters in $\Omega$ or $C$. 
To calculate $H(\Omega|C)$, for each cluster $m$ from the set $C$, one counts the number of samples belonging to each ground-truth cluster, which are then divided by the total number of samples in the cluster $m$ to get the probability $p_n^m$, where $n$ is the index of the ground-truth cluster labels in $\Omega$. 
Then, $H(\Omega|C)$ is equal to $\frac{1}{|C|}\sum_m\sum_n-p_n^mlogp_n^m$, where $|C|$ is the number of clusters in the set $C$. 
Therefore, the calculation of NMI only needs the ground-truth cluster labels and the predicted cluster labels and does not involve the number of clusters. 
But, NMI can reflect whether the two cluster sets have the same number of clusters. For example, if the number of clusters in $C$ is 1, which is much smaller than the number of clusters in $\Omega$, then NMI will be very small. 

This paper proposes an orthogonal subspace decomposition (OSD) module, which helps the deep network learn more discriminative node features in graph-based face clustering. The experimental results indicate the superiority of OSD compared with other types of loss functions.

 - Strengths:
     1. The proposed OSD module is grounded in mathematical. 
     2. The OSD module requires a light additional computation cost, and it is easy to implement.
     3. The comparison with SOTA and the ablation study prove the effectiveness of the four components of the OSD module.
  
 - Weakness:
     1. The proposed OSD is only an auxiliary module under the LP(GCN-M) framework, which seems with insufficient value for an ICLR paper. Could this module be applied to other frameworks like contrastive learning? More experiments could be conducted to study the generality of the module.
     2. Why are the best results in Table 2 inconsistent with those in Table 1? Does it due to the different experimental settings?
     3. The first ablation study seems to have little relevancy with the proposed module, but is a study of hyper-parameters of LP(GCN-M). It could be replaced by other ablation studies which provide a deeper understanding of the OSD module.
    4. why specifically consider face clustering? is it impossible to employ your method to handle other data? e.g., imagenet, cifar100, stl, etc.
     


For graph face clustering purpose, this paper proposed subspace learning as a new way to learn discriminative graph node features, which is implemented by a new orthogonal subspace decomposition (OSD) module. OSD aims at more discriminative node features, better reflecting the relationship between each pair of face. Extensive experiments show that OSD outperforms state-of-the-art results on IJB-B and VoxCeleb2 face datasets. 

It is good to see this paper made an attempt on leveraging subspace learning as a feature selection tool into GCN. And the proposed space reconstruction (SR) loss seems good for improving the accuracy. 

I have the following comments/concerns. 

1. It is unclear how much computational cost was added due to the introduction of OSD module in training stage, as the authors have claimed no computational cost for the inference because the two subspace matrices are no longer needed. 
2. According to Equation 6 and 7, the total loss actually has 5 terms, which are equally weighted in this study. Is this reasonable? Of course, it is good to have no free hyper-parameters to tune. However, it seems no reason for equal weights of these different terms. So, it can be good to either have some theoretical proof for equal weights or add some empirical evaluations to explore better possibilities. 
3. For the evaluation, is the number of clusters preset or automatically learned by the algorithm? If it was not preset (but auto-learned), how F1 and NMI take into account the possible error in this number? If it was preset, it is good to clearly explain in the paper and discuss how the method can be extended in this direction. 


This paper proposes an orthogonal subspace decomposition (OSD) method for the face clustering task. The OSD module can learn more discriminative node features by combining subspace learning and feature selection. A space reconstruction (SR) loss and cross-entropy loss are joint as the loss function. Experiments on datasets demonstrate the good performance of the proposed method.
However, there are still several issues that should be addressed.
1. The novelty is my major concern. The graph-based clustering and subspace learning are well-explored fields. Many works have been developed. The authors should highlight the motivations and the contributions of the proposed method that are different from other methods.
2. It said that “We propose orthogonal subspace decomposition (OSD), which ...combine subspace learning and feature selection with neural networks”. 
1) What do the neural networks refer to? Is it the GCN-M network? Please give detailed information.
2) In this paper, the authors didn’t introduce the baseline GCN-M. Please give a brief description of the role of GCN-M in this method.
3) What is the effect of the proposed OSD module when replacing the benchmark GCN-M with other networks?
3. The authors propose the SR loss for the OSD module. In Table 2, we can observe the performance decline when removing any subterm of the SR loss. However, the influence of different combinations of these loss subterms is not analyzed sufficiently. In particular, how about the performance of OSD-LP(GCN-M) without L_{SR}?
4. The description of the LP module is not intuitive enough. It will be better to provide the framework of the LP module.
5. Maybe authors will present an estimation of running time compared with state-of-the-art methods.

Meta-review: During the discussion phase, although the reviewers acknowledge the effectiveness of the proposed approach, they raised the concern about the novelty of the paper.

In my opinion, I also agree that the novelty is not well justified in this paper. In the related work section, although the authors put an effort to review the existing studies of subspace learning and feature selection, their relationship (similarity and/or difference) to the proposed method is not discussed. Since the idea of using subspace learning and feature selection in clustering is standard, the novelty of this work should be introduction of the integration step into neural networks, which is not significant enough in its current state. The paper becomes more significant if, for example, theoretically discuss the unique characteristics of the integration into NNs which does not appear in the usual setting. 

In addition, the motivation of face clustering is not convincing. I recommend either (1) use and discuss the domain specific property of the problem of face clustering in the proposed method, or (2) construct a general clustering method. Since the authors present additional experiments in the author response, I guess (2) fits. Then, however, the paper should be re-organized.

The readability can be improved. For example, Algorithm 1 receives training data {X, A}, but I cannot find the definition of A. Also, please italicize mathematical symbols.

Overall, the paper is still not ready for publication, I will therefore reject the paper.


Decision: withdrawn-rejected-submissions