Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper aims to improve face recognition performance in case each probe has multiple images by proposing Cluster and Aggregate (CAFace), a two-stage technique to better aggregate extracted probe features. In the Cluster stage, the model learns to softly assign N inputs into M global clusters, with the number of clusters M predefined. It first passes the extracted features through a Style Input Maker (SIM) to extract "style" features, then passes the "style" features through a Cluster Network (CN) to softly assign them to clusters based on learnable cluster centers. In the Aggregation stage, the model computes M clustered features, then fuses them using an Aggregation Network (AGN) to get the final output. CAFace divides inputs into T-step sequential inference with incremental averaging and guarantees set order invariance by employing a Set Permutation Consistency loss. CAFace provides the best performance boost compared with other feature fusion techniques on IJB-A, IJB-B, IJB-C, and IJB-S benchmarks.
strengths_and_weaknesses: ### Strengths
- As shown in Fig. 1b, CAFace avoid the weaknesses of previous methods. It could handle a large number of images per probe (N) while considering Intra-set relation. It is batch order invariant, and guarantees set order invariance by employing a Set Permutation Consistency loss.
- CAFace provides the best performance boost compared with other feature fusion techniques on IJB-A, IJB-B, IJB-C, and IJB-S benchmarks.
- CAFace seems to group low-quality images into a cluster, which has the fusion weith near zero.

### Weaknesses
- L18-19: This definition is only valid for Face Identification, one of two subproblems in Face Recognition. In Face verification, we do not have Gallery or Probe.
- It is unclear to me when the cluster centers are trained. Are they trained along with the fusion network and be universal to all probes? Or are they optimized to fit each probe? In the case of universal cluster centers, some analyses of facial images near the cluster centers are recommended to add.
- The proposed method seems over-complicated. The authors should add more ablation studies to show the need for the proposed network components. For example, what if we do not use the style embeddings but learn the cluster centers for identity features f_i themselves? What if we do not use the Aggregation Network and simply average the clustered features in F'?
- In Section 3.2, the definition of f^{(p)}\_{GT} is confusing. Is not all training data labeled? What is the difference between f^{(p)} and f^{(p)}_{GT}?
- The authors should improve the description for Fig. 6:
  - What are C0, C1, and C2? Are they cluster centers? C3 is not visualized?
  - In CAFace plot, there are many blue points with high cosine similarity to the Gallery. The authors should visualize and analyze the differences between the images corresponding to these points and the red/yellow points that make them have different weights.
- Which network layer was used to extract the intermediate feature maps M_i in the experiments, and what is the feature map size?
- In Table 1, an empty cell can be either in a merged cell or means "No". It may cause confusion. The authors should improve this, e.g., use 'x' to denote 'No' instead of using an empty cell.
- The norm embedding definition should be moved to the main text.
questions: - It is unclear to me when the cluster centers are trained. Are they trained along with the fusion network and be universal to all probes? Or are they optimized to fit each probe? In the case of universal cluster centers, some analyses of facial images near the cluster centers are recommended to add.
- The proposed method seems over-complicated. The authors should add more ablation studies to show the need for the proposed network components. For example, what if we do not use the style embeddings but learn the cluster centers for identity features f_i themselves? What if we do not use the Aggregation Network and simply average the clustered features in F'?
- In Section 3.2, the definition of f^{(p)}\_{GT} is confusing. Is not all training data labeled? What is the difference between f^{(p)} and f^{(p)}_{GT}?
- The authors should improve the description for Fig. 6:
  - What are C0, C1, and C2? Are they cluster centers? C3 is not visualized?
  - In CAFace plot, there are many blue points with high cosine similarity to the Gallery. The authors should visualize and analyze the differences between the images corresponding to these points and the red/yellow points that make them have different weights.
- Which network layer was used to extract the intermediate feature maps M_i in the experiments, and what is the feature map size?
limitations: The paper mentioned Limitations and Potential Negative Societal Impacts.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposed a two-stage feature fusion paradigm, Cluster and Aggregate, that can both scale to large N and maintain the ability to perform sequential inference with order invariance. 

The clustering stage is a linear assignment of N inputs to M global cluster centres, and the aggregation stage is a fusion over M clustered features. The clustered features play an integral role when the inputs are sequential as they can serve as a summarization of past features. 

By leveraging the order-invariance of incremental averaging operation, this paper designed an update rule that achieves batch-order invariance, which guarantees that the contributions of the early image in the sequence do not diminish as time steps increase.
strengths_and_weaknesses: Strengths:

(1) Feature fusion is a practical research topic for set-based face recognition.

(2) Experiments on IJB-B and IJB-S benchmark datasets show the superiority of the proposed two-stage paradigm in unconstrained face recognition.

 Weaknesses:

(1) IJB-C is mentioned in this paper but only the performance on IJB-B is reported.
(Line 227: IJB-C is an updated version of IJB-B with more complex motions in the video)

(2)  Efficiency comparison is given in the supp, it is better to include it in the main paper as 'Intra-set calculation becomes infeasible when N is large or sequential for intra-set methods' is one of the main motivations of the proposed method.

(3) The idea of clustering and aggregation is clear. However, the motivations to implement in this way (Fig 3) are hard to follow.

(4) Typos in abstract "diminish"; M is the cluster number but also used in line 138 C_M.
questions: Please reply to my questions raised in the weaknesses:

(1) Why IJB-C is not tested and reported?

(2) Efficiency comparison is suggested to move to the main paper.

(3) There are too many implementation details in Fig 3 and the motivations behind these designs are hard to follow.

(4) Typo needs to be fixed and overlapped subscript needs to be changed.
limitations: The limitations and potential negative societal impact is discussed at the end of this paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper, the authors introduce a feature fusion framework, namely Cluster and Aggregate (CAFace), for face recognition with a large probe set.
CAFace consists of two main modules: Cluster Network (CN) and Aggregation Network (AGN). 
While the first module aims at grouping images into M clusters (based on their quality) and fusing the features within a cluster (i.e. based on their correlation to the centers), the second modules aggregates features of all M clusters (i.e. different quality) into a single feature to represent the ID.
The proposed approach is validated on IJB-B, IJB-C, and IJB-S.
strengths_and_weaknesses: STRENGTHS:
- The paper is well-motivated.
- The idea of two-step feature fusion (I.e. from N features to M intermediate cluster features and aggregate these intermediate features for a single fused output) is interesting. 
While the first step aims at fusing features within a cluster (i.e. similar image quality), the second step aggregate features of all clusters (i.e. different quality).
- Experimental results show improvements against prior works.

Weaknesses
Although the paper is well-motivated and experiments seem to be completed with ablation study, there are some concerns regarding the design and efficiency of the proposed approach. Please see the Questions section.


questions: 1. For the cluster network (line 153-155), Multihead self-attention is adopted with the attention matrix of (N + M) x (N + M). As N could be very large, how efficiency is this design? 

2. Does the correlation of samples within a batch affect the assignment of a sample to one of M centers? If yes, the set element permutation may significantly affect the accuracy. Moreover, How is the contribution of the intra-set relationship in Aggregation Network?
In experiment of Table 3, CAFace seems to be robust to the order. How are the probe images shuffled in this experiment? In other words, does the set elements permutation differ significantly among runs?

3. When M = 1, the accuracy partially indicates the contribution of Clustering Network and its attention as most of the fusion is taken place in this stage. However, as shown in Table 1, its accuracy is similar to Naive Average Fusion.
How effective is the attention learned by Clustering Network?

4. For prior works such as PFE, CFAN, are they retrained with the same training set as CAFace? 

5. The experiment would be more complete with the analysis of the effect of N (i.e. from small to very large number of sample per ID to show the advantage of CAFace in comparison to prior works.
limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a method for better aggregation of embeddings used to describe faces in a face recognition problem. For aggregation a pair of networks are used, named Cluster Network and Aggregation Network. The method is trained on WebFace4M and tested on IJB-B, IJB-C and IJB-S. 
strengths_and_weaknesses: Strengths:
 - the idea to use a set of networks that approach different task for aggregation of embeddings is interesting
- results better than SOTA; ablation inclusion

Weaknesses:
- the method is dedicated to FR problem, which is important but rather limited in auditorium in NIPS context
 - while the idea to use two different nets is appealing, the approach for each one of them is know. Clustering may be tracked to Center-Loss. ref 31 and 52 brought the idea of aggregation.
 - the method is very hard to follow. Overall the figures complement rather well the description, but the later rather un-inspiring. For instance the second part of eq (1) says nothing that was not explainable in plain text. eq(2-3-4) are combination of mathematical equations and acronyms and so on. Overall too many non-intuitive notations are gathered in a very condensed text so to easy to read.    

Overall I see the paper as being potentially acceptable, but it needs a through revision of the method presentation to improve its readability.

Post rebuttal: My questions and other raised questions have been answered, therefore I still view this paper as acceptable 
questions: What does it mean:
- "The magnitude of P is an interpretable quantity showing the importance of each cluster during fusion" 
- "sequential inference with incremental averaging" l180
- "sample orders" from "invariant to sample orders" l184
- "invariance to the order of batch" l190
- "identity," l197 "person identity" or the mathematical function of identity?
limitations: The paper addresses the face recognition theme, which is one of the topics in ML/CV that raised most ethical issues. It's main usage refers to retrieving the identity of a person from a low quality (potential surveillance camera)
 
The authors included a short discussion on potential negative societal impacts. Also, the paper clearly states that it avoided datasets that were withheld "by their distributors due to privacy and other issues".

Yet the later aspect is debatable: the paper did not used VGGFace-2 because it was upheld by VGG authors as being under investigation by their university legal department, but used WebFace4M dataset collected in the same conditions as VGG, just larger, but still publicly available.

However, again, I must emphasize that this paper does not introduce any new dataset, and therefore has not broke any confidentiality.
review_title: null

Meta-review: metareview: The paper received 4 positive reviewers, and the reviewer increased/remained their scores after the rebuttal. 

The paper pursues a useful direction of unconstrained face recognition. All reviewers agree that the results are impressive and the method has novelty.
confidence: Certain
Decision: Accept