Learning Unified Representations for Multi-Resolution Face Recognition
Authors: Hulingxiao He
Wu Yuan
Yidian Huang
Shilong Zhao
Wen Yuan
HanQing Li
Venue: Submitted to ICLR 2023 (2023)
Keywords: multi-resolution face recognition
deep representation learning

Abstract: In this work, we propose Branch-to-Trunk network (BTNet), a novel representation learning method for multi-resolution face recognition. It consists of a trunk network (TNet), namely a unified encoder, and multiple branch networks (BNets), namely resolution adapters. As per the input, a resolution-specific BNet is used and the output are implanted as feature maps in the feature pyramid of TNet, at a layer with the same resolution. The discriminability of tiny faces is significantly improved, as the interpolation error introduced by rescaling, especially up-sampling, is mitigated on the inputs. With branch distillation and backward-compatible training, BTNet transfers discriminative high-resolution information to multiple branches while guaranteeing representation compatibility. Our experiments demonstrate strong performance on face recognition benchmarks, both for multi-resolution face verification and face identification, with much less computation amount and parameter storage. We establish new state-of-the-art on the challenging QMUL-SurvFace 1: N face identification task.


Reviews:
[{"review_id": "rKr0EoFGpR", "reviewer": "Reviewer_yRHk", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a Branch-to-Trunk network (BTNet) for multi-resolution face recognition, which consists of a trunk network (TNet) and multiple branch networks (BNets). With branch distillation and backward compatible training, BTNet transfers discriminative high-resolution information to multiple branches. Experiments on face recognition benchmarks show the better performance of the proposed BTNet for multi-resolution face verification and face identification.", "summary_of_the_review": "The main concerns of the work are its weaknesses as\n- Waht are contributions of the proposed BTNet compared to existing dynamic resolution network (Zhu et al., NeurIPS 2021)? And the paper should compare with dynamic resolution network.\n- The paper evaluate the proposed BTNet on the challenging QMUL-SurvFace dataset, and it is recommend to conduct comparisons on more fae recognition benchmarks, e.g., WebFace260M [1].", "strength_and_weaknesses": "Strength:\n+ The paper is well written and easy to read.\n+ The proposed BTNet obtains the state-of-the-art performance on the challenging QMUL-SurvFace 1: N face identification task.\n\nWeaknesses:\n- Waht are contributions of the proposed BTNet compared to existing dynamic resolution network (Zhu et al., NeurIPS 2021)? And the paper should compare with dynamic resolution network.\n- The paper evaluate the proposed BTNet on the challenging QMUL-SurvFace dataset, and it is recommend to conduct comparisons on more fae recognition benchmarks, e.g., WebFace260M [1].\n[1] Zhu et al., WebFace260M: A Benchmark Unveiling the Power of Million-Scale Deep Face Recognition, CVPR 2021.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "mheZL_WeER", "reviewer": "Reviewer_Lmwt", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a Branch-to-Trunk network for multi-resolution face recognition. In this paper, by setting multiple branches and inputting images with different resolutions into different branches, the interpolation error caused by up-sampling is reduced. It requires less computation amount and parameter storage. The paper conducts experiments on six face verification benchmarks and QMUL-SurvFace dataset.", "summary_of_the_review": "See Strength And Weaknesses.", "strength_and_weaknesses": "Strength:\nMost low-resolution face recognition works require up-sampling low-resolution images to a fixed size (112*112). This paper proposes a multi-branch network to reduce the interpolation error by piecewise processing the input image according to the resolution, which is a good idea. Meanwhile, this method only stores the learned branches and resolution-aware BNs, which requires less computation amount and parameter storage.\n\nWeaknesses:\n1. For the training set MS1Mv3, we need to first down-sample the high-resolution image to the low-resolution. Although this paper avoids up-sampling by branching strategy, down-sampling will still introduce interpolation errors and there is no solution for this question. \n\n2. For the test on real low-resolution face images, the model does not know the specific resolution to select the branch. If the face resolution needs to be judged according to the image size, many images of QMUL-SurvFace are not full face (after face alignment), and the image collected in real scene may have background and need face detection. But low-resolution face detection is difficult to achieve. At this time, there is no judgment strategy for face resolution, which make it difficult to select branches.\n\n3. This paper inputs images with different resolutions by adding branch headers in front of the backbone network, which is not novel enough. \n\n4. The paper lacks experiments compared to the SOTA methods, such as 1:N face identification on SCFace dataset, 1:1 face verification on QMUL-SurvFace dataset and 1:N face identification on QMUL-TinyFace dataset.\n\n5. In Table 2, the paper does not provide results for each of the six datasets. Tables 4 and 5 do not indicate which dataset the experiment was performed on. In figure 8, further allocation (floor/near/ceil) lacks a specific explanation.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "WBCCS7RX1f", "reviewer": "Reviewer_FLmy", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the author designed a new model,named Branch-to-Trunk network (BTNet), to match faces with different resolutions, and the experimental result demonstrates that the method is feasible. A possible potential contribution of this paper is the design of the network structure, mainly the introduction of multiple branches.", "summary_of_the_review": "  The quality of this article is average and not recommended for acceptance.", "strength_and_weaknesses": "Strength: The design of the model structure, but the pyramid structure is already very common in other work and cannot be considered as a very innovative point.\n\nWeakness: \nThere are several points in the article that confused me, as follows:\n1. In Table 4, the number of parameters of Pretraining + BCT is 43.59, while the number of parameters of Pretraining + BCT + Fix Trunk is 2.29. It seems unreasonable.\n2. As can be seen from Table 4, it is the model distillation operation that provides the largest performance improvement across multi-resolution and same resolution task. What puzzles me is: 1) whether distillation is capable of such a significant improvement and 2) if the distillation operation can improve to such an extent, then the innovation points claimed in this paper may not be reliable and more gains come from the distillation operation.\n3. The task of this paper is to obtain a better cross-resolution representation, and I think it may be more intuitive to visualize it using t-SNE.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Vy-LmduoS1G", "reviewer": "Reviewer_JeWQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a multi-resolution face recognition algorithm. It is based on a CNN encoder backbone (e.g. ResNet50), denoted trunk Net, and multiple lateral resolution-specific nets, denoted BTNets. Each BTNet receives as input an image with a specific resolution producing a representation of the same resolution. This representation is combined with the backbone at the layer with a resolution matching that of the BTNet. The experimentation shows how different components of the model contribute to the final solution. It compares favorably with the state of the art on a low resolution data set.", "summary_of_the_review": "The paper addresses a relevant problem in face recognition. However, the presentation of the approach is difficult to follow and is not complete. The experimentation is weak. It compares with the state of the art in only one data set, failing to evaluate its multi-resolution performance against its competitors.\n", "strength_and_weaknesses": "Strengths.\n\nThe paper addresses a relevant problem in the literature, namely, multi-resolution face recognition.\n\nWeaknesses. \n\nThe experimentation needs to be improved. It only compares with the state-of-the-art in a low-resolution data set. If it it claims to learn a multi-resolution representation, it should be compared with other competing algorithms and on many more data sets with different resolutions/characteristics.\n\nThe writing should also be improved. The paper is difficult to understand. Important information is missing, e.g. how is the representation produced by BTNet combined with the backbone. In section 4.1 it reads that the  experimentation has been performed with 6 data sets that go missing in the paper. Some tables at the end of the paper were not cited in the text.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject