Title: AugGen: Synthetic Augmentation using Diffusion Models Can Improve Recognition
Venue: NeurIPS 2025
Accepted: yes
Keywords: Generative Models

Synthetic Data

Diffusion Models

Synthetic Augmentation

Face Recognition
Abstract: The increasing reliance on large-scale datasets in machine learning poses significant privacy and ethical challenges, particularly in sensitive domains such as face recognition. Synthetic data generation offers a promising alternative; however, most existing methods depend heavily on external datasets or pre-trained models, increasing complexity and resource demands. In this paper, we introduce **AugGen**, a self-contained synthetic augmentation technique. AugGen strategically samples from a class-conditional generative model trained exclusively on the target FR dataset, eliminating the need for external resources. Evaluated across 8 FR benchmarks, including IJB-C and IJB-B, our method achieves **1–12% performance improvements**, outperforming models trained solely on real data and surpassing state-of-the-art synthetic data generation approaches, while using less real data. Notably, these gains often exceed those from architectural modifications, underscoring the value of synthetic augmentation in data-limited scenarios. Our findings demonstrate that carefully integrated synthetic data can both mitigate privacy constraints and substantially enhance discriminative performance in face recognition. Code and datasets will be made publicly available upon publication.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents AugGen, a simple yet effective framework for data augmentation using synthetic images generated from a conditional diffusion model. The method aims to enhance discriminative models, particularly in face recognition, by introducing pseudo-novel classes constructed through weighted linear combinations of one-hot class condition vectors. The generated images are filtered using a feature-space-based criterion that maximizes inter-class dissimilarity and intra-class consistency. These selected synthetic samples are then mixed with real data to form an augmented training set. The approach is validated across a wide range of face recognition benchmarks. Extensive experiments show that AugGen achieves performance improvements comparable to architectural modifications or real-data expansion, highlighting its practical value in low-resource scenarios.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n1. Strong motivation with partially inconsistent presentation\nThe motivation behind this work is compelling and well-argued: the authors aim to enhance discriminative model performance by leveraging synthetic augmentation without relying on external models or auxiliary data. This is a relevant and timely objective, particularly in privacy-sensitive domains such as face recognition. \n\n2. Simple yet practical augmentation strategy\nThe paper proposes a lightweight, reproducible strategy for synthetic data augmentation by generating pseudo-novel classes through linear interpolation of class condition vectors in a conditional diffusion model. This approach avoids the use of external models or labelers, and instead relies solely on the internal structure of the training data and the generative model itself. While the concept of interpolating between classes is not entirely new, the authors effectively adapt it into a generative augmentation pipeline that is particularly well-suited for scenarios where acquiring or annotating real data is costly or constrained by privacy concerns. The fact that the method can be implemented without complex architecture changes or training objectives makes it attractive for deployment in real-world face recognition systems and potentially other classification settings.\n\n3. Strong empirical performance with extensive evaluation\nThe experimental results are a notable strength of the paper. The authors evaluate their method across a diverse set of face recognition benchmarks. These datasets span both high- and low-quality image domains, and the consistent improvements over baseline methods demonstrate the robustness of the proposed strategy.  Furthermore, the paper includes comprehensive ablation studies that analyze the effects of mixing ratios, pseudo-class selection, and augmentation volume.\n\nWeaknesses:\n1. Limited methodological novelty\nWhile the overall system is well-designed and thoughtfully implemented, the core components—namely, class interpolation through condition vector mixing and heuristic sample selection based on feature-space metrics—are primarily composed of existing ideas. The paper focuses on a clean and pragmatic integration of known techniques, rather than introducing new loss functions, training paradigms, or theoretical formulations. As such, its contribution lies more in empirical innovation and practical value than in conceptual novelty. \n\n2. Heuristic and non-differentiable design\nThe current pipeline depends on a manual, non-learnable search over class pairs and mixing coefficients (\\alpha, \\beta), followed by a scoring mechanism based on hand-crafted similarity and dissimilarity metrics. This makes the system less flexible for integration into end-to-end differentiable learning frameworks. Moreover, such a heuristic process may become inefficient or brittle when scaling to larger class sets or tasks requiring dynamic sample generation.\n\n3. Minor presentation issues\nAlthough the paper is generally clear in its motivation and organization, there are several presentational inconsistencies that hinder readability. Some paragraphs—especially in the methodology section—are excessively long and would benefit from better structural segmentation. In addition, there are typographical issues (e.g., “with synthetic images.ss”) and formatting inconsistencies, such as the unexplained use of boldface for variables like \\mathbf{X} and \\mathbf{Z}, despite no distinction from their standard forms. While these issues do not detract from the technical content, addressing them would improve clarity and overall polish.\n\nQUESTIONS:\nPlease refer to the Weaknesses section for additional context. Below are three questions and suggestions that target specific aspects of the current design and presentation, which, if addressed, could improve the clarity and impact of the work:\n\n1. Can the heuristic search over (\\alpha, \\beta) be made learnable or end-to-end?\nThe proposed framework relies on manually enumerating mixing weights and selecting candidates based on a non-differentiable score. Have the authors considered making the weight selection process learnable, for example through a soft parameterization of the class mixing vector or meta-learning over mixing coefficients? A demonstration of a differentiable or learnable variant of the sampling mechanism could significantly enhance the method’s generality and scalability, and would strengthen the contribution.\n\t\n2. Is diffusion necessary for the framework, or could other generators be used?\nThe current method is built on conditional diffusion models, but the core idea—class mixing and sample selection—appears generator-agnostic. Could the authors comment on whether similar results would hold with GANs, VAEs, or pretrained generative models like StyleGAN? An ablation or discussion on the necessity of diffusion models versus alternative generators would clarify the scope and portability of the approach.\n\n3. Clarify presentation inconsistencies and formatting choices\nSome parts of the paper, particularly in the methodology section, suffer from overly long paragraphs and occasional typographical issues (e.g., “synthetic images.ss”). Additionally, the use of boldface for variables like \\mathbf{X} and \\mathbf{Z} is inconsistent and may confuse readers, especially since they do not appear to carry a distinct semantic meaning. Clarifying these presentation details would improve the clarity of the paper.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents AugGen, a self-contained synthetic enhancement technique. By strategically sampling from a class-conditional generative model trained exclusively on the target face recognition dataset, it generates synthetic samples without external resources, effectively boosting the model's discriminative ability. Evaluations on 8 FR benchmarks (including IJB-C and IJB-B) show that this approach achieves a 1-12% performance improvement, outperforming models trained solely on real data and state-of-the-art synthetic data generation methods. It uses less real data, and its gains often exceed those from architectural modifications, demonstrating the value of synthetic enhancement in data-constrained scenarios.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength:**\n1. The paper proposes a self-contained synthetic augmentation technique called AugGen. The mixed-class samples generated by it can significantly enhance the performance of FR benchmarks, fully demonstrating its effectiveness.\n2. The writing and structural organization of the paper are good, and the comparative experiments are comprehensive.\n\n**Weakness:**\n1. As shown in Figure 2, the paper involves multiple stages, including Generator training and data generation. The overhead brought by these processes should be considered in practical application scenarios. It is suggested to supplement information on the training time and resource consumption of different stages.\n2. State-of-the-art methods like GPT-4O and Qwen-VLo are capable of generating highly realistic face data. Could these be used to enhance the performance of FR? It would be beneficial to include some comparative analysis on this.\n3. Table 2 in the paper shows that more forged samples are needed to achieve results comparable to real samples, but there is a lack of in-depth analysis on the reasons. Is it due to the interference of mixed identities in synthetic data, or should some strategies be introduced in training to handle real and synthetic data differently?\n\nQUESTIONS:\nPlease see the Weakness part.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method that leverages an existing face recognition dataset to train a generative model, which is then used to synthesize images for training the face recognition model. The approach involves using the one-hot label of the recognition model as a conditional input to the generator, and interpolating between existing training data to create face images of intermediate identities as new data. These synthesized samples are then used to train the face recognition model, leading to improved recognition performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths :\n1.\tThe paper proposes synthesizing new identities by interpolating between two existing identities based on the available data. This approach enables the generation of novel identities, and an algorithm is introduced to evaluate whether the generated data is suitable for recognition training.\n2.\tThe method utilizes one-hot labels as input to control image generation. This relies on the training of the original recognition model (M_ori) and uses the original data to train the generative model. Additionally, the one-hot representation allows for interpolation between two identities.\n3.\tThe experimental results are promising and demonstrate the effectiveness of the proposed approach.\n\n Weaknesses:\n1.\tThe paper uses one-hot vectors as conditional input, but it is unclear how interpolated faces are generated when values between two identities (i.e., numbers not seen during training) are used. The paper does not clearly explain whether there are additional constraints during training to address this issue.\n2.\tThe experiments are conducted on a dataset with strong training data. It is unclear whether the proposed method would still be effective when the dataset contains only a single image per identity. The effectiveness of privacy protection in such cases is also not discussed. Alternatively, as a data augmentation method, experiments should be conducted on a larger dataset (such as MS1M).\n3.\tAlthough new identities are synthesized, these identities are interpolated from real identities. This approach does not demonstrate that truly new identities are generated, and thus it is unclear whether privacy is actually protected.\n4.\tWhen optimizing αα and ββ in the paper, it is only performed on a subset of the identities. The paper does not explain how these identities are selected. The search space would be larger if all identities were considered, and more identity data could potentially be synthesized.\n5.\tIt is recommended to standardize the mathematical notation throughout the paper.\n\nQUESTIONS:\n1.\tIf the cosine similarity of two human faces is small but the difference is huge, is the interpolated face still representative?\n2.\tIf we interpolate two faces that are very similar, will the resulting faces still be distinguishable?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}