RectifID: Personalizing Rectified Flow with Anchored Classifier Guidance

Authors: zhicheng  sun
zhenhao  yang
yang  jin
haozhe  chi
kun  xu
liwei  chen
hao  jiang
yang  song
kun  gai
yadong  mu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 15:30:37

Abstract: Customizing diffusion models to generate identity-preserving images from user-provided reference images is an intriguing new problem. The prevalent approaches typically require training on extensive domain-specific images to achieve identity preservation, which lacks flexibility across different use cases. To address this issue, we exploit classifier guidance, a training-free technique that steers diffusion models using an existing classifier, for personalized image generation. Our study shows that based on a recent rectified flow framework, the major limitation of vanilla classifier guidance in requiring a special classifier can be resolved with a simple fixed-point solution, allowing flexible personalization with off-the-shelf image discriminators. Moreover, its solving procedure proves to be stable when anchored to a reference flow trajectory, with a convergence guarantee. The derived method is implemented on rectified flow with different off-the-shelf image discriminators, delivering advantageous personalization results for human faces, live subjects, and certain objects. Code is available at https://github.com/feifeiobama/RectifID.

Main paper: 
 introduction: 
 Recent advances in diffusion models (Sohl-Dickstein et al., 2015; Song and Ermon, 2019; Ho et al., 2020) have ignited a surge of research into their customizability. A prominent example is personalized image generation, which aims to integrate user-defined subjects into the generated image. This plays a pivotal role in AI art creation, empowering users to produce identity-consistent images with greater customizability beyond text prompts. Nevertheless, there remain significant challenges in accurately preserving the subject's identity and being flexible to a variety of personalization needs.Existing personalization methods are limited in these two aspects, as they require an extra finetuning or pre-training stage. For example, the pioneering works (Gal et al., 2023; Ruiz et al., 2023a) finetune conditional embeddings or model parameters per subject, resulting in suboptimal efficiency and identity consistency due to lack of domain knowledge. On the other hand, the recently prevailing tuning-free methods (Wei et al., 2023; Ye et al., 2023; Li et al., 2024; Wang et al., 2024b) pre-train a conditioning adapter to encode subject features into the generation process. However, their models must be pre-trained on extensive domain-specific data, e.g. LAION-Face 50M (Zheng et al., 2022) , which is costly in the first place, and cannot be transferred flexibly across different data domains, e.g. from human faces to common live subjects and objects, and even to multiple subjects.To address both challenges of identity consistency and flexibility, we advocate a training-free approach that utilizes the guidance of a pre-trained discriminator without extra training of the generative model. This methodology is well-known as classifier guidance (Dhariwal and Nichol, 2021) , which modifies an existing denoising process using the gradient from a pre-trained classifier. The rationale behind our exploitation is twofold: first, it directly harnesses the discriminator's domain knowledge for identity preservation, which may be a cost-effective substitute for training on domain-specific datasets; secondly, keeping the diffusion model intact allows for plug-and-play combination with different discriminators, as shown in Fig. 1 , which enhances its flexibility across various personalization tasks. However, the original classifier guidance is largely limited in the reliance on a special classifier trained on noised inputs. Despite recent efforts to approximate the guidance (Kim et al., 2022a; Liu et al., 2023b; Wallace et al., 2023; Ben-Hamu et al., 2024) , they have mainly focused on computational efficiency, and have yet to achieve sophisticated performance on personalization tasks.Technically, to extend classifier guidance for personalized image generation, our work builds on a recent framework named rectified flow (Liu et al., 2023a) featuring strong theoretical properties, e.g. the straightness of its sampling trajectory. By approximating the rectified flow to be ideally straight, the original classifier guidance is reformulated as a simple fixed-point problem concerning only the trajectory endpoints, thus naturally overcoming its reliance on a special noise-aware classifier. This allows flexible reuse of image discriminators for identity preservation in personalization tasks. Furthermore, we propose to anchor the classifier-guided flow trajectory to a reference trajectory to improve the stability of its solving process, which provides a convergence guarantee in theoretical scenarios and proves even more crucial in practice. Lastly, a clear connection is established between our derived anchored classifier guidance and the existing approximation practices.The derived method is implemented for a practical class of rectified flow (Yan et al., 2024) assumed to be piecewise straight, in combination with face or object discriminators. This provides flexibility for a range of personalization tasks on human faces, live subjects, certain objects, and multiple subjects. Extensive experimental results on these tasks clearly validate the effectiveness of our approach. Our contributions are summarized as follows: (1) We propose a training-free approach to flexibly personalize rectified flow, based on a fixed-point formulation of classifier guidance. (2) To improve its stability, we anchor the flow trajectory to a reference trajectory, which yields a theoretical convergence guarantee when the flow is ideally straight. (3) The proposed method is implemented on a relaxed piecewise rectified flow and demonstrates advantageous results in various personalization tasks. 
background: 
 Personalized image generation studies incorporating user-specified subjects into the text-to-image generation pipeline. To preserve the subject's identity, the seminal works Textual Inversion (Gal et al., 2023) and DreamBooth (Ruiz et al., 2023a) finetune conditional embeddings or model parameters for each subject, which imposes high computational costs. Subsequent literature resorts to more efficient parameters (Hu et al., 2022; Han et al., 2023; Yuan et al., 2023) or a pre-trained subject encoder (Wei et al., 2023; Ye et al., 2023) to allow personalization within a few minutes or even without tuning. At the other end, a recent trend is the reuse of existing discriminators to improve identity consistency, such as extracting discriminative face features as the condition (Ye et al., 2023; Wang et al., 2024b) or as a training objective for the encoder (Peng et al., 2024; Gal et al., 2024; Guo et al., 2024) . However, these models require extensive pre-training on domain-specific data, e.g. LAION-Face 50M (Zheng et al., 2022) . In contrast, our method is a training-free approach that exploits existing discriminators based on the recent rectified flow model, allowing flexible personalization for a variety of tasks.Rectified flow is an instance of flow-based generative models (Song et al., 2021; Xu et al., 2022; Liu et al., 2023a; Albergo and Vanden-Eijnden, 2023; Lipman et al., 2023) . They aim to learn a velocity field v that maps random noise z 0 ∼ π 0 to samples from a complex distribution z 1 ∼ π data via an ordinary differential equation (ODE):dz t = v(z t , t)dt.(1)Instead of directly solving the ODE (Chen et al., 2018) , rectified flow (Liu et al., 2023a) simply learns a linear interpolation between the two distributions by minimizing the following objective:EQUATIONThis procedure straightens the flow trajectory and thus allows faster sampling. Ideally, a well-trained rectified flow is a straight flow with uniform velocity v(z t , t) = v(z 0 , 0) following:EQUATIONRecently, rectified flow has shown promising efficiency (Liu et al., 2024b) and quality (Esser et al., 2024; Yan et al., 2024) in text-to-image generation. Our work extends its capabilities and theoretical properties to personalized image generation via classifier guidance.Classifier guidance, initially proposed for class-conditioned diffusion models (Dhariwal and Nichol, 2021) , introduces a test-time mechanism to adjust the predicted noise ϵ(z t , t) based on the guidance from a classifier. Given condition c and classifier output p(c|z t ), the adjustment is formulated as:EQUATIONwhere s denotes the guidance scale, and σ t is determined by the noise schedule. Noteworthy, the condition c is not restricted to class labels, but can be extended to text (Nichol et al., 2022) and beyond. However, it is largely limited by the reliance on a noise-aware classifier for noised inputs z t , which restricts the use of most pre-trained discriminators that only predict the likelihood p(c|z 1 ) on clean images. Consequently, its usefulness is limited in practice. See Appendix B for more related work.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: The limitations are discussed in Appendix E. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 This work aims at customizing rectified flow with classifier guidance. We show that the above limit of classifier guidance may be solved with a simple fixed-point solution for rectified flow (Section 3.1).To improve its stability, Section 3.2 proposes a new anchored classifier guidance with a convergence guarantee. Lastly, the implementation and applications are described in Sections 3.3 and 3.4.This section first derives the vanilla classifier guidance for rectified flow, and then present an initial attempt to remove the need for the noise-aware classifier p(c|z t ), which is based on a new fixed-point solution of classifier guidance assuming that the rectified flow is ideally straight.The classifier guidance can be derived as modifying the potential associated with the rectified flow.According to the Helmholtz decomposition, a velocity field v may be decomposed into:EQUATION)where ϕ is a scaler potential and r is a divergence-free rotation field. They can be determined by solving the Poisson's equation∇ 2 ϕ(z t , t) = ∇ • v(z t , t), but this is beyond our focus. We directly add a new potential proportional to the log-likelihood to simulate classifier guidance, as follows:EQUATIONwhere s denotes the guidance scale, and ˆis used to distinguish the new flow from the original one. Subtracting the above two equations yields the vanilla classifier guidance, similar in form to Eq. (4):EQUATIONWhile this classifier guidance should allow for test-time conditioning of rectified flow, it cannot be applied in the absence of noise-aware classifier p(c|z t ). In the following, we show that this limitation may be overcome by exploiting the straightness property of rectified flow. Attempt to bypass noise-aware classifier. We make a key observation that the intermediate classifier guidance ∇ zt log p(c|z t ) can be circumvented by approximating the new flow trajectory to be straight (an ideal guidance should preserve the properties of rectified flow) and focusing on the endpoint z 1 . Formally, substituting t = 1 in Eqs. ( 3) and ( 7) allows skipping any intermediate guidance terms:EQUATIONInterestingly, this turns out to be a fixed-point problem w.r.t. z 1 , suggesting that the classifier-guided flow trajectory could be solved iteratively by numerical methods such as the fixed-point iteration, without knowing the noise-aware classifier. This greatly enhances the flexibility of classifier guidance to a variety of off-the-shelf image discriminators. However, our further analysis reveals both empirical (Section 4.3) and theoretical evidence questioning the convergence of this iterative approach: Proposition 1. There exist Lipschitz continuous functions v(z 1 , 1) and ∇ z1 log p(c|z 1 ), such that the fixed-point iteration for solving the target trajectory based on Eq. ( 8) is not guaranteed to converge by the Banach fixed-point theorem (Banach, 1922) , irrespective of the choice of s > 0.Proof. Consider the following construction. Let v(z 1 , 1) and ∇z 1 log p(c|z 1 ) be identical functions with a Lipschitz constant greater than 1. Then, the Lipschitz constant of the right-hand side of the fixed-point equation is greater than 1 for any s > 0. This violates the Banach fixed-point theorem's requirement for a Lipschitz constant strictly less than 1, thus convergence is not guaranteed.Proposition 1 shows that the derived fixed-point solution may not always be practical. Intuitively, even with a small perturbation at z 1 , the target flow trajectory estimated by Eq. ( 8) could diverge significantly after iterated updates, which hinders the controllability of rectified flow. This motivates us to anchor the target flow trajectory to a reference trajectory to stabilize its solving process.This section establishes a new type of classifier guidance based on a reference trajectory. The idea is to constrain the new trajectory to be straight and near the reference trajectory, as illustrated in Fig. 2 . It provides a better convergence guarantee and a certain degree of interpretability.Let ẑt and z t represent two flow trajectories originating from the common starting point z 0 with or without classifier guidance. The symbol ˆdenotes the new trajectory with classifier guidance.Assuming the two trajectories are close and straight (ideally preserving the characteristics of rectified flow), their difference can be estimated based on Eq. ( 7) and the first-order Taylor expansion:EQUATIONwhere the final step is derived from Eq. (3). From here, a new form of classifier guidance is obtained:EQUATIONThis new classifier guidance anchors the target velocity to a predetermined reference velocity v(z t , t) that is dependent only on t and irrelevant to the current state ẑt , thereby constraining the target flow trajectory near the reference trajectory and improving its controllability. Next, we extend its applicability to the more common scenarios where the noise-aware classifier p(c| ẑt ) is absent.Bypassing noise-aware classifier. To circumvent the intermediate classifier guidance, we follow the previous practice of substituting t = 1 into Eqs. ( 3) and ( 10), yielding a fixed-point problem w.r.t. ẑ1 :EQUATIONAs can be seen, the target endpoint ẑ1 is also anchored to a known reference point z 1 , which should enhance its stability in the solving process via fixed-point iteration or alternative numerical methods. Below, we exemplify its favorable theoretical property using the fixed-point iteration: EQUATIONThis suggests that our method is secretly estimating the intermediate classifier guidance with gradient backpropagation. While this is implicitly assumed or directly used in recent works that adapt classifier guidance to flow-based models (Wallace et al., 2023; Liu et al., 2023b; Ben-Hamu et al., 2024) , it is explicitly derived here based on a very different assumption (the straightness of the flow trajectory). Such a connection helps to rationalize both our adopted assumption and the existing practice.Extension to piecewise rectified flow. The above analyses are performed based on the assumption that the rectified flow is well-trained and straight, which is often not the case in reality. In fact, existing rectified flow usually require multiple sampling steps due to the inherent curvature in the flow trajectory. Inspired by Yan et al. (2024) , we adopt a relaxed assumption during implementation that the rectified flow is piecewise linear. Let there be K time windows{[t k , t k-1 )} 1 k=K where 1 = t K > • • • > t k > t k-1 > • • • > t 0 = 0,and the flow trajectory is assumed straight within each time window, then the inference procedure can be expressed as:EQUATION)where k is the index of the time window [t k , t k-1 ) that t belongs to. Note that this framework is also compatible with the vanilla rectified flow by setting K to the number of sampling steps.The previously derived fixed-point iteration in Eq. ( 11) cannot be applied directly, since its assumption that the target and reference trajectory segments share the same starting point (e.g. ẑt k-1 = z t k-1 ) may be violated after updates. A quick fix is to reinitialize the reference trajectory every round with predictions for updated target starting points. This allows to formulate the following problem:EQUATIONwhere the last step is obtained by recursively applying Eq. ( 12) to backpropagate the guidance signal, and a superscript e is introduced to denote the endpoint of the previous trajectory segment, as the above fix may disconnect different segments of the reference trajectory. Meanwhile, a straight-through estimator (Bengio et al., 2013) Solving target flow trajectory. The target trajectory under classifier guidance, subject to Eq. ( 14), can be estimated iteratively by starting withẑ[0] t k = z [0]t k and performing the following iterations:EQUATIONEQUATIONwhere the superscript [i] is used to indicate the target and reference trajectories at the i-th iteration. Specifically, Eq. ( 15) implements the prediction of updated target starting points by extrapolating from history updates, and Eq. ( 16) tackles the derived problem. Note that there are more sophisticated methods for predicting target starting points and solving this problem, e.g. quasi-Newton methods, but we opt for simplicity here and leave their exploration to future work. The complete procedure for implementing the proposed classifier guidance is summarized by Algorithm 1.The proposed algorithm is flexible for various personalized image generation tasks on human faces and common subjects. Given a reference image z ref and our generated image ẑ1 , we use their feature similarity on an off-the-shelf discriminator f , e.g. the face specialist ArcFace (Deng et al., 2019) or a self-supervised backbone DINOv2 (Oquab et al., 2023) , as classifier guidance. In addition, to improve the guidance signal, a face detector or an open-vocabulary object detector g is employed to locate the identity-relevant region for feature extraction. Formally, the classifier output is as follows:EQUATIONMore details are described in Appendix C. Notably, both configurations can be flexibly extended to a multi-subject scenario by incorporating a bipartite matching step between multiple detected subjects.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
experimental settings: 
 Datasets. Our method does not involve training data, as it operates only at test time. For face-centric evaluation, we follow Pang et al. (2024) to evaluate on 20 prompts with the first 200 images from CelebA-HQ (Liu et al., 2015; Karras et al., 2018) as reference images. For subject-driven generation, we conduct qualitative studies on a subset of examples from the DreamBooth dataset (Ruiz et al., 2023b) , spanning 10 subjects across two live subject categories and three object categories.Metrics. Three metrics are considered: identity similarity, prompt consistency, and computation time.The first two are measured using an ArcFace model (Deng et al., 2019) and CLIP encoders (Radford et al., 2021) , while the latter is tested on an NVIDIA A800 GPU. We reproduce the latest methods IP-Adapter (Ye et al., 2023) , PhotoMaker (Li et al., 2024) , and InstantID (Wang et al., 2024b) for a comprehensive comparison, and also include the existing baselines in Pang et al. (2024) .Implementation details. We experiment with a frozen piecewise rectified flow (Yan et al., 2024) finetuned from Stable Diffusion 1.5 (Rombach et al., 2022) with 4 equally divided time windows. The number of sampling steps is set to a minimum K = 4 given the memory overhead of backpropagation. A naive implementation takes 14GB of GPU memory, which fits on a range of consumer-grade GPUs.More results on alternative rectified flows can be found in Appendix D.4. For hyperparameters, the guidance scale is fixed to s = 1 in quantitative evaluation. Meanwhile, for stability, the gradient is normalized following Karunratanakul et al. (2024) . The number of iterations is set to N = 100.Face-centric personalization. For face-centric qualitative comparison in Fig. 3 , our method remains advantageous as its generated images by the guidance of the face discriminator exhibit high identity consistency. In comparison, InstantID (Wang et al., 2024b ) delivers a near level of consistency by controlling face landmarks, but sometimes distorts the face shape (the first and third images) and contains much less natural variation. More generated samples are provided in Figs. 11 and 12 in the appendix.Subject-driven generation. Our approach is flexibly extended beyond human faces towards more subjects, including certain common animals and regularly shaped objects. To validate our flexibility, Fig. 4 qualitatively compares it on three cats or dogs and a regularly shaped can, where the images generated by our method achieve highly competitive identity and prompt consistency. In comparison, the state-of-the-art method Emu2 (Sun et al., 2024) , as a generalist multimodal large language model, yields high identity similarity largely by reconstructing the input image, which limits its usefulness. The tuning-based Textual Inversion (Gal et al., 2023) and DreamBooth (Ruiz et al., 2023a) only work well with multiple images and exhibit inferior prompt consistency due to finetuned model parameters or prompt embeddings. See Fig. 13 in the appendix for additional results from more subjects. The base learning rate for gradient descent is 0.4, with momentum of 0.9 and an ℓ 2 regularizer of 1.0.Multi-subject personalization. Our method can be further extended to multi-subject scenarios via a bipartite matching step. Figure 5 compares it to the domain experts FastComposer (Xiao et al., 2023) and Cones 2 (Liu et al., 2023c) on composing multiple faces, live subjects and objects. As can be seen, our method achieves overall advantageous identity consistency, in spite of differences in non-persistent attributes such as hairstyle. Image semantics and quality are also well preserved, as exemplified by the amusement park details in the first image, with some others even surpassing the SD 2.1-based specialized model Cones 2. More generated samples can be found in Fig. 14 in the appendix. To justify the effectiveness of our proposed classifier guidance, Fig. 6 and Table 2 compare it with two variants: the previously derived guidance without anchor, namely using Eq. ( 8), and a gradient descent method on the initial noise similar to DOODL (Wallace et al., 2023) and D-Flow (Ben-Hamu et al., 2024) . The figure depicts that the gradient descent is unstable (left) and converges relatively slowly (right) despite using momentum and ℓ 2 regularization. And its identity preservation is sensitive to the learning rate. Though our new fixed-point formulation allows for a more stable layout, the initial version fails to converge as the face feature keeps drifting.In contrast, our full method exhibits better stability (left) and faster convergence (right) by implicitly regularizing the flow trajectory to be close and straight. This is further supported by the quantitative comparison, where our method delivers better identity and prompt consistency than the alternatives. Further analysis for hyperparameter sensitivity is provided in Appendix D.3.To validate the generalizability of our approach to broader application scenarios, we have extended it to more controllable generation tasks by directly using the guidance functions from Universal Guidance (Bansal et al., 2024) . The experimental results under the guidance of segmentation map or style image are illustrated in Fig. 7 . As shown, our classifier guidance can perform both tasks without any additional tuning, faithfully following the various forms of control signals provided by the user. This confirms the adaptability of our approach for various controllable generation tasks. Additional generalization analysis of our method for broader diffusion models is presented in Appendix D.1. Figure 7 : Experimental results for more controllable generation tasks. The first column shows the guidance, and the rest are the generated results. Our method is extended to various controllable generation tasks by incorporating the guidance functions from Universal Guidance (Bansal et al., 2024) . 
conclusion: 
 This work presents a training-free personalized image generation method using anchored classifier guidance. It extends the applicability of the original classifier guidance based on two key findings: first, by developing on a rectified flow framework assuming ideal straightness, the classifier guidance can be transformed into a new fixed-point formulation involving only clean image-based discriminators; secondly, anchoring the flow trajectory to a reference trajectory greatly improves its solving stability. The derived anchored classifier guidance allows flexible reuse of existing image discriminators to improve identity consistency, as validated by extensive experiments on various personalized image generation tasks for human faces, live subjects, certain objects, and multiple subjects. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper does not release any datasets or pre-trained models.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We have cited the original paper that produced the assets used in this paper and summarized their licenses in Appendix C.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: The code is available at this anonymized repository https://github.com/ feifeiobama/RectifID, along with a README documentation. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Most of the equations in our paper are accompanied by their derivations, except for Eqs. ( 11) and ( 12).For the sake of completeness, their derivations are supplemented here.Equation ( 11): We substitute t = 1 into Eqs. ( 3) and ( 10) to obtain:EQUATIONEquation ( 12): By applying Eq. ( 10) twice and utilizing the straightness property of rectified flow, we have:EQUATIONSince the proposal of classifier guidance (Dhariwal and Nichol, 2021) which uses a special noiseaware classifier as training-free guidance for diffusion models, many efforts have been made in order to extend its applicability to off-the-shelf loss guidance. They can be grouped into three categories:(1) Early literature focuses on simpler objectives for linear inverse problems such as image superresolution, deblurring, and inpainting (Chung et al., 2022 (Chung et al., , 2023;; Wang et al., 2023; Zhu et al., 2023) .(2) These methods can be extended to more complex discriminators through various approximations. (3) A recent line of work directly uses gradient descent with specific diffusion models. To enable their gradient computation, DiffusionCLIP (Kim et al., 2022a) relies on shortened ODE trajectories, while FlowGrad (Liu et al., 2023b ) adopts a non-uniform ODE discretization and decomposed gradient computation. DOODL (Wallace et al., 2023) , DNO (Karunratanakul et al., 2024) and D-Flow (Ben-Hamu et al., 2024) use invertible models or flow models to backpropagate gradient to the initial noise.Our proposed method is related to the third category, focusing on rectified flow whose approximation remains understudied. Moreover, it features a fixed-point formulation with a convergence guarantee for ideal rectified flow, which allows potentially better stability over the existing approaches, e.g. gradient descent on initial noise, as empirically validated in Section 4.3.Recent studies also explore the use of pre-trained classifiers to guide personalized image generation, but mostly during model training. DiffFace (Kim et al., 2022b) and PhotoVerse (Chen et al., 2023) directly apply a face discriminator to noised images to compute an identity loss. PortraitBooth (Peng et al., 2024) improves loss quality by computing it at less noisy stages. More relevant to our work, LCM-Lookahead (Gal et al., 2024) and PuLID (Guo et al., 2024) utilize distilled diffusion models to generate images in few steps, allowing for direct gradient backpropagation of image-space losses. However, these personalization methods must first be trained on extensive face recognition data, e.g. LAION-Face 50M (Zheng et al., 2022) . In comparison, we harness off-the-shelf face discriminators at test time based on the methodology of classifier guidance, enabling more flexible customization without training. And it can be generalized to other subjects by simply replacing the classifier.Method. We mainly experiment on the recently proposed piecewise rectified flow (Yan et al., 2024) .The model is finetuned from Stable Diffusion 1.5 (Rombach et al., 2022) on the LAION-Aesthetic-5+ dataset (Schuhmann et al., 2022) without special pre-training on human faces or subjects. We adopt a fixed image size of 512 × 512, number of sampling step K = 4, and a classifier-free guidance scale of 3.0 during quantitative evaluation. The newly incorporated anchored classifier guidance uses a default guidance scale s = 1.0 and number of iterations N = 100. For qualitative studies, a few results are generated with a slightly different guidance scale s = 0.5 or 2.0 for better visual quality or identity consistency. But in general, our method is not very sensitive to these hyperparameters.In terms of computational and memory overhead, our method takes less than 0.5s per iteration on an NVIDIA A800 GPU and fits on consumer-grade GPUs such as NVIDIA RTX 4080, the latter of which may be further improved with gradient checkpointing or the techniques in Liu et al. (2023b) .For face-centric personalization, our method is implemented with the antelopev2 model pack from the InsightFace library.2 Specifically, it detects and crops the face regions with an SCRFD model (Guo et al., 2022) , and then extracts face features using an ArcFace model (Deng et al., 2019) trained on Glint360K (An et al., 2021) , consistent with most personalization methods that use a face model (Ye et al., 2023; Wang et al., 2024b; Gal et al., 2024) . The resulting face feature is compared with the reference image to compute a cosine loss, which serves as the classifier guidance signal.For subject-driven generation, we use an open-vocabulary object detector OWL-ViT (Minderer et al., 2022) to detect the region of interest, and extract visual features with DINOv2 (Oquab et al., 2023) .The extracted feature is compared with the reference to calculate a cosine loss as the guidance signal. While this guidance already works well for various live subjects including multiple dogs and cats, we add an optional ℓ 1 loss with a coefficient of 10.0 to help preserve the identity of certain objects, such as cans, vases and duck toys. The current implementation is still limited in not capturing the details of some irregularly shaped objects, e.g. plush toys, and we expect to resolve this issue with an improved discriminator, or by combining with existing image prompt techniques (Ye et al., 2023) .For multi-subject personalization, we consider a simplified case of exactly two subjects and perform the following: first detect the two subjects, then enumerate all possible bipartite matches with the reference subjects to minimize the matching cost. For more complex scenarios, a possible workaround is to formulate it as an quadratic assignment problem and apply efficient solvers (Tan and Mu, 2024).Evaluation. For face-centric personalization, we follow Pang et al. (2024) in evaluating on the first 200 images in the CelebA-HQ dataset (Liu et al., 2015; Karras et al., 2018) with 20 text prompts including 15 realistic prompts and 5 stylistic prompts. The evaluation process reuses the code from Celeb Basis (Yuan et al., 2023) ,3 which first detects the face region using a PIPNet (Jin et al., 2021) with a threshold of 0.5 and then computes the cosine similarity on face features extracted by an ArcFace model (Deng et al., 2019) . It should be noted that our method adopts a different face detector and different alignment and cropping methods, so it does not overfit the evaluation protocol. For the baselines, in addition to those compared in Pang et al. (2024) , we also evaluate the recently proposed IP-Adapter (Ye et al., 2023) , PhotoMaker (Li et al., 2024) , and InstantID (Wang et al., 2024b) on their recommended settings. The number of their sampling steps is set to 30 for a fair comparison.The checkpoint version of IP-Adapter is ip-adapter-full-face_sd15. The image size is set to 512×512 for IP-Adapter and 1024×1024 for PhotoMaker and InstantID based on SDXL (Podell et al., 2024) .In the qualitative analysis, we also include Celeb Basis (Yuan et al., 2023) as a baseline method.For subject-driven generation, we perform a qualitative rather than a quantitative comparison on a subset of the DreamBooth dataset (Ruiz et al., 2023a) due to the previously mentioned limitations.Nevertheless, many subjects are considered during the qualitative study, including 7 live subjects from two categories (cats and dogs) and 3 regularly shaped objects from different categories (cans, vases, and teapots). For the baselines, we incorporate Textual Inversion (Gal et al., 2023) , DreamBooth (Ruiz et al., 2023a) , BLIP-Diffusion (Li et al., 2023) , and Emu2 (Sun et al., 2024) for extensive comparison.Their diffusion models and hyperparameter settings follow the official or Diffusers implementation.4 Licenses. The piecewise rectified flow (Yan et al., 2024) used in the main experiments is released under the BSD-3-Clause License and the 2-rectified flow (Liu et al., 2023a (Liu et al., , 2024b) ) used in Appendix D.4 is released under the MIT License. The InsightFace library for face detection and recognition is released under the MIT License, while its pre-trained models are available for non-commercial research purposes only. The OWL-ViT (Minderer et al., 2022) and DINOv2 (Oquab et al., 2023) models for object detection and feature extraction are released under the Apache-2.0 License. For evaluation, the code of Celeb Basis (Yuan et al., 2023) is licensed under the MIT License. The (Sauer et al., 2024 ) and phased consistency model (Wang et al., 2024a) , both distilled from SD and using 4 sampling steps in inference. The results show that our method is effective for personalizing broader diffusion models.CelebA-HQ (Liu et al., 2015; Karras et al., 2018) and DreamBooth (Ruiz et al., 2023a) datasets for quantitative and qualitative evaluation are released under the CC BY-NC 4.0 and CC-BY-4.0 licenses.While our classifier guidance is derived based on rectified flow, the same idea can be generalized to some few-step diffusion models by assuming straightness of their trajectories within each time step. We empirically demonstrate this in Fig. 8 with two popular few-step diffusion models, SD-Turbo (Sauer et al., 2024) and phased consistency model (Wang et al., 2024a) . As the results indicate, our method effectively personalizes these diffusion models to generate identity-preserving images. We will continue to explore this approach for other generative models in future research.More examples of our generated image are provided in Figs. 9 to 14. For face-centric personalized image generation, it is shown that our method can follow a variety of text prompts to generate both realistic or stylistic images while preserving the user-specified identity from a diverse group of people. Although there exist minor differences in the person's age and hairstyles, the face looks very similar to the reference image. In particular, the method demonstrates good generalizability among different piecewise rectified flows based on SDXL (Podell et al., 2024) and SD 1.5 (Rombach et al., 2022) . For subject-driven generation, new results are presented from additional subject types, including different breeds of cats and dogs, and some regularly shaped objects such as vases, demonstrating the flexibility of our approach across different use cases. For multi-subject generation, it naturally blends multiple human faces or live subjects into a single image while maintaining the visual quality and semantics, revealing a wider range of potentially interesting applications. Overall, our method demonstrates to be effective and flexible for various personalization tasks.In addition to the ablation experiments in the main paper, we perform a sensitivity analysis of the two hyperparameters in our method, namely the guidance scale s and the number of iterations N . Specifically, we study the effect of different s under a fixed N = 20 and then the effect of different N under a fixed s = 1.0. The results are summarized in Fig. 15 . As can be observed, increasing both hyperparameters from 0 leads to a significant improvement in identity consistency, confirming the effectiveness of our proposed classifier guidance. Also, the performance is stable over a fairly wide range of hyperparameters, indicating that our approach is not very sensitive to hyperparameters. Furthermore, we find that the use of a small classifier guidance scale is actually beneficial for prompt consistency, possibly because it enhances the visual features, as demonstrated in Fig. 2 .Figures 16 and 17 present additional qualitative results on a vanilla 2-rectified flow (Liu et al., 2023a (Liu et al., , 2024b) ) finetuned on Stable Diffusion 1.4 (Rombach et al., 2022) . As can be seen, our method continues to deliver satisfactory identity preservation when moving to a different model, despite a noticeable drop in the generation quality. Concretely, it integrates target subjects with some quite challenging prompts, such as a person swimming or getting a haircut, while showing very little interference with the original background, e.g. jungles and cityscapes. These results clearly validate the effectiveness of our proposed method in alternative rectified flow models.Note that we also experimented with K = 1 on a single-step InstaFlow (Liu et al., 2024b) distilled from this 2-rectified flow, but found that it tended to converge to slightly distorted images. This may be attributed to the larger modeling error inherent in InstaFlow's distillation process, which reduces the effectiveness of our approach assuming each flow trajectory segment is well-trained and straight.Broader impacts. The proposed method can be integrated with the emerging rectified flow-based models to enhance identity preservation and versatile control over existing AI art production pipelines. However, as a training-free personalization technique, it may increase the risk of image faking and have negative societal effects. Some immediate remedies include text-based prompt filters and AIgenerated image detection, but it remains an open problem for a more principled solution, for which we advocate further research on data watermarking and model unlearning as potential mitigations. To further clarify it, we provide a more detailed explanation of these mitigations below:• Prompt filtering, model unlearning: Since our method keeps the diffusion model intact, existing techniques for regulating diffusion models can be applied seamlessly, including prompt filters or unlearning methods. The former can be applied explicitly like the text filters in SD models, or implicitly via CFG as in Schramowski et al. (2023) . The latter approaches involve finetuning the diffusion model to remove the ability to generate harmful content (Gandikota et al., 2023; Kumari et al., 2023) . • Data watermarking: To prevent misuse of personal images, one could add a protective watermark to their images (Van Le et al., 2023; Liu et al., 2024a) . With this watermark or perturbation, the image can no longer be learned by common personalization methods. However, it is unclear how robust these watermarks are to training-free methods such as ours. An alternative watermarking scheme is to embed special watermark to the images generated by our proposed model, which would be invisible to the users yet identifiable by us (i.e., the tech provider). Images with such watermarks will be marked as being artificial. • AI-generated image detection: As a post-hoc safety measure, it helps to distinguish fake images generated by the attackers from real images. Beyond above watermark-based scheme, more sophisticated data-driven methods have attracted increasing interest from the AI community. Despite that current methods still lack accuracy, we believe that developing reliable and widely available AI-generated image detectors is an important research direction.Limitations. Our theoretical guarantee is limited to ideal rectified flow and cannot be generalized to more complex flow-based models. Empirically, anchoring the new flow trajectory to a reference trajectory only proves effective for faces, live subjects and certain regularly shaped objects, and remains insufficient for many objects with large structural variations, e.g. plush toys. Furthermore, while our method is training-free, its inference time has yet to match several training-based baselines, which may be addressed by applying more advanced numerical solvers to the derived problem.Another important issue is the lack of pre-trained discriminators. To address this in the short term, we suggest first training a specialized discriminator and then applying our classifier guidance. There are two reasons for doing this instead of finetuning the generator directly: (1) training/finetuning a discriminator is usually more efficient and stable than training/finetuning a generator; (2) it can take full advantage of domain images that have no captions or even labels by using standard contrastive learning loss. In the future, scaling up vision-language models may be a general solution for these domains. The current models such as GPT-4V (OpenAI, 2023) have demonstrated certain generalizability across visual understanding tasks. As they continue to improve in generalizability and robustness, they will become a viable source for guiding diffusion models in new domains. Figure 9 : Additional face-centric personalization results with piecewise rectified flow (Yan et al., 2024) based on SDXL (Podell et al., 2024) . Our method is compatible with more advanced base models and provides sophisticated personalization results. (Yan et al., 2024) , which is based on Stable Diffusion 1.5 (Rombach et al., 2022) . Our method achieves high identity consistency. See Fig. 16 for results on the vanilla 2-rectified flow (Liu et al., 2023a (Liu et al., , 2024b)) . (Yan et al., 2024) , which is based on Stable Diffusion 1.5 (Rombach et al., 2022) . Our method achieves high identity consistency. See Fig. 16 for results on the vanilla 2-rectified flow (Liu et al., 2023a (Liu et al., , 2024b)) . Figure 13 : Additional subject-driven generation results with piecewise rectified flow (Yan et al., 2024) , which is based on Stable Diffusion 1.5 (Rombach et al., 2022) . Our approach preserves the identity of both live subjects and some regularly shaped objects. Please see Fig. 17 for more examples using the vanilla 2-rectified flow (Liu et al., 2023a (Liu et al., , 2024b)) . Figure 16 : Face-centric personalization results with vanilla 2-rectified flow (Liu et al., 2023a (Liu et al., , 2024b)) , which is based on Stable Diffusion 1.4 (Rombach et al., 2022) . Our method preserves their identities well while remaining faithful to the text prompt during generation. (Liu et al., 2023a (Liu et al., , 2024b)) , which is based on Stable Diffusion 1.4 (Rombach et al., 2022) . Examples for additional categories of cats, dogs, and objects are included to demonstrate the effectiveness of our approach.NeurIPS Paper Checklist 1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: The main claims and contributions of this paper are summarized at the end of the introduction (Section 1). Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.https://github.com/deepinsight/insightfaceThis is mentioned at https://github.com/lyuPang/CrossInitialization#metrics.For example, we use https://huggingface.co/docs/diffusers/training/text_inversion. 


Decision: This paper was reviewed by three experts in the field. The reviewers agreed on paper acceptance due to its clear writing, well-motivated problem, interesting solution, highly efficiency and adaptability, flexibility to extend for other tasks, and strong experimental results. The scores are Accept, Weak Accept, and Borderline Accept after the rebuttal. Based on the reviewers' feedback, the decision is to recommend the paper for acceptance to NeurIPS 2024.

Reviews: review: paper_summary: paper_summary: The paper introduces a training-free method for subject-driven generation using diffusion models. This approach utilizes a new classifier guidance with off-the-shelf image discriminators and anchors the flow trajectory to a reference, ensuring stability and convergence. The method shows promising results in various personalization tasks for human faces, and other subjects.
strength_weakness: strength_weakness: 1. The approach heavily relies on the availability and quality of pre-trained image discriminators, which may limit its applicability in domains lacking robust pre-trained models. For example, the method for personalizing live objects (dog, cat) is quite ad-hoc and engineering for me.

2. The scope of this paper is quite limited while the method is very general. Therefore, experiments on other tasks, as mentioned below in the Questions section, can further strengthen the paper.
questions: questions: 1. How to deal with the case of lacking pre-trained discriminators ?

2. The experimental settings are primarily focused on personalization with face-centric and specific object categories. However, the method is quite general. Have the authors tried other tasks like in Universal Guidance [1] ?

[1] Universal Guidance for Diffusion Models. Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, Tom Goldstein. 2023
limitations: limitations: They have sufficiently addressed the limitations and potential societal impacts in their work.
paper_summary: paper_summary: The paper introduces a training-free method based on rectified flow and classifier guidance for personalized image generation. The experimental results show that the proposed method performs better than other state-of-the-art baselines in generating personalized images for human faces, live subjects, and certain objects.
strength_weakness: strength_weakness: - The theoretical justification or at least an intuition for Equation (6) is essential.
- Does the "Time" column in Table 1 refer to the training time or the inference time? Does calling the gradient with respect to a classifier affect the inference speed of the proposal compared to other baselines?
questions: questions: - The theoretical justification or at least an intuition for Equation (6) is essential.
- Does the "Time" column in Table 1 refer to the training time or the inference time? Does calling the gradient with respect to a classifier affect the inference speed of the proposal compared to other baselines?
limitations: limitations: NA
paper_summary: paper_summary: This paper introduces a new method for Rectified Flow models to perform classifier-guidance sampling without needing a noise-aware classifier. Specifically, the authors leverage a fixed-point method to overcome the need for a noise-aware classifier and anchor the classifier-guided flow trajectory to a reference trajectory to stabilize the sampling process. In practice, the authors apply the method to personalization tasks, transferring the identity of a face or an object from a reference image to new generations paired with customized prompts. The process looks promising as demonstrated in the paper.
strength_weakness: strength_weakness: Not as I can think of
questions: questions: Not as I can think of
limitations: limitations: The idea only applies to rectified flow models right now, but I think it could be generalized to border diffusion models, it would be interesting to elaborate more on this in the future.

Comments: title: Reply by Reviewer
Response by Authors
Ethics Review
after rebuttal
comment: Thank you to the authors for their responses. I would like to keep my positive evaluation on the paper.
Thank authors for your additional experiments for controllable generation tasks and an interesting answer of using LLM for Q1. Here, the additional experiments demonstrate that their method can be applied to other tasks with good results so I raise my score to 7
Thank you for your thoughtful review. We would remind that the potential ethical issues had been acknowledged **in Appendix E** of the original submission of this work, where we proposed some mitigating measures. To further clarify it, we provide  a more detailed explanation of these mitigations below:

* Prompt filtering, model unlearning: Since our method keeps the diffusion model intact, existing techniques for regulating diffusion models can be applied seamlessly, including prompt filters or unlearning methods. The former can be applied explicitly like the text filters in SD models, or implicitly via CFG as in [1]. The latter approaches involve finetuning the diffusion model to remove the ability to generate harmful content [2, 3].
* Data watermarking: To prevent misuse of personal images, one could add a protective watermark to their images [4, 5]. With this watermark or perturbation, the image can no longer be learned by common personalization methods. However, it is unclear how robust these watermarks are to training-free methods such as ours. An alternative watermarking scheme is to embed special watermark to the images generated by our proposed model, which would be invisible to the users yet identifiable by us (i.e., the tech provider). Images with such watermarks will be marked as being artificial.
* AI-generated image detection: As a post-hoc safety measure, it helps to distinguish fake images generated by the attackers from real images. Beyond above watermark-based scheme, more sophisticated data-driven methods have attracted increasing interest from the AI community. Despite that current methods still lack accuracy, we believe that developing reliable and widely available AI-generated image detectors is an important research direction.

We are dedicated to addressing the ethical concerns of our personalization method. Indeed, one of our other submissions to this conference proposed an unlearning-based solution for text-to-image models , which can be used to resolve the raised ethics issues to this work. We also promise to expand the related safety discussions in the revised manuscript.

---

[1] Schramowski, Patrick, et al. Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models. CVPR 2023.

[2] Gandikota, Rohit, et al. Erasing concepts from diffusion models. ICCV 2023.

[3] Kumari, Nupur, et al. Ablating concepts in text-to-image diffusion models. ICCV 2023.

[4] Van Le, Thanh, et al. Anti-dreambooth: Protecting users from personalized text-to-image synthesis. ICCV 2023.

[5] Liu, Hanwen, et al. Countering personalized text-to-image generation with influence watermarks. CVPR 2024.
Dear authors,

We kindly request that you check and response to this Ethics Review.

Best regards,  
AC
sorry for the initial short review and thanks for the author's rebuttal. After reading other reviewer's comments and the author's rebuttal, I don't think I have more insights to add here. And I would like to keep my rating.