Preventing Shortcuts in Adapter Training via Providing the Shortcuts
Authors: Anujraaj Goyal
Guocheng Qian
Huseyin Coskun
Aarush Gupta
Himmy Tam
Daniil Ostashev
Ju Hu
Dhritiman Sagar
Sergey Tulyakov
Kfir Aberman
Kuan-Chieh Wang
Venue: NeurIPS 2025 poster ()
Keywords: text to image generation
adapter
personalized generation
controllable generation
spurrious correlation
shortcut learning
Primary area: deep_learning
TL;DR: We find that rerouting spurious shortcuts in adapter training enables robust disentanglement for text-to-image generation with adapters.

Abstract: Adapter-based training has emerged as a key mechanism for extending the capabilities of powerful foundation image generators, enabling personalized and stylized text-to-image synthesis. These adapters are typically trained to capture a specific target attribute, such as subject identity, using single-image reconstruction objectives. However, because the input image inevitably contains a mixture of visual factors, adapters are prone to entangle the target attribute with incidental ones, such as pose, expression, and lighting. This spurious correlation problem limits generalization and obstructs the model's ability to adhere to the input text prompt.  In this work, we uncover a simple yet effective solution: provide the very shortcuts we wish to eliminate during adapter training. In Shortcut-Rerouted Adapter Training, confounding factors are routed through auxiliary modules, such as ControlNet or LoRA, eliminating the incentive for the adapter to internalize them. The auxiliary modules are then removed during inference. When applied to tasks like facial and full-body identity injection, our approach improves generation quality, diversity, and prompt adherence. These results point to a general design principle in the era of large models: when seeking disentangled representations, the most effective path may be to establish shortcuts for what should NOT be learned.

Preventing Shortcuts in Adapter Training via Providing the Shortcuts
Anujraaj Argo Goyal
Guocheng Gordon Qian
Huseyin Coskun
Aarush Gupta
Himmy Tam
Daniil Ostashev
Ju Hu
Dhritiman Sagar
Sergey Tulyakov
Kfir Aberman
Kuan-Chieh Jackson Wang
Snap Inc., https://snap- researchh http://a....rrrrr.aaai.iiiiia/ea aithbbii//hoottaut-rrrrringg
Abstract
Adapter-based training has emerged key mechanism for extending the pabilities of powerful foundation image generators, enabling personalized and stylized text-to-image synthesis. These adapters are typically trained to capture specific target attribute, such as subject identity, using single-image reconstruction objectives. However, because the input image inevitably contains a mixture visual factors, adapters are prone to entangle the target attribute with incidental ones, such as pose, expression, and lighting. This spurious correlation problem limits generalization and obstructs the model's ability to adhere to the input text prompt. this work, uncover simple yet effective solution: provide the very shortcuts we wish to eliminate during adapter training. In Shortcut- Rerouted Adapter Training, confounding factors are routed through auxiliary modules, such ControlNet or LoRA, eliminating the incentive for the adapter to internalize them. The auxiliary modules are then removed during inference. When applied tasks like facial and full-body identity injection, approach improves generation quality, diversity, and prompt adherence. These results point to a general design principle in the era of large models: when seeking disentangled representations, the most effective path may be to establish shortcuts for what should not be learned.
Shorteut-Rerouted
Introduction
In recent years, text-to-image (T2I) models have undergone remarkable progress, revolutionizing the way we generate and manipulate visual content from natural language prompts [Rombach et al., 2022, Ramesh et al., 2022]. While the expressive power of these foundation models has unlocked myriad creative and practical applications, much of their flexibility is realized not through retraining the backbone itself, but through the introduction of lightweight adapters. These adapters-ranging from low-rank adaptation modules (LoRAs) [Hu et al., 2022] to encoders [Ye et al., 2023] as modular steering mechanisms, enabling tailored functionality atop frozen foundation model. LoRA- based adapters, for instance, have empowered stylized and user- wwer-preference-ee. preffrence-coondttoonee conditioned generation, while encoder-based adapters facilitate personalized synthesis and style injection with impressive specificity [Zhang et al., 2023, Wang et al., 2023, Luo et al., 2024]. In essence, adapter training has emerged key enabler of fine-grained control in the modern image generation landscape.
Yet, adapters face fundamental challenge inherent to their training paradigm. The predominant approach-ssigle--meee reconstruction, thanks its simplicity and scalability-asks the adapter
*Conresooning authors: gqian wnw.eeataapcc..mmmmmmmm jwang23 snapchat.com
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Prior
Roaring
Laugh & Tilt
Selfie landmark
Relaxing w drink
+
+
+
Input
W/o SR-Training
With SR-Training
Figure 1: Shortcut Rerouting re-enables text control of pose and expression after adapter training. In the context personalized generation, without shortcut rerouting, the adapter overfits the reference image and reproduces its pose and expression, ignoring the prompt. With Shorteut- Rerouted Adapter Training, the adapter disentangles identity from other factors, allowing the model respond faithfully prompt-specified expressions and head poses. This restores compositionality. preserves the prior, and leads to more expressive and diverse generations.
faithfully reproduce target image from conditioning signal. An image, the adage goes, is worth a thousand words; more precisely, an image encodes an entire constellation of attribues style, geometry, camera parameters, lighting, and beyond. In most cases, however, wish to learn to encode only some specific attributes. and a thousand words are simply too many. The reconstruction loss, being agnostic this distinction, indiseriminately incentivizes the adapter to reproduce all visual factors present in the image. As a consequence, the adapter entangles the target factor with myriad incidental ones (i.e. shortcuts). See Fig. 2. An identity adapter intended to inject only the subject/person's 's appearance undesirably also copies and pastes their expression, pose, and leaks lighting background style. Nowhere is this conflation more problematic than in facial personalization, where isolating immutable appearance traits from mutable factors like head pose expression proves difficult. Moreover, as another key compounding factor, the distribution of the finetuning dataset is often significantly different from that of the foundation model. Such copy-and- copy-and-paste paste adapter training often introduces artifacts, such as degraded background generation, distorted human anatomy, and reduced esthetic quality (see Fig, 1). Ideally, identity encoder must only inject identity.
Our central idea is simple: to prevent the adapter from learning undesirable shortcuts, we explicitly provide those shortcuts during training (Fig. 3). Rather than hoping the adapter would disentangle complex factors on its own, we architect the learning process to route incidental factors through auxiliary modules-thh relieving the adapter of burden accounting for them. This reshapes the optimization landscape: when components of the reconstruction target are already explained by dedicated controllers (e.g., respective modules handling distribution shifts, pose, or expression), the adapter has no incentive to duplicate that behavior. The result is principled factorization of

---

ATTRIBUTES
Common Adapter Training
IMAGE
Identity
T2I
Background
Mode
Adapter
Cop
Pose
Figure 2: Common adapter training is susceptible to learning undesired shortcutes. The common single-image reconstruction objective used in adapter training inadvertently encourages the adapter to pick up all the attributes the adapter input (c.g. pose, expression, background, distribution) and leak them into the generation. While some confounding attributes like background can be factored out using masking, many other cannot. This makes learning a pure "identity" adapter challenging.
responsibility, wherein each module specializes in its designated role. Whereas a naive adapter tends copy pose and expression directly from the input image-thereby reducing the fidelity of the model prior and leading to degraded background generation adapter trained with shortcut rerouting learns to inject only the target identity. This restores prompt-based control over pose and expression and improves prior preservation (Fig. 1). To that end, this approach not only improves compositionality but also enhances the overall realism and diversity of the generated images.
In summary, our contributions are follows:
1. We propose a simple yet effective training paradigm, Shortcut Rerouting, for adapter training for large text-to-image (T2I) models.
2. We apply Shortcut Rerouting to the task of personalized image generation, addressing confounding factors such as distribution shift and spurious correlations. We demonstrate two practical instantiations of this idea using well-established tools-LoRA and ControlNet- explicitly factor out these shorteuts.
3. We empirically validate Chortcut-Reroutee adapters in two distinct settings-facial and full- body personalization and show improved controllability (via text prompts) with respect to expression, head pose, and body pose, well as stronger prior preservation. This leads to higher overall image quality, fidelity, and naturalness compared several strong baselines.
2 Related Work
Adapters in T2I Generation & Personalized Generation. The advent of text-to-image (T2I) diffusion models [Ho et al., 2020, Rombach al., 2022] has spurred a growing interest modular methods for task-specific and personalized generation [Ruiz al., 2023, Gal et al., 2022, Voynov al., 2023]. Central to this movement is the concept of aulapters-lightweiige
modules that steer the behavior of a frozen generative backbone. Among these, LoRA [Hu et al., 2022] has emerged as a widely adopted technique, enabling fine-tuning via low-rank parameter updates. For improved inference efficiency, encoder-based adapters, which inject conditioning signals through learned embeddings or attention modulation, have likewise been instrumental in enabling fine- ane-ggriiedd control over appearance, style, and compositionality [Ye et al., 2023, Wang et al., 2024, Xiao et al., 2023, Qian et al., 20.5a.b]. Yet, a well-known issue of adapter fine-tuning is that entangles identity injection with other undesired attributes like style, lighting, pose, and expression. In $4, apply Shortcut Rerouting to a simple adapter, IP-Adapter, and compare to strong recent baselines including InfU [Jiang et al., 2025), PulID [Guo et al., 2024], and a community implementation of IP-Adapter.
Shortcuts & Spurious Correlation. The phenomenon of shortcut learning- where models exploit spurious or unintended correlations in the data to optimize the training abjective-hat been extensively

---

studied in the broader machine learning literature [Geirhos et al., 2020, Luo et al., 2021]. In vision tasks, shorteuts often manifest when models rely on superficial cues, such as texture or background, instead of learning the intended high-level semantics [Geirhos et al., 2019]. Within the generative modeling community, recent works have noted analogous behaviors: generative models and their adapters may entangle target factors with irrelevant or transient features present in training data, leading to poor generalization and a lack of modularity. Several approaches have been proposed to combat shortcut learning, including data augmentation [Geirhos et al., 2020], causal regularization [Arjovsky et al., 2019], and architectural interventions [Islam et al., 2020J. In the context of T2I generation, methods such ControlNet [Zhang al., 2023] explicitly inject structural conditioning (e.g.. pose, layout) to guide synthesis, offering a promising avenue for disentanglement. However, to knowledge, no prior work has systematically leveraged such auxiliary modules during adapter training to proactively absorb spurious factors. Our method bridges this gap by architecting modular training process that reroutes undesired correlations through dedicated controllers, thereby preventing their entanglement in the adapter's representation.
Lastly, the notion of employing stage-wise training-wher solely during training to mitigate distribution shift-has been explored in several prior works. For instance, Jones et al. [2024] disentangled style and content learning by first training a content LoRA and subsequently a style LoRA, using only the latter at inference to achieve clean style transfer. Similarly, Guo et al. [2023] fine-tuned LoRA on the final video dataset better absorb the target distribution shift. Ostris [2024] proposed a LoRA variant capable of "un-distilliin"" Flux, allowing users to fine-tune the already step-distileed Flux[ [schnell] model. Building on these insights, our work generalizes this concept beyond LoRA-based adapters: we demonstrate that auxiliary modules such ControlNet can likewise be trained to absorb spurious correlations e.g.. those related to pose pppression-thereby isolating shortcut factors from the main adapter's 's representation.
auxiliary "training LoRA" is used
3 Shorteut-Rerouter Adapter Training
3.1 Mathematical Formalism
We formalize adapter training within a probabilistic framework to clarify the core challenge of disentanglement. Let X denote the observed image, which depends on two underlying factors: the target factor factor T (e.g., identity, style) and the confounding factors C' (e.g., pose, distribution shift, expression). Formally, we assume:
X~ p 17,C),
where T' is the factor we wish to faithfully capture via adapter training, and C represents incidental attributes that are not of primary interest.
The adapter a function that takes X as input and produces representation A(X), which is used to steer the generative model G. Ideally, we seek:
G(A(X)) ~ p(. 7),
meaning that the adapter should extract and inject only the information relevant to T, regardless of the confounds C.
However, in typical single-image reconstruction training (Fig. 2), the objective to minimize:
Exx) [C(G(A())))
which implicitly encourages A(X) to encode both T and C, since embodies all these factors. As result, the adapter becomes entangled: instead of isolating the target factor T. it captures spurious correlations mediated by C. These shortcuts, by minimizing the objective through confounding factors, inadvertently become reinforced and impair generalization test prompts.
We propose Chorteent-Reouted (SR) Adapter Training. As demonstrated in Fig. 3, our key insight is to reroute the shortcuts, i.e. the influence of C, through an auxiliary module Sc, which explicitly utilizes the confounding factors. The generative process modified to:
X SC()())
(4)

---

1. Shortcut-Rerouted Training
2. Inference
"standing in profile, gazing to the left"
T21
(eg. ControlNeti
Adapter
Adapter
Figure 3: Method. The Shortcut- (SR)-Module serves as generic shortcut adapter that can take various forms-such as ControlNet, LoRA, or Adapter-depent factor being addressed (e.g., pose, distribution, style). Illustrated here is the case of SR with ControlNet, where pose and expression cues are explicitly rerouted via the ControlNet during adapter training. At inference time, the ControlNet is removed, restoring independent pose and expression control from the text prompt alone.
on the confounding
where Sc is pre-trained and frozen module that directly provides C' to the generator, i.e. establishes shortculs. The revised training objective becomes:
Ex (C(G))(
(5)
which ensures that the confounding factors are explained away by Sc, leaving A(X) with no incentive to encode them. In effect, we turn the entanglement problem into a modular decomposition: AA is pressured to specialize in T, while Sc accounts for C' during training.
Finally, the shortcut module Se is removed during inference, recovering the original model, but equipped with disentangled adapter. Now, the generative process in inference, X G(A(X)), is less likely be impacted by the confounding factors from X.
This formulation reflects general principle: by explicitly modeling nuisance factors during training, we prevent the adapter from internalizing them, yielding cleaner and more robust representations.
3.2 Personalized Generation via Chorteut-Rerouteddde T2I Adapter Training
Adapter training aims to steer frozen text-to-image model by injecting additional signals-typically derived from reference inage-into its generation process. In the setting of personalized generation, adapters lightweight mod- ules that encode the input subject's identity and modulate the diffusion model personalize its output accordingly.
Personaliaation Finetuning
Model Distribution
3.2.1 Instantiations of SR Module
central challenge in conventional adapter training is that the adapter often encodes confounding factors-such such distribution biases in the fine-tuning dataset or pose and lighting leakage from the input images-thereby -thereyy entan- gling the target identity with spurious features. The SR module Sc in Eq. (4) is versatile, and can be realized by many different modules capable of absorbing specific con- founders. While in principle multiple such modules can be composed to form single unified SR module, in this work we focus two primary instantiations: SR-LORA, A, which addresses dataset-level distribution shifts, and SR-CN, which handles pose and expression leakage.
Figure 4: Distribution shift between model and finetuning dataset. Due to the distribution shift, directly training personalization adapter on the finetuning dataset leads degraded quality.
5

---

3.2.2 SR-LoRA: Addressing distribution shift
The first application of Chortcut-RRroutee Adapter Training addresses the issue of distribution shift between the model distribution and the data distribution used during adapter finetuning. In many real-world scenarios paatiiossprttillarrly with proprietary models such as Flux-the training distribution of the backbone model is unknown or opaque. Meanwhile, personalization pipelines often finetune on curated datasets with specific styles, subjects, or domains. This mismatch introduces latent confounding factor: the domain gap between the foundation model and the finetuning data.
To absorb this domain-induces shortcut, we instantiate Sc as a light-weight LoRA module, trained specifically capture this distributional gap. Concretely, we pretrain this LoRA on the finetuning dataset (e.g., studio-lit identity images), allowing it to absorb the dataset-specific style, lighting, and low-level features that differ from the base model's s prior. During adapter training, we then freeze the LoRA and train the identity encoder A as the only active module, allowing it to focus solely on identity, independent of the dataset domain:
== Sc((A))
where Se provides the latent adjustment required to bridge the domain gap, rerouting the shortcut through controlled path. As in our general formulation, this ensures that A(X) is no longer incentivized to account for the domain discrepancy, and instead focuses learning representation faithful to T.
At inference time, remove Sc, resulting in generation pipeline governed solely by A(X). This yields identity adapters that generalize beyond the specific visual domain of the training data and respond more reliably to test-time prompts across domains. In effect, this use case demonstrates that even abstract or latent confounders such as dataset shift-can can be systematically absorbed via shorteut modules, extending our methodology beyond structured confounds like pose expression.
3.2.3 SR-CN: Addressing Pose and Expression Leakage
Going beyond absorbing distribution shift, we aim to address the challenge of absorbing the shorteuts of expression and pose from the input image during inference. The target factor 7 is facial identity, while the target confounding factors Can for this SR module include head pose and facial expression. To absorb CcN. employ a pre-trained ControlNet [Zhang et al., 2023] module SCN that conditions generation on pose and expression maps derived from the training images.
During adapter training, we augment the generative pipeline as follows: given a training image X and its corresponding identity T and posc/expression CCN, we generate pose/expression maps (c.g.. via pose estimation and landmark detection) and feed these to Scx- The adapter A is then trained to inject identity alone, while SeN accounts for the pose and expression. The overall objective in adapter training stage becomes:
L(G(A(T), (CcN)),
(6)
which, as shown in Fig. 1, leads adapters that robust across wide range of poses and expressions. See Fig. for an illustration.
This approach highlights the generality and modularity of our method: by providing controlled pathway for spurious factors, we relieve the adapter from modeling them, resulting in cleaner and more generalizable representations.
4 Experiments
In this section, show number of experiments for different instance of Shorteut-Rerouted Adapter Training. First, we show results for Shortcut Rerouting in the setting of 'face' adapters using both LoRA and ControlNet as Shorteut Rerouting mechanisms ($4.2). The resulting adapters demonstrate improved prior preservation, head pose control and expression control. Then, we show results for Shortcut Rerouting in the setting of 'body' adapters ($4.3).
4.1 Experimental Setup
Datasets. We curate an internal large-scale dataset of few million high-quality human images, filtered to retain only single-subject photos and remove low-quality, NSFW, or watermarked content. To

---

IPA w/ SR-CN (Ours)
Input Face
InfU
PuLID
IPA
Prior
Snarling
Shouting hands raised
Laugh Tilt
YOOO
Figure 5: Qualitative comparison of different "face" adapters. Top: close-up portraits with varied expressions. Botto:: full-body generations. Our approach preserves the visual prior more faithfully, enabling expressive and identity-consisent personalized image generation.
accelerate training, we bucket images by aspect ratio and cache auxiliary modalities such as landmarks. segmentation masks, and text embeddings. For adapter inputs, we extract and align face crops using facial landmarks; for full-body, we extract body crops via segmentation and apply background removal. Captions are generated using Qwen2.5- 14B (text) and Intern VTT-300M-V2.5 (vision), both state-of-the-an large-scale captioning models. We also provide the details and visualization for test input images and prompts in the Appendix.
Training and Implementation details. All methods are implemented in PyTorch [Paszke al., 2019] using the HuggingFace Diffusers [von Platen et al., 2022] framework, based on the FLUX. 1 [Dev] [Labs, 2024] model with DiT [Peebles and Xie, 2023] backbone and Conditional Flow Matching objective [Esser et al., 2024]. Training is performed on 8 100 GPUs (80GB each) using Adam W [Loshchilov and Hutter, 2019] with a learning rate of 5e-5 and global batch size of for 250K iterations. Inference is standardized across all methods with IP scale 1.0, CFG 3.5. 28 steps, and 1024 x 1024 resolution. For identity encoding, we use openai/clip-vit--arggee parte- 14 [Rad- ford et al., 2021].

---

Table Quantitative comparison for "face" adapters. Our Chorteut-Rerouter methods (SR-LoRA and SR-C ER-ControlNet) outperform prior baselines in head pose control and prior preservation, while maintaining competitive identity fidelity. All models are based on Flux Dev.
Method
LLM Id.
FaceNet Id. 1
LLM Expr.
EMOCA Expr.
Head Pose
Prior (LPIPS)
InfU Diang et al., 2025]
3.3824
0.7402
3.7664
0,5420
17.7139
0,4490
PuLID (Guo et al. 2024]
4.2826
0,7742
3,5899
0,4890
17.5345
0,4584
IPA Ye et al., 2023]
4.7929
0.7150
3.0714
0,3470
16.1199
0,4800
SR-LoRA IPA (Ours)
4.7194
0.6708
3,4286
0,4580
13.2701
0.4330
SR-CN IPA (Ours)
4.7941
0.7118
3,6934
0,5800
12.6755
0,3937
Table 2: Quantitative evaluation of "body" personalization methods. Our SR-ControINe out- performs both InstantX and baseline IPA across all metrics, showing improved disentanglement and better adherence to pose and expression prompts without sacrificing identity.
Method
LLM Id.
FaceNet Id.
LLM Espr.
EMOCA Expr.
Head Pose
Body Pose 1
Prior (LPIPS) 1
InstantX [InstamXX 2024]
2,9930
0.3533
3,4736
0.4657
186, 7454
IPA IYo et al., 2023]
4,3986
0.5733
3,3000
0.3466
20.70
167.4000
0,4566
SR-CN IPA (Ours)
4.6510
0.5855
3.5263
0.4794
18.05
137.6888
0.4133
Metrics used to measure id preservation, prompt following, and prior preservation are follows:
1. FaceNet Id. (f) cosine similarity of the FaceNet [Schroff et al., 2015] embeddings of the generated image and the input subject.
LLM Id. (t) - a juuddge score for holistic identity. A good metric for personalization should capture the resemblance of the face, head, and hair. While existing Face metrics based on recognition models can capture the cropped face, it does not measure the head/hair. To have a more holistic measure, use LLM-as-a-jjdge similar recent studies [Luo et al., 2024].
LLM Expr. (t) - a LuMdee-a-judge score for alignment between the expression specification in the prompt, and the expression in the generated images.
4. EMOCA Sim. (f) cosine similarity between facial expression embeddings of the generated image and the prior image. The embeddings are extracted using EMOCA model [Dančček et al., 2022].
5. Head Pose. (4) the mean absolute difference in head orientation (i.e. yaw, pitch, and roll) between the generated and prior images, measured in degrees. We use HopeNet [Ruiz et al., 2018) estimate these angles.
Body Pose. (L) the mean L2 distance between estimated 2D body keypoints of the generated and prior images in pixel space. HRNet [Sun et al., 2019] is used for keypoint estimation.
7. Prior (LPIPS) (L) the LPIPS [Zhang et al., 2018 8] between the generated image and the prior image. Used to measure how much the generated image deviated from the prior.
More details such as the instructions for the LLM-as-a-juda metrics can be found the Appendix.
Baselines. Our experiments are conducted under two different settings: one where only the face used as input, and another where the fully body is provided. Baselines for face-input setting include InfU [Jiang et al., 2025], PuLID [Guo et al., 2024], and IP Adapter [Ye al., 2023] trained by us without shortcut rerouting. For the body-input setting, we include an open source model -P-apttt [Team, 2024]. This is general-
from InstantX, namely InstantX/FLa purpose IPA and not specifically trained for human faces, but serves as a representative baseline for comprehensive evaluation.
4.2 "Face" Adapters
Absorbing distribution shift. Our first key result, shown in Table 1, is the substantial improvement in prior preservation scores for both SR-LoRA and SR-CN. These gains are also clearly reflected in the qualitative examples in Fig. 5, where the image layout, texture, and overall visual quality remain closely aligned with the reference prior. In contrast, all baseline methods- -nncludinnn the IPA variant without shortcut -xhibit noticeable deviations in texture and scene fidelity. The

---

Input Body
InstantX
IPA
IPA w/ SR-CN (Ours)
Prior
Figure 6: Qualitative comparison of different "body" adapters. Our approach shows much stronger identity preservation than InstantX [InstantX, 2024], and much better adherence the prior and enhanced image quality when compared to vanilla IPA [Ye et al., 2023].
improvement in visual consistency highlights the effectiveness of shortcut rerouting in absorbing distributional differences during training.
Enabling text- eext-guided control of pose and expression. Our second key result that SR-CN better preserves and generalizes over mutable aspects of person's identity as pose and expres- sion-compared to standard IPA. Vanilla IPA suffers from pose and expression shortcuts, often copying these altributes directly from the inpul image. For example, in the first row of Fig. 5, the subject is smiling in the reference image, causing the output to ignore the prompt-specified expression of "snarling." Other baselines, such as InfU and PuLID, also struggle with identity fidelity, showing noticeable inaccuracies in head shape and hairstyle. distance cannot measure this identity shift caused by head shape and hairstyle. In contrast, the more disentangled adapter trained with shortcut rerouting not only respects prompt-driven expression but also supports natural and coherent full-body generations (see bottom half of Fig. 5).
worth noting that unfortunately face identity
4.3 "Body" Adapters
Beyond just face: Capturing identity aspects like body type. clothing, and limb proportions. We explore training adapters using full-body crops as input, which provide richer signal for capturing holistic identity traits such as body type, clothing, and limb proportions-factoo and character consistency downstream generations. However, this richer input also increases the risk of shortcut learning, particularly the tendency to copy the body pose from the reference image. As result, desired applications like reposing subject through text prompts become more difficult.
critical for realism
As shown in Table 2, IPA trained with shortcut rerouting achieves the highest performance across all metrics: identity fidelity, pose controllability, and prior adherence. These improvements are clearly visible in Fig. 6, where SR-CN IPA not only preserves subject identity more faithfully. but also produces outputs that are more consistent with the prior image layout and appearance.
4.4 "Background" Adapter
Additional variants that include further modules (e.g., background adapters) and extended combi- nations such as SR-LORA+CN, and SR-LORA+CN+BC are discussed the Appendix, along with ablation results illustrating their complementary effects. As example depicted in Fig. 7, the LoRA

---

w/ SR-LORA-ON
IPA SR-LORA-BG
w/ GR-LORA-CN-BO
Prior
IPA
SR-LoRA
Figure 7: SR- RR-Training is a versatile framework supporting different combinations of shortcut modules. The LoRA shorteut mitigates quality degradation, producing generations more consistent with the prior compared to the baseline. The ControlNet (CN) shortcut preserves pose priors, while the background (BG) shorteut prevents lighting leakage from the input. Notably, SR-LORA-CN follows the pose of the prior but deviates in the background, whereas SR-LoRA A-B preserves the background but deviates slightly in pose. Finally, SR-LORA-CN-BC aligns closely with both pose and background, thereby isolating and injecting only the target identity.
shorteut substantially alleviates quality degradation, the ControlNet (CN) shorteut reliably maintains pose priors, and the background (BG) shortcut effectively suppresses illumination leakage.
Limitations. In this work, we introduced Chortcut-RRroutce Adapter Training, a simple yet broadly applicable framework for disentangling spurious correlations in text-to-image personalization. One limitation with the current evaluation is our focus on encoder-based adapter training. In principle, our approach could also be applicable to LoRA training, such as learning a style LoRA free of undesired shifts like layout, or content. Specifically for the task of personalization, one limitation is the fact that we applied Shortcut-Reroutinn Rerouting to IP-Adapter only, which is fairly simple baseline. Applying our approach to stronger baselines might lead to overall better performance.
Ethical Considerations. As our method improves identity preservation and expression control in personalized generation, it naturally raises concerns about misuse, particularly in the creation of hyper-realistic synthetic identities or deepfakes. We acknowledge that enhanced controllability and realism may lower the barrier for malicious use. To mitigate such risks, advocate for responsible deployment practices, such as model watermarking, usage restrictions, and alignment with ethical frameworks for generative media.
Conclusion
We introduced Infortcut-Reroutingg simple yet general framework for disentangling spurious corre- lations in text-to-image personalization. By explicitly providing shorteut pathways for confounding factors during training-viis modules such as LoRA ControlNet prevent adapters from in- ternalizing undesirable attributes like pose, expression, or domain-speciffe biases. This leads to cleaner, more controllable representations that preserve identity while restoring the model's ability to respond to prompt-based based instructions. Our experiments across both face and full-body personalization demonstrate improved controllability, prior preservation, and generation quality.
More broadly, our results suggest a general principle: when training models to focus on what matters. it is often most effective to explicitly route away what does not. We believe this perspective of Shortcut Rerouting has implications beyond personalization: the idea of absorbing confounding variation through targeted pathways can inform future approaches to modular, interpretable, and more controllable generative systems.
10

---

References
Martín Arjovsky. Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. CoRR, 0et119977.0293,, 2019.
Radek Daněček, Michael Black, and Timo Bolkart. Emoca: Emotion driven monocular face capture and animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20311-20322 2022.
Jiankang Deng, Jia Guo, Jing Yang, Niannan Xue, Irene Kotsia, and Stefanos Zafciriou. ArcFace: Additive angular margin loss for deep face recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10):5962-5979, oct 2022. doi: 10. 10110///amm222211357777777 URL
https: //opi. org/10. 1109%2Ftpami. 2021. 2021. 3087709.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. In Proceedings of the International Conference on Machine Learning (ICML). OpenReview.net, 2024.
Rinon Gal, Or Patashnik, Haggai Maron, Gal Chechik, and Daniel Cohen-Or. Stylegan-nada: Clip-guided domain adaptation of image generators. arXiv preprint arXiv:2108. 00946, 2021.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual 2022.
inversion. arXiv preprint
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-traine cnns are biased towards texture; increasing shape bias improves accuracy and robustness. In ICLR. OpenReview.net, 2019.
Robert
Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard S. Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. Shorteut learning in deep neural networks. Nat. Mach. Intell., 2(1 1)x665)-6111 2020.
Yuwei Guo, Ceyuan Yang. Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307. 04725, 2023.
Zinan Guo, Yanze Wu, Zhuowei Chen, Lang Chen, and Qian He. Pulid: Pure and lightning ID customization via contrastive alignment. CoRR, abs/2404. 16022, 2024.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-663 0000068811111 2020.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
InstantX.
FLUX. -ey------aae
https: http://///ggggggat..eoooeaeaaaae
1-dev-Ir -P- Adapter, 2024. Accessed: 2024-11-01.
Md. Amirul Islam, Sen Jia, and Neil D. B. Bruce. How much position information do convolutional neural networks encode? In ICLR. OpenReview.net, 2020.
Liming Jiang, Qing Yan, Yumin Jia, Zichuan Liu, Hao Kang, and Xin Lu. Infiniteyou: Flexible photo recrafting while preserving your identity. arXiv preprint arXiv:2503. 2025.
Maxwell Jones, Sheng-Yo Yu Wang, Nupur Kumari, David Bau, and Jun- Yan Zhu. Customizing text-to- image models with a single image pair. In SIGGRAPH Asia 2024 Conference Papers, pages 1-13, 2024.
Black Forest Labs. Flux. https://gittie
http/blackk http://o/oooaaaaaooootttaaaa/a// 2024.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, In ICLR, 2019.

---

Michael Luo, Justin Wong, Brandon Trabucco, Yanping Huang, Joseph E Gonzalez, Ruslan Salakhut- dinov, Ion Stoica, et Stylus: Automatic adapter selection for diffusion models. Advances in Neural Information Processing Systems, 37:32888-329155 2024.
Xu Luo, Longhui Wei, Liangjian Wen, Jinrong Yang. Lingxi Xie, Zenglin Xu, and Qi Tian. Rectifying the shortcut learning of background for few-shot learning. Advances in Neural Information Processing Systems, 34:13073-13085 2021.
OpenAL. Gpt-4o technical report, 2024. URL httppp::/oopoaa..o/ooee///e///// Accessed: 2025-05-22.
Ostris. Flux. ----chelll-raainnn--daa--- 2024. URL www.lltttraainnnnaaaaaaaa
Accessed: 2025-10-15.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4172-4182 IEEE, 2023.
Guocheng Qian, Kuan-Chieh Wang, Or Patashnik, Negin Heravi, Daniil Ostashev, Sergey Tulyakov, Daniel Cohen-Or, and Kfir Aberman. Omni-id: Holistic identity representation designed for generative tasks. Proceedings of the Computer Vision and Pattern Recognition Conference, pages 8786-8795, 2025a.
Guocheng Gordon Qian, Daniil Ostashev, Egor Nemchinov, Avihay Assouline, Sergey Tulyakov, Kuan-Chieh Jackson Wang, and Kfir Aberman. Composeme: Attribute -rrubutee-pecific image prompts for controllable human image generation. arXiv preprint arXiv:2509.18092, 2025b.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.0612: 06125, 1(2):3, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, pages 10684-10695, 2022.
Nataniel Ruiz, Eunji Chong, and James M Rehg. Fine-grained head pose estimation without keypoints. In Proceedings of the IEEE Conference on Computer Vision and Patlern Recognition Workshops (CVPRW), 2018
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, pages 22500-225100 2023.
Florian Schroff. Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pages 815-823, 2015.
Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5693-5703, 2019.
InstantX Team. Instantx
ip-adapter page.
1-dev- IP- Adapter, 2024.
Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, and Thomas Wolf. Diffusers: State-of-the-art diffusion models. https:
//github com/huggingface. http://ggggg..aaafaaff..sts.e diffusers,, 2022.
12

---

Andrey Voynov, Qinghao Chu, Daniel Cohen-On and Kfir Aberman, p+: Extended textual condi- tioning text-to-image generation. arXiv preprint 0rXiv:2333.09522 2023.
Kuan-Chieh Wang, Daniil Ostashev, Yuwei Fang, Sergey Tulyakov, and Kfir Aberman. Moa: Mixture- of-attention for subject-context disentanglement in personalized image generation. In SIGGRAPII Asia 2024 Conference Papers, pages 1-12, 2024.
Zhouxia Wang, Xintao Wang, Liangbin Xie, Zhongang Qi, Ying Shan, Wenping Wang, and Ping Luo. Styleadapter: A single-pas, pass lora-free model for stylized image generation. ArXiv, abs/2309.01770, 2023. URL https: / / api. semanticscholarr org/ he//oorpusIDD :D::1551D: 261531689.
Guangxuan Xiao, Tianwei Yin, William T Freeman, Frédo Durand, and Song Han. Fastcom- poser: Tuning-free multi-subject image generation with localized attention. arXiv preprint 1044: 2023.
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2330..00 06721,, 2023.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to -ext-to-image diffusion models, 2023.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
13

---

Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [NA]
Justification: At the time of submission, we are unable to release the code or dataset due to legal and compliance constraints within organization. We recognize the value of open access and reproducibility and are actively exploring the possibility of releasing portions of the code evaluation tools pending internal review. We provide detailed descriptions of our methodology, datasets, and evaluation setup in the paper and appendix to support reproducibility in principle.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
public/ cepllcc/gudees guides/ Saaiides/ooeSubmiise mom.iisssoooiiiyy. for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /CodeSubmic / /nips. cccpublic//eeeeecccccccccceeeeee httpl////uuiieesseeSuuiiiiii cettubmmsssoonii ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : /nwurripss..../ http/pullc/ttcistuuuteeee
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https: : http///eeepppp///eee//ee/ htttp::/earrrps..ccoofffeeeeeeee /ot.e///eeeeeeeee///2/////////// for what should or should not be described.
20

---

A
Acknowledgments
We thank Denis Bondarev, Ergeta Muca, Bridget Briley-Snook, Jackie Fuhrman, Jonathan Solichin, Julia Krysko, and Svitlana Stern for their guidance, production support, reviewing paper drafts, and coordination across AR engineering and creative efforts that made this work possible.
B Comparing LLM-as-a-judge with FaceNet as a metric for Identity Preservation
Re'erance
Cross Reforence
Ours
JURE RoasOn
PULID
LU ROUsOn
InVU
LUM Reason
2:23
Racaber
Figure 8: Qualitative comparison between LLM-based identity score and FaceNet Similarity. The face crop from the reference image is used as input to the face adapter. The cross reference image is shifted view of the same person and is used to compute identity scores.
We observe that face recognition models such as FaceNet [Schroff et al., 2015] and ArcFace [Deng al., 2022], commonly used for face similarity metrics, fail to account for variations in hairstyle, hair color, headwear, and head shape. We hypothesize that this limitation stems from their training setup, where the input face is tightly cropped the size of 0.8x of FFHQ [Gal et al., 2021]), resulting in narrow understanding of facial identity. Consequently, these models provide unreliable assessments of identity similarity in more holistic capture including hairstyle. To address this, we leverage large ChatGPT-40 [OpenAI,
language models (LLMMs) with strong multimodal 2024]-to perform identity assessments. As illustrated in Fig. 8, the L.MM accurately captures identity similarity in cases where FaceNet or ArcFace fails.
21

---

C LLM-Based Identity Evaluation Prompt
We LLM--as-a-judge evaluate identity preservation in generated images. The LLM is given two face images and asked to judge whether they show the same person. The prompt shown below:
You are expert in facial recognition. Given two images of faces, your task is to judge whether they show the same person. Steps:
1. Carefully compare facial features like shape, eyes, nose, mouth, jawline, eyebrows, and overall proportions.
Ignore lighting, angle, pose, worn accessories, or expressions.
Rate identity similarity on a scale from to
5 Definitely the same person
4 Very likely the same person
3 Possibly the same person
2 = Unlikely the same person
Definitely not the same person
Explain your reasoning by referencing specific facial features.
Respond in the following format:
Rating: <1-5>
Reason: <explanation>
D LLM-Based Expression Evaluation Prompt
We use an LLM-as-a-judge to assess how well generated images reflect the intended facial expres- sions described in text prompts. The model is instructed to evaluate *only* the facial expression, disregarding pose, clothing, background, or lighting. The exact prompt provided below:
You are an expert judge tasked with evaluating how accurately a generated im- age captures the facial expression described in text prompt. Your evaluation should focus only on the expression (such as happiness, sadness, anger, surprise, etc.) and ignore other factors like the person's 's pose, clothing, background, lighting.
Steps:
Carefully read the text prompt.
Examine the facial expression in the generated image.
Compare the image's 's expression the description the prompt.
Rate the expression fidelity on scale of to 5:
5 Perfect match.
4 Mostly accurate with minor discrepancies.
3 Partially matches but has clear inaccuracies.
2 Mostly incorrect expression.
1 = Completely wrong expression.
5. Briefly explain why you gave this rating, pointing out specific facial features
(mouth, eyes, brows, etc.) that contributed to your assessment.
Respond in the following format:
Rating: <1-5>
Reason: <explanation
E Enabling expression, pose, and lighting control
In this section, we highlight the fine- fine-grained control over pose, expression, and lighting enabled by our shortcut-rerouted training. In 9, pose varies across columns while holding expression constant, and expression/lighting vary across rows with fixed pose. This disentangled control achieved while faithfully preserving identity from the reference image.
22

---

Looking up
Looking sbrighht
Posing Left
Looking over
spiden www.igutiigg........... from lef
Laughing hysterisally
Deep not sioee lightting
IADENGE shouting
Figure 9: Shorteut Rerouting enables to precisely control lighting, pose, and expression from text. Adapter preservers from prior while utilizing only personalization related attributed from the reference image. Zoom for best view.
F Additional Qualitative Results for Face Adapters
G
Additional Qualitative Results for Fullbody Adapters
H
Evaluation Prompts
We randomly sample four identity images for each of the text prompts below, where serves as placeholder for the gender noun (e.g., man, woman).
Accessory
distingusshed (0) weasing
gold irimmed munocle, puired with classic black suit. where the monocle illuminated under bright studio lights. focusing hut' details.
Inshionbble posing
wearing
A glamorous (0) entertaaser stage, wearing
feathered carnings, dramaic lightng accentuating glaooos (0) entertuiner stage, wearing extra-ong,, feathern cartings,
dimatic lighing sccentuating
A modem (0) bride wearing an unconventiona black lave veil, highlighted by sofi backlighiing that empbasizes the veil's intricase pallern.
A [0] strikig pose in an alley, weating chinky, staided wrishhads as the focal point against graffii-cowene walls.
A [0] walking through a basy city street, wearing bold. oversized sunglasses with mirroted lenses that reflect the urban skyline.
vetro-chic (0) fashion model in vintage palka-doe scuef matching glowes. with the guttemed acoessories taking center stage.
tugged (0; adventumer stunding cliff, spurting high tech, reflective helmet with built goggles, miking helmet the cenierpiece the shot, survondings, wearing knse-high lac-upp sparkling sepiins, odgy street performer showersing multiple silver pieccings spiked choker,
(0) dancing
up angles capturing every
Activity
A [0] coffer shop.
office.
A [0] painting studio.
playing basketball NBA.
A [0] riding a hike.
sitting bench park.
23

---

Prompt
HU
PLUIO
IPA
[Ours)
[Dunt
(Dun)
(Dun)
Patting
Prior
lenoeand mabds
Vellad dala wnth
agann per
tarniing
Right
Figure 10: Qualitative comparison different "face" adapters. We evaluate adapters on their ability to edit and preserve wide variety of facial expressions, poses, accessories, lighting, and activities. SR-LoRA addresses quality degradation; SR-CN achieves better pose control; SR-LoRA- BG improves background and lighting consistency: and using both SR-LoRA and SR-CN, namely SR-LoRA-CH effectively maintains pose while maintaining the quality; Zoom in for best view.
24

---

IP we Sk-
IPA Ja cR-
Ih SeLokitCCM IAA ot SR coiteRG
Figure 11: Qualitative comparison of different "face" adapters. We evaluate adapters on their ability to edit and preserve wide variety of facial expressions, poses, accessories, lighting, and activities. SR-LoRA addresses quality degradation; SR-CN achieves better pose control; SR-LoRA- BG improves background and lighting consistency; and using both SR-LoRA and SR-CN, namely SR-LoRA-CN effectively maintains pose while maintaining the quality; Zoom in for best view.
Expression
(0) biting lower lip nervously. slight frown, brows knit, awkward tension in the face
(0) blowing a kiss, lips puckered, eyebrows raised gently, soft lighting
[0) caught muidexxpression between a laugh and cry. watery eyes, twisted smile, bittersweet emotion
[0) crying intensely, eyebrows arched upward, mouth twisted in pain, eyes squeezed shut, emotion
[0) crying with one eye shut tighter than the other, mouth open mid-sob, flushed cheeks
[0] laughing hysterically with eyes shut tight and mouth wide open, cheeks raised, expressive lighting
[0] laughing with head tilted back, eyes closed, mouth wide open, pure joy on the face
[0] roaring anger, brows furrowed, nostrils flared, teeth bared, dramatic shadowing
(0) screaming with eyes wide and jaw fully dropped, intense emotion on face, dramatic lighting
[0] shocked with uneven brows, wide eyes, jaw slack, vivid emotion in facial pose
[0) shouting loudly, mouth wide, eyes intense, hands near face, motion blur
[0) smiling with eyes squeezed shut, mouth open in pure elation, cheeks lifted high
[0) smirking with head tilted slightly, one eyebrow raised, subtle attitude in the eyes
[0] snarling with clenched teeth, nose scrunched, eyes narrowed in aggression
[0] sneering in disgust, nose wrinkled, upper lip curled, one eye slightly squinted, gritty atmosphere
(0) sticking out tongue in mocking expression, playful eyes, raised eyebrow, casual setting
(0) stunned with jaw dropped, eyebrows raised high, eyes wide open, sharp lighting
[0] wincing in pain with eyes tightly shut, lips twisted, brow tense, expressive emotion
[0) winking with mischievous grin, eye squinting and shut, playful mood
(0) winking with mischievous grin, eye squinting, playful mood
(0) yawning deeply, mouth stretched wide, eyes half-closed, mid-motion, natural lighting
25

---

Lighting
[0] dancing shifting multicolor lights
A [0) gazing sideways, lace partially lit by peach morning light
[0] dynamic
under cold cyan lighting
A [0] in crouched pase under deep indigo overhead spollight
A [0] leaning
wall, lit from below with greenish hoc
[0] leaning forward cool violet light
A [0] mid- motion, under red and purple colored strobe lights
[0) reaching upward under golden
rays
A [0] realining on couch under dusty rose lighting
A [0] sitting
legged, with emerald [0] sitting sideways under diffused lighting
hue
A [0] standing tall, shadow cast long under low amher light
orange sunset light, profile facing right A (0) turning back toward camera, lit from behind with white light
[0] stunding
A [0] under
candleligh. hunds resting
A [0] walking into warm golden spollight from the side
A (0) with amms crossed, under vibrant magenta cim lighting
hright orange studio lights A [0] with bead ulted back, lit by soft lavender haze
[0] hands hips,
[0] with hand pocket, standing bluc-tinted window light
A [0) gazing up towering waterfall, their bead tilted back and ams outstretched
A (0) knceling one knce dramatic, torchtit leaning forward with lixed the horizon, exuding focus and intensity.
pockets, shoulders slighly slouched, glancing away with expression. A [0) seated cross-legee legged on a wonden floor, hands resting on knees, gazing slightly downward in meditative calm.
(0) leaning against grafitti-overr covered
A (0) sanding profile, they gazing the
A (0) sanding protile, they
A [0) standing tall, their hands on their hips, and their chin held high. in front of modern office backdrop.
A [0) standing tall, with their hands their hips, and their chin held high, in froet of modern office backdrop.
stylish turning glance their right shoulder.
profiie partial silhouctie against a glowing city skyline sunset.
A confident (0) facing forward, looking directly into the cameru lens with poised, straight backed posture, set against clean studio backgrundd
26


Reviews:
[{"review_id": "s3WmAXnsKx", "reviewer": "Reviewer_wPqP", "summary": "This paper concerns the training of \"adapters\" for text-to-image models. Specifically, the paper discusses the cases where the adapter is supposed to capture the identity or unique look of a reference input image. The proposed SR method (Shortcut-Rerouting) suggests that existing adapter training methods result in confounding factors being transferred from the reference input image to the generated image and that to avoid this, extraneous factors should simply be provided directly as a condition (provide the shortcut). The authors evaluate identity consistency, expression alignment, head/body-pose alignment and other metrics in comparison to existing adapter methods such as IPA and show that their proposed modification (SR-LoRA or SR-CN) result in improvements across the board. The qualitative results show better adherence to both the identity contained in the reference input image as well as better adherence to the content of the prior image (e.g. lighting, pose, expression).", "questions": "1. This paper only discusses the potential of learning pose/keypoint related shortcuts using LoRA/ControlNet. Did you consider learning other shortcuts using existing encoders? E.g. expression parameters from EMOCA, CelebA attributes, head-pose from HopeNet.\n2. Did you consider using an ArcFace/AdaFace model to compute identity consistency as is common in face avatar research?\n3. Is an LLM a good judge of identity or facial expressions?\n4. Qualitative results for the SR-LoRA method is not shown in neither the main paper nor the supplementary materials. Is there any particular reason for this?", "limitations": "The authors briefly discuss the limitation of their work (adapter-focused training, not LoRA), but do not discuss any potential negative societal impact.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper is written well with clear figures that are easy to understand. The intuition for learning shortcuts (or confounding factors directly) is explained very well in Section 3.1. The overall concept can be easily understood and the method is presented with sufficient details to allow reproduction. The qualitative results are particularly impressive, showing that the proposed modifications can drastically improve identity consistency and adherence to details of the prior image such as pose/expression/etc.\n\nA few questions do arise, however. For instance, the authors provide keypoints as explicit \"shortcut\" signals during training, but do not discuss the possibility of using other specialist encoders such as those for expression, body shape, attributes (e.g. CelebA). This leaves the user feeling that the overall \"shortcuts\" and \"shortcut-rerouting\" idea sounds great, but may only apply to specific cases such as pose / landmarks, without being very general. Furthermore, how can this idea of shortcuts be applied to T2I tasks for non-human subjects?\n\nIn addition, the metrics used in Sec. 4.1 are mostly sensible, but the Expr. metric in particular is using an LLM-as-a-judge even though encoders such as EMOCA exist for directly predicting expression parameters. The models used are generally dated, especially because identity is determined using FaceNet instead of the more widely used ArcFace or AdaFace models.\n\nIn the quantitative evaluations, the authors do not mention which dataset they use. It appears as though they are using a single internal dataset, which may not allow generalizable insights. As a potential improvement, the authors could consider evaluating on datasets of sufficiently different characteristics, e.g. studio images, in-the-wild, cartoon/drawings.", "was_revised": true, "final_justification": "I have gone through my peers' reviews and have reviewed my own review and prior assessment.\n\nThe authors' answers have satisfied my questions and doubts. Especially, I now understand (also based on the supplementary materials) the suitability of LLM-as-a-judge for aspects that ArcFace or EMOCA cannot cover/assess. I also appreciate the EMOCA similarity evaluations and the evaluation on cartoons/drawings. I believe that the qualitative improvements in the supplementary materials are impressive as there is better adherence to both prior as well as input condition when compared to baselines. I raise my rating accordingly.", "rebuttal": "We thank reviewer wPqP for the constructive and thoughtful feedback on our submission. We appreciate your recognition of the clarity of our presentation, the intuition behind shortcut rerouting, and the strong qualitative results demonstrating improved identity and attribute preservation. Your comments have been very helpful, especially the suggestion to use the EMOCA model for expression evaluation. We address the concerns raised below.\n\n\n### **Q1**: “Did you consider learning other shortcuts using existing encoders? E.g. expression parameters from EMOCA, CelebA attributes, head-pose from HopeNet ...”\nWe are grateful for the suggestions to explore shortcuts using existing encoders like EMOCA or HopeNet. Our method establishes pose and expression shortcuts via ControlNet. Achieving a similar outcome with EMOCA/HopeNet would require learning an adapter compatible with their outputs, which would be less user-friendly than the presented approach.  \n\nWe believe our extensive experiments sufficiently demonstrate the effectiveness of SR training in human-centric generation tasks. While our SR training strategy is inherently task-agnostic and broadly applicable, extending it to non-human subjects is beyond the scope of this work. As a team focused on human-centric personalization, we currently lack the resources (e.g., data and compute) and domain expertise to train adapters for general objects. Nevertheless, we hope our findings can inspire researchers in related areas to explore this direction further. For instance, training a background adapter (SR-BG) and incorporating it into object adapter training may help mitigate background and lighting leakage—potentially enhancing generalization in non-human scenarios.\n\n### **Q2**: Usage of FaceNet vs. ArcFace/AdaFace\nWe appreciate the suggestion to use ArcFace for identity evaluation. However, we note that competing baselines such as InifinteYou and PuLID use ArcFace as their image encoder.  To avoid coupling and biasing the evaluation, we chose FaceNet (i.e. an encoder not used by any of the methods) to ensure a neutral comparison.  FaceNet was also used as a standard metric in recent studies on face swap [Huang et al., 2024] (where face similarity is the focus). \n\n### **Q3**: Evaluation for identity preservation and expression control\nWe included LM-as-a-judge for several reasons.\n\nExisting encoder-based face identity metrics (e.g., FaceNet, AdaFace, ArcFace) are limited to evaluating only facial features. They fail to assess other critical aspects of appearance, such as head shape and hair. To overcome this limitation and achieve a more comprehensive identity evaluation, we adopted LM-as-a-judge.\n\nFurthermore, LM-as-a-judge has been previously employed in papers requiring custom metrics for controlled generation, as exemplified by Stylus [Luo et al., 2024], a work cited in our submission from the last NeurIPS.\n\nFor evaluating expression, using LM-as-a-judge followed a similar thought process.  Yet, we really appreciate the suggestion of using existing blendshape prediction models like EMOCA for evaluation.  We evaluated the similarity of expressions by inputting both prior and generated images into the EMOCA model. We then calculated the cosine similarity of their respective EMOCA outputs. Prior images were created using the base Flux model, without personalization. A higher cosine similarity indicates a greater resemblance in expressions. Ideally, a personalization adapter should achieve a high value, demonstrating that it does not leak expression from the input image.  Below are the table of results:\n\n|Methods | InfU | PuLID | IPA | SR-LoRA | SR-CN |\n|----------|----------|----------|----------|----------|----------|\n|EMOCA Sim.↑ | 0.542 | 0.489 | 0.347 | 0.458 | **0.580** |\n\n\n\nThe result of EMOCA similarity shows that IPA trained with SR-CN (i.e., factoring out the 2D keypoints) indeed achieves a higher similarity with the expression in the prior model. In terms of downstream impact, this indicates that such a model is more effective in adhering to the expression prompt, hence allowing users to create more diverse generations without being limited by a personalization adapter with expression leakage. \n\n### **Q4**: \"Evaluation set\". \n\nOur evaluation set includes 32 identities from real-world photos, encompassing both studio images and in-the-wild selfies, as well as 146 diverse prompts covering 5 different test cases (see Appendix G for details). We thank the reviewer for the suggestion to evaluate on cartoon or drawing-style inputs, which is out of our training distribution—we view this as a valuable and interesting direction for future work (to include stylized inputs in training).\n\nThat said, we have included 10 stylized prompts with cartoon/drawing aesthetics to validate the capacity in generating stylized images with photorealistic input faces. In these cases, our method (IPA with SR-CN) significantly outperforms both the baseline IPA (without SR-CN) and other state-of-the-art methods, InfU and PuLID. The baseline IPA consistently fails to produce stylized outputs, generating photorealistic images regardless of the prompt. While InfU and PuLID are capable of generating stylized images, they struggle to preserve identity. In contrast, our IPA with SR-CN successfully follows the stylized prompts and produces cartoonized images while maintaining identity fidelity. As images are not allowed in rebuttal, we conducted a user study in which participants were asked to select their favorite generation among all methods:\n\n| InfU | PuLID | IPA | **IPA w/ SR-CN** |\n|------|--------|-----|------------------|\n| 6.7% | 16.7%  | 6.7% | **73.3%**        |\n\n### **Q5**: “Qualitative results for the SR-LoRA method is not shown in neither the main paper nor the supplementary materials”\nWe would like to clarify that SR-LoRA qualitative results are included in the supplementary material (see Appendix Fig. 3, 4, and 5 with the column name “IPA w/ LoRA”).  We apologize if this was not sufficiently highlighted in the main text. We will rename it as “IPA w/ SR-LoRA (Ours)” and make clearer references in the revision to guide readers to these results.  \n\n### Additional Comment: Potential negative societal impact\nWe thank the reviewer for pointing out that there was no explicit discussion regarding the potential negative societal impact.  We will add the following paragraph in the revision: \n\n*As our method improves identity preservation and expression control in personalized generation, it naturally raises concerns about misuse, particularly in the creation of hyper-realistic synthetic identities or deepfakes. We acknowledge that enhanced controllability and realism may lower the barrier for malicious use. To mitigate such risks, we advocate for responsible deployment practices, such as model watermarking, usage restrictions, and alignment with ethical frameworks for generative media.*\n\n\nIn conclusion, we hope our responses have sufficiently addressed your questions and clarified the contributions and design choices of our work. We respectfully request you to consider updating your rating, as we believe the additional analyses and clarifications strengthen the case for acceptance. Should any further questions arise during the discussion period, we would be happy to engage in continued dialogue and refine our submission accordingly.\n\n\n### References\n- [Luo et al., 2024] Luo, Michael, et al. \"Stylus: Automatic adapter selection for diffusion models.\" Advances in Neural Information Processing Systems 37 (2024): 32888-32915.\n- [EMOCA] https://github.com/radekd91/emoca\n- [Huang et al. 2024]  Huang, Ziyao, Fan Tang, Yong Zhang, Juan Cao, Chengyu Li, Sheng Tang, Jintao Li, and Tong-Yee Lee. \"Identity-preserving face swapping via dual surrogate generative models.\" ACM Transactions on Graphics 43, no. 5 (2024): 1-19.\n- [InfU code reference of using ArcFace] https://github.com/bytedance/InfiniteYou/blob/5fff05b3cba6c119857d62d9cc27a90bcc501187/pipelines/pipeline_infu_flux.py#L269"}, {"review_id": "9lfTVVg7zy", "reviewer": "Reviewer_NBa4", "summary": "In adapter-based training for image generation models (e.g., LoRA, IP-Adapter), the single-image reconstruction objective causes adapters to overfit to confounding factors (e.g., pose, expression, lighting) in input images rather than the target attribute (e.g., subject identity). This entanglement compromises model modularity, preventing response to textual prompts (e.g., outputs retain training-image expressions despite requests to \"change the subject's expression\").", "questions": "Please see above", "limitations": "Please see above", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 1, "significance": 1, "originality": 1, "strengths_and_weaknesses": "Weakness\n\n* First of all, the title is very difficult to understand and fails to convey the core idea of the paper. It is unclear what “..well..” is supposed to mean.\n\n* The use of line breaks in the abstract is quite unconventional—I have never seen this formatting style before.\n\n* The introduction fails to provide a clear problem definition. The writing is semantically confusing and includes some bizarre expressions, such as the word “fig” in line 44.\n\n* The mathematical formulations are unnecessarily complex and lack clear meaning. Most of them are quite naïve and could have been simplified. Moreover, the proposed method does not effectively leverage or align with these definitions.\n\n* The methodological novelty is weak. The approach appears to be a straightforward combination of existing techniques, namely LoRA and ControlNet.\n\nI have serious concerns that this paper may have been generated entirely by an AI large model, starting with its problematic title which is fundamentally nonsensical. Furthermore, the entire manuscript exhibits disorganized descriptions and is riddled with errors, rendering it unacceptable as a substandard manuscript. Consequently, I strongly recommend rejection of this submission.", "was_revised": true, "final_justification": "Thank you for your reply. I have decided to increase my score, but at the same time, I hope you can handle the current issues in the final version with caution", "rebuttal": "We respectfully address the criticism raised by the reviewer.\n\n\n1. “Title” – We acknowledge that the title can appear informal. It was a very deliberate decision, and understandably risky. Our intent was to use a conversational tone to highlight the counterintuitive insight that “providing the shortcuts” can prevent shortcut learning.  We are happy to discuss the possibility of revising the title, and removing the “ .. well ..”.   \n2. “Format error and typo” – Thanks for pointing out the two syntax errors.  We will remove the line break in the abstract, and remove the straying “fig” on page 2. \n3. “Math” – Our formalism aims to outline the core idea of shortcut rerouting in adapter training – by explicitly routing confounders through separate modules, we change the optimization path and effectively guide the adapter toward disentangled representations.\n4. “.. approach appears to be a straightforward combination of existing techniques, namely LoRA and ControlNet” – We would like to highlight that our contribution is the “shortcut rerouting” training scheme, and the use of LoRA and ControlNet are just to provide the shortcut.  We use these existing tools in very unintended (and novel) ways.  In fact we argue that using well-established tools as part of our method adds to the usability of the method, not diminishes it. We respectfully refer the reviewer to Strength 2 highlighted by reviewer fa9d, which emphasizes the advantage of leveraging mature tools to enable shortcut rerouting.\n5. “have been generated entirely by an AI large model” – We are disheartened to see such a comment directed towards our project. First, as indicated in our authors’ checklist, we acknowledge using LLMs for editing our paper.  Second, if the paper were **entirely** generated using an LLM, it would likely pick a much more conventional title and not cause the concern you had."}, {"review_id": "7OJjJG5fky", "reviewer": "Reviewer_8c2C", "summary": "This paper proposes a new training method for adapters that aims to disentangle confounding factors that may be present in the input by explicitly providing shortcuts for them. For example, if one had to train an adapter for only facial identity transfer this paper proposes adding additional modules to the base model (via LoRA or a ControlNet) to regress other features corresponding to other factors such as pose, background features, facial expression etc. The paper claims that providing such “shortcuts” helps the target adapter to only focus on the desired attribute (in this case facial identity) and ignore the confounding factors (the aforementioned attributes). As a consequence, during inference all other confounding adapters can be dropped and the only the trained adapter can be used to transfer the desired attribute. The paper tests the proposed method on two target attributes Facial Identity Transfer, and Body Transfer. Both qualitative and quantitative results demonstrate the efficacy of this method. However, more attributes need to be tested as well for example, the body pose where the person identity is a shortcut.", "questions": "I would strongly encourage the authors to include an ablation of the results wrt the relative representational strength allocated to the shortcut", "limitations": "Test cases are too limited", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is well written and easy to follow\n\n2. The Idea is very intuitive and well motivated\n\n3. For two specific attributes the paper presents ample evidence, \n\nWeaknesses\n\n1. Only two attributes tested", "was_revised": true, "final_justification": "The authors addresed my concers clearly by the lighting shortcut experiment. Looking at the other reviews and rebuttal I am convinced that this method provides usefull insights into conditional large scale foundation models in a modality specific manner and will be very valuable to the research community in general.", "rebuttal": "We thank the reviewer for the encouraging feedback and thoughtful comments. Below, we address the main concern raised:\n\n### Limited number of tested attributes\nWe agree that expanding the range of evaluated attributes would further strengthen our claims. \n\nWe investigated specific shortcuts due to their practical implications. During IPA development and testing, we observed that generated images often failed to align with the prompt, as pose and expression from the input image \"leaked\" through (i.e., pose/expression shortcuts). Additionally, the aesthetic quality of the trained model deteriorated compared to the base FLUX.1-Dev model (i.e., distribution shift). These observations motivated our study of these attributes.\n\nMotivated by the suggestion to test SR on additional attributes and the practical observation that lighting often \"leaks\" from the input image, we conducted a new experiment. When prompted to generate an image with artistic lighting, this effect is frequently diminished in the personalized model, as the input image typically features more ordinary, everyday lighting. Based on this, we experimented with shortcut rerouting for lighting.\n\nTo establish a lighting shortcut, we trained an IPA where the input image was the target image with the human masked out. This adapter is designed to encode all target image information except the human. We then used this adapter as a shortcut to train our personalization adapter (**IPA w/ SR-LG**). Our preliminary results indicate that this mechanism effectively factors out lighting and improves adherence to stylistic prompts during generation. Due to the inability to include qualitative results in the rebuttal, we used the LPIPS between the generated and prior images as a proxy for lighting preservation. Additionally, we conducted a user study across all testing prompts where raters viewed the prompt, prior image, and a generated image, and answered: *“Is the lighting harmonized with the prompt?”*\n\n| Methods       | LPIPS ↓  | User Ratings ↑ |\n| ------------- |:---------:|:--------------:|\n| IPA           | 0.433     | 33.6%        |\n| IPA w/ SR-BG  | 0.344  | 93.2%            |\n\nThese results demonstrate that SR effectively factors out attributes such as \"lighting\" and improves adherence to the prior. We will include this new experiment and its qualitative results in the revision.\n\nIn conclusion, we hope our responses have sufficiently addressed concerns.  We respectfully request you to consider updating your rating, as we believe the additional analyses and clarifications strengthen the case for acceptance. Should any further questions arise during the discussion period, we would be happy to engage in continued dialogue and refine our submission accordingly."}, {"review_id": "gBp5l8uubY", "reviewer": "Reviewer_fa9d", "summary": "Aiming at the \"shortcut learning\" problem of adapter training in text-to-image (T2i) generation, this paper proposes a shortcut rerouted adapter training method. The traditional adapter training is easy to entangle the target attributes (such as identity) with the hybrid factors, such as posture and expression, resulting in the generated results being unable to respond to the prompt words. By introducing auxiliary modules (such as ControlNet or LoRA) to explicitly divert the confounding factors in training, the author forces the adapter to learn only the target attributes. Experiments show that this method significantly improves the controllability of posture and expression and the quality of generation in face and whole body personalized generation tasks, which is superior to baseline methods such as InfU and PuLID.", "questions": "1. Does the SR-IPA mentioned in this paper have this decoupling ability for any input image after training, or does it simply need to be trained for each case, which means that each case corresponds to a specific SR-IPA weight?\n2. How to define these confounding attributes? So far, this paper focuses on the decoupling of face identity information and its pose or background. Is there any other decoupling ability? For example, for face only, can we decouple makeup, hair color/style, glasses, and other attributes?\n3. This paper uses a text-to-image generation model. How to input the prior examples given in this paper? Is it to directly use some existing tools, such as ip-adapter, to encode and input images? Or enter the text prompt of the corresponding image? If a text prompt is entered, how can to ensure the consistency with the prior image?\n4. In the experiment section, it is declared that a dataset has been collected. Will the dataset be open-accessed?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. A counterintuitive but effective solution is proposed: by actively providing a \"shortcut\" to eliminate the dependence of the adapter on confounding factors, the theoretical framework is clear (formal problem of probability modeling), and there is no need to modify the basic model.\n2. Combining Lora and ControlNet, two mature tools to realize shortcut rerouting, it has both practicality and scalability.\n\nWeaknesses:\n1. The method relies on specific tools such as ControlNet or LoRA, and does not verify the applicability of other confounding factors (such as background style, lighting), or whether it applies to more complex scenes.\n2. From the qualitative experiments shown, it seems that the improvement is limited.", "was_revised": true, "final_justification": "Thank you very much for the detailed response, which effectively resolved my question. I am very pleased with the contribution of this work to the community and look forward to the author's future work with stronger attribute control abilities. Considering the author's response and the opinions of other reviewers, I have decided to increase my rating.", "rebuttal": "We thank reviewer fa9d for their constructive comments and feedback. We are delighted that they found our solution \"counterintuitive but effective,\" which precisely reflects the sentiment we aimed to convey with our title.\n\n### **Weakness 1**: Other confounding factors like “background” and “lighting”, or “complex scenes”\nMotivated by the suggestion to test SR on additional attributes and the practical observation that lighting often \"leaks\" from the input image, we conducted a new experiment. When prompted to generate an image with artistic lighting, this effect is frequently diminished in the personalized model, as the input image typically features more ordinary, everyday lighting. Based on this, we experimented with shortcut rerouting for lighting.  \n\nTo establish a lighting shortcut, we trained an IPA where the input image was the target image with the human masked out. This adapter is designed to encode all target image information except the human. We then used this adapter as a shortcut to train our personalization adapter (IPA w/ SR-LG). Our preliminary results indicate that this mechanism effectively factors out lighting and improves adherence to stylistic prompts during generation. Due to the inability to include qualitative results in the rebuttal, we used the LPIPS between the generated and prior images as a proxy for lighting preservation. Additionally, we conducted a user study across all testing prompts where raters viewed the prompt, prior image, and a generated image, and answered: *“Is the lighting harmonized with the prompt?”*\n\n| Methods       | LPIPS ↓   | User Ratings ↑ |\n| ------------- |:---------:|:--------------:|\n| IPA           | 0.433     | 33.6%        |\n| IPA w/ SR-LG  | 0.344  | 93.2%            |\n\nThese results demonstrate that SR effectively factors out attributes such as \"lighting\" and improves adherence to the prior. We will include this new experiment and its qualitative results in the revision.\n\n### **Weakness 2**: Limited improvements in qualitative results\nOur primary contribution is \"shortcut rerouting training.\" We aim to validate its effectiveness in eliminating shortcuts during inference, specifically \"expression,\" \"pose,\" and \"distribution shift.\" Figures 5 and 6 qualitatively demonstrate these improvements, showing that SR-CN results more closely resemble \"Prior\" generations than baselines. \"Prior\" generations were produced by the same T2I model without the adapter, using identical prompts and seeds. An ideal adapter, free of shortcuts, should optimally maintain the pose and structure of this \"prior\" image.  We will make this point more prominent in the revised writing. \n\n### **Q1**: Does the SR-IPA mentioned in this paper have this decoupling ability for any input image after training, or does it simply need to be trained for each case, which means that each case corresponds to a specific SR-IPA weight?\nOnce trained, the SR-IPA operates in a feedforward manner and generalizes to unseen input identities without requiring retraining. The adapter is trained to factor out specific shortcut cues (e.g., lighting, background, pose, expression), and this decoupling behavior holds for new test images as long as the shortcut definition remains the same. If users wish to factor out a different type of shortcut, a new adapter must be trained accordingly. In other words, the decoupling generalizes across identities, but not across shortcut types.\n### **Q2**: How to define these confounding attributes? So far, this paper focuses on the decoupling of face identity information and its pose or background. Is there any other decoupling ability? For example, for face only, can we decouple makeup, hair color/style, glasses, and other attributes?\nWe came to the choice of these confounding attributes–“expression”, “pose” and “distribution shift”--from practically trying to build a useful appearance adapter.  We seek a “face” (or “fullbody”) adapter that can generate anybody in any imaginary scene.  Yet, in practice, we found that existing adapters failed to control the character’s expression through prompt, and failed to adapt the characters' pose through prompt.  These severely limited the kind of images we can generate.  \nYour suggestion of decoupling makeup, hair, and accessories sounds interesting!  “Hair” and “accessories” can likely be masked out by segmentation masks to a large extent.  “Makeup” sounds like a good attribute to factor out given a dataset that isolates “makeup”.  \n\n### **Q3**: This paper uses a text-to-image generation model. How to input the prior examples given in this paper? Is it to directly use some existing tools, such as ip-adapter, to encode and input images? Or enter the text prompt of the corresponding image? If a text prompt is entered, how can to ensure the consistency with the prior image?\n\nThe prior images in the paper are generated using the same base text-to-image model (FLUX.1 Dev) with the same prompt and a fixed random seed— just without using any image encoder or IP-Adapter. These \"prior images\" are not used as inputs to our method. Instead, they serve as a reference to visualize what the model would produce without any adapter conditioning. Their purpose is to illustrate how much the adapter alters the layout, lighting, pose, or expression. An ideal \"appearance\" adapter should preserve these attributes (e.g. layout of IPA generation should remain same as in prior image). As shown in Figures 5 and 6, our method produces images that stay more faithful to the prior compared to baseline methods.\n\n### **Q4**: In the experiment section, it is declared that a dataset has been collected. Will the dataset be open-accessed?\nWe are currently exploring options for responsibly sharing the dataset. In the meantime, we emphasize that our contribution, Shortcut Rerouting training, is simple in both concept and implementation. To support reproducibility, we provide detailed methodological descriptions and include additional experiments that demonstrate the generality of our approach. Given the minimal requirements of our method, we anticipate that it can be readily reproduced using similar data.\n\nWe believe our responses have addressed your concerns and clarified any confusion. We respectfully request you to consider updating your rating, as the additional analyses and clarifications strengthen the case for acceptance. We are available for further dialogue and to refine our submission should any questions arise during the discussion period."}]

Author rebuttal:

Meta-review: This paper received overall positive reviews. Reviewers appreciated the interesting idea, solid results, and the enhanced clarity provided in the rebuttal. The AC concurred with the reviewers' consensus and considers this a solid contribution worthy of acceptance. Please integrate the promised improvement into the camera ready, including title. 

However, there is room for improvement. The paper would benefit from deeper analysis to draw more insightful conclusions and from broadening its scope beyond facial-related text-to-image generation. Currently, the method for decoupling attributes in generic images (beyond faces) remains unclear. Additional research and development are needed to strengthen the impact of this work, potentially making it suitable for spotlight or oral presentations.
Decision: Accept (poster)