Title: Preventing Shortcuts in Adapter Training via Providing the Shortcuts
Venue: NeurIPS 2025
Accepted: yes
Keywords: text to image generation

adapter

personalized generation

controllable generation

spurrious correlation

shortcut learning
Abstract: Adapter-based training has emerged as a key mechanism for extending the capabilities of powerful foundation image generators, enabling personalized and stylized text-to-image synthesis. These adapters are typically trained to capture a specific target attribute, such as subject identity, using single-image reconstruction objectives. However, because the input image inevitably contains a mixture of visual factors, adapters are prone to entangle the target attribute with incidental ones, such as pose, expression, and lighting. This spurious correlation problem limits generalization and obstructs the model's ability to adhere to the input text prompt.  In this work, we uncover a simple yet effective solution: provide the very shortcuts we wish to eliminate during adapter training. In Shortcut-Rerouted Adapter Training, confounding factors are routed through auxiliary modules, such as ControlNet or LoRA, eliminating the incentive for the adapter to internalize them. The auxiliary modules are then removed during inference. When applied to tasks like facial and full-body identity injection, our approach improves generation quality, diversity, and prompt adherence. These results point to a general design principle in the era of large models: when seeking disentangled representations, the most effective path may be to establish shortcuts for what should NOT be learned.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAiming at the \"shortcut learning\" problem of adapter training in text-to-image (T2i) generation, this paper proposes a shortcut rerouted adapter training method. The traditional adapter training is easy to entangle the target attributes (such as identity) with the hybrid factors, such as posture and expression, resulting in the generated results being unable to respond to the prompt words. By introducing auxiliary modules (such as ControlNet or LoRA) to explicitly divert the confounding factors in training, the author forces the adapter to learn only the target attributes. Experiments show that this method significantly improves the controllability of posture and expression and the quality of generation in face and whole body personalized generation tasks, which is superior to baseline methods such as InfU and PuLID.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. A counterintuitive but effective solution is proposed: by actively providing a \"shortcut\" to eliminate the dependence of the adapter on confounding factors, the theoretical framework is clear (formal problem of probability modeling), and there is no need to modify the basic model.\n2. Combining Lora and ControlNet, two mature tools to realize shortcut rerouting, it has both practicality and scalability.\n\nWeaknesses:\n1. The method relies on specific tools such as ControlNet or LoRA, and does not verify the applicability of other confounding factors (such as background style, lighting), or whether it applies to more complex scenes.\n2. From the qualitative experiments shown, it seems that the improvement is limited.\n\nQUESTIONS:\n1. Does the SR-IPA mentioned in this paper have this decoupling ability for any input image after training, or does it simply need to be trained for each case, which means that each case corresponds to a specific SR-IPA weight?\n2. How to define these confounding attributes? So far, this paper focuses on the decoupling of face identity information and its pose or background. Is there any other decoupling ability? For example, for face only, can we decouple makeup, hair color/style, glasses, and other attributes?\n3. This paper uses a text-to-image generation model. How to input the prior examples given in this paper? Is it to directly use some existing tools, such as ip-adapter, to encode and input images? Or enter the text prompt of the corresponding image? If a text prompt is entered, how can to ensure the consistency with the prior image?\n4. In the experiment section, it is declared that a dataset has been collected. Will the dataset be open-accessed?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

comments: SUMMARY:
This paper proposes a new training method for adapters that aims to disentangle confounding factors that may be present in the input by explicitly providing shortcuts for them. For example, if one had to train an adapter for only facial identity transfer this paper proposes adding additional modules to the base model (via LoRA or a ControlNet) to regress other features corresponding to other factors such as pose, background features, facial expression etc. The paper claims that providing such “shortcuts” helps the target adapter to only focus on the desired attribute (in this case facial identity) and ignore the confounding factors (the aforementioned attributes). As a consequence, during inference all other confounding adapters can be dropped and the only the trained adapter can be used to transfer the desired attribute. The paper tests the proposed method on two target attributes Facial Identity Transfer, and Body Transfer. Both qualitative and quantitative results demonstrate the efficacy of this method. However, more attributes need to be tested as well for example, the body pose where the person identity is a shortcut.

STRENGTHS AND WEAKNESSES:
Strengths:

1. The paper is well written and easy to follow

2. The Idea is very intuitive and well motivated

3. For two specific attributes the paper presents ample evidence, 

Weaknesses

1. Only two attributes tested

QUESTIONS:
I would strongly encourage the authors to include an ablation of the results wrt the relative representational strength allocated to the shortcut

LIMITATIONS:
Test cases are too limited
TITLE: Official Review
CLARITY: 4
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn adapter-based training for image generation models (e.g., LoRA, IP-Adapter), the single-image reconstruction objective causes adapters to overfit to confounding factors (e.g., pose, expression, lighting) in input images rather than the target attribute (e.g., subject identity). This entanglement compromises model modularity, preventing response to textual prompts (e.g., outputs retain training-image expressions despite requests to \"change the subject's expression\").\n\nSTRENGTHS AND WEAKNESSES:\nWeakness\n\n* First of all, the title is very difficult to understand and fails to convey the core idea of the paper. It is unclear what “..well..” is supposed to mean.\n\n* The use of line breaks in the abstract is quite unconventional—I have never seen this formatting style before.\n\n* The introduction fails to provide a clear problem definition. The writing is semantically confusing and includes some bizarre expressions, such as the word “fig” in line 44.\n\n* The mathematical formulations are unnecessarily complex and lack clear meaning. Most of them are quite naïve and could have been simplified. Moreover, the proposed method does not effectively leverage or align with these definitions.\n\n* The methodological novelty is weak. The approach appears to be a straightforward combination of existing techniques, namely LoRA and ControlNet.\n\nI have serious concerns that this paper may have been generated entirely by an AI large model, starting with its problematic title which is fundamentally nonsensical. Furthermore, the entire manuscript exhibits disorganized descriptions and is riddled with errors, rendering it unacceptable as a substandard manuscript. Consequently, I strongly recommend rejection of this submission.\n\nQUESTIONS:\nPlease see above\n\nLIMITATIONS:\nPlease see above", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 1, "IMPACT": 1, "ORIGINALITY": 1, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper concerns the training of \"adapters\" for text-to-image models. Specifically, the paper discusses the cases where the adapter is supposed to capture the identity or unique look of a reference input image. The proposed SR method (Shortcut-Rerouting) suggests that existing adapter training methods result in confounding factors being transferred from the reference input image to the generated image and that to avoid this, extraneous factors should simply be provided directly as a condition (provide the shortcut). The authors evaluate identity consistency, expression alignment, head/body-pose alignment and other metrics in comparison to existing adapter methods such as IPA and show that their proposed modification (SR-LoRA or SR-CN) result in improvements across the board. The qualitative results show better adherence to both the identity contained in the reference input image as well as better adherence to the content of the prior image (e.g. lighting, pose, expression).\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is written well with clear figures that are easy to understand. The intuition for learning shortcuts (or confounding factors directly) is explained very well in Section 3.1. The overall concept can be easily understood and the method is presented with sufficient details to allow reproduction. The qualitative results are particularly impressive, showing that the proposed modifications can drastically improve identity consistency and adherence to details of the prior image such as pose/expression/etc.\n\nA few questions do arise, however. For instance, the authors provide keypoints as explicit \"shortcut\" signals during training, but do not discuss the possibility of using other specialist encoders such as those for expression, body shape, attributes (e.g. CelebA). This leaves the user feeling that the overall \"shortcuts\" and \"shortcut-rerouting\" idea sounds great, but may only apply to specific cases such as pose / landmarks, without being very general. Furthermore, how can this idea of shortcuts be applied to T2I tasks for non-human subjects?\n\nIn addition, the metrics used in Sec. 4.1 are mostly sensible, but the Expr. metric in particular is using an LLM-as-a-judge even though encoders such as EMOCA exist for directly predicting expression parameters. The models used are generally dated, especially because identity is determined using FaceNet instead of the more widely used ArcFace or AdaFace models.\n\nIn the quantitative evaluations, the authors do not mention which dataset they use. It appears as though they are using a single internal dataset, which may not allow generalizable insights. As a potential improvement, the authors could consider evaluating on datasets of sufficiently different characteristics, e.g. studio images, in-the-wild, cartoon/drawings.\n\nQUESTIONS:\n1. This paper only discusses the potential of learning pose/keypoint related shortcuts using LoRA/ControlNet. Did you consider learning other shortcuts using existing encoders? E.g. expression parameters from EMOCA, CelebA attributes, head-pose from HopeNet.\n2. Did you consider using an ArcFace/AdaFace model to compute identity consistency as is common in face avatar research?\n3. Is an LLM a good judge of identity or facial expressions?\n4. Qualitative results for the SR-LoRA method is not shown in neither the main paper nor the supplementary materials. Is there any particular reason for this?\n\nLIMITATIONS:\nThe authors briefly discuss the limitation of their work (adapter-focused training, not LoRA), but do not discuss any potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}