Ar2Can: An Architect and an Artist Leveraging a Canvas for Multi-Human Generation

Ar2Can is a two-stage LoRA adapter pipeline for multi-reference face composition in multi-human image generation. It uses two adapters: an Architect (on FLUX.1-schnell) for scene layout, and an Artist (on FLUX.1-Kontext-dev) for identity-preserving rendering.

---

Abstract

Despite recent advances in personalized image generation, existing models consistently fail to produce reliable multi-human scenes, often merging or losing facial identity. We present Ar2Can, a novel two-stage framework that disentangles spatial planning from identity rendering for multi-human generation. The Architect predicts structured layouts, specifying where each person should appear. The Artist then synthesizes photorealistic images, guided by a spatially-grounded face matching reward that combines Hungarian spatial alignment with identity similarity. This approach ensures faces are rendered at correct locations and faithfully preserve reference identities. We develop two Architect variants, seamlessly integrated with our diffusion-based Artist model. This is optimized via Group Relative Policy Optimization (GRPO) using compositional rewards for count accuracy, image quality, and identity matching. Evaluated on the MultiHuman-Testbench, Ar2Can achieves substantial improvements in both count accuracy and identity preservation, while maintaining high perceptual quality. Notably, our method achieves these results using primarily synthetic data, without requiring real multi-human images.

---

Model Details

This repo contains two LoRA adapters in separate subdirectories:

Adapter	Subdirectory	Base model	Purpose
Architect	architect/	black-forest-labs/FLUX.1-schnell	Generates scene layout / probe image
Artist	artist/	black-forest-labs/FLUX.1-Kontext-dev	Renders final identity-preserving composition

Both adapters share the same LoRA configuration:

Field	Value
Adapter type	LoRA (PEFT)
LoRA rank (r)	64
LoRA alpha	128
Dropout	0.0
Weight init	Gaussian

Architect target modules: attn.to_q, attn.to_k, attn.to_v, attn.to_out.0, attn.to_add_out, attn.add_q_proj, attn.add_k_proj, attn.add_v_proj, ff.net.0.proj, ff.net.2, ff_context.net.0.proj, ff_context.net.2

Artist target modules: attn.to_q, attn.to_k, attn.to_v, attn.to_out.0, attn.to_add_out, attn.add_q_proj, attn.add_k_proj, attn.add_v_proj, ff.net.0.proj, ff.net.2, ff_context.net.0.proj, ff_context.net.2, proj_mlp

---

Usage

Option 1: Ar2Can inference repo (recommended)

Clone the repo — weights are pulled automatically from HuggingFace on first run:

git clone https://github.com/Qualcomm-AI-research/ar2can.git
cd ar2can

Run inference:

python infer.py \
  --faces faces/person_1.webp faces/person_2.webp faces/person_3.webp \
  --prompt "Three people standing side-by-side in a winter landscape, 8K, ultra-realistic, preserve exact facial identity."

Or launch the Gradio demo:

python app.py
# open http://localhost:7865

If you prefer locally downloaded weights (e.g. from GitHub Releases), place them at loras/architect/adapter_model.safetensors and loras/artist/adapter_model.safetensors — the code will use the local copies automatically.

Recommended prompts

Two people on a sunny beach, ultra-realistic, 8K, preserve exact facial identity.
Three people standing side-by-side in a winter landscape, realistic light rendering, preserve exact facial identity.
Four people in a city plaza at midday, high fidelity, preserve exact facial identity.

---

Hardware Requirements

• GPU: 2× NVIDIA A100 40 GB (or equivalent, ≥ 70 GB total VRAM for both models)
• CUDA: 12.4
• Python: 3.11
• PyTorch: 2.6.0

---

Training Details

Ar2Can is trained with Flow-GRPO, a flow-matching adaptation of Group Relative Policy Optimization (GRPO):

• Architect base model: FLUX.1-schnell
• Artist base model: FLUX.1-Kontext-dev
• Training algorithm: Flow-GRPO (RL via composite reward signal)
• Reward components:
  • Spatially-grounded face matching (Hungarian alignment + ArcFace identity similarity)
  • Person count accuracy
  • HPS image quality score
• Primarily synthetic training data — no real multi-human images required

---

Citation

@inproceedings{borse2026ar2can,
  title={Ar2can: An architect and an artist leveraging a canvas for multi-human generation},
  author={Borse, Shubhankar and Pham, Phuc and Farhadzadeh, Farzad and Choi, Seokeon and Nguyen, Phong and Tran, Anh and Yun, Sungrack and Hayat, Munawar and Porikli, Fatih},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={550--560},
  year={2026}
}

---

Notes

Disclaimer: The model weights released here are not identical to the model reported in the paper. They are provided for research and demonstration purposes only. Performance may vary from the results reported in the paper.

---

License

The Architect LoRA weights are derived from FLUX.1-schnell, which is released under the Apache 2.0 License.

The Artist LoRA weights are derived from FLUX.1-Kontext-dev, which is released under a Non-commercial License. Any use of the Artist LoRA weights must comply with those Non-commercial License terms.

Copyright (c) Qualcomm Technologies, Inc. and/or its subsidiaries.