baki0115/gaze-controlnet-qwen-image-edit

GazeCtrl — a gaze-conditioned ControlNet for Qwen-Image-Edit-2509

Turn a person in a photo to face a target gaze direction (dx, dy, dz) — head and body, over the full 360° including facing away — while keeping the same identity.

This is a ControlNet adapter, not a standalone model. It does nothing without the frozen Qwen-Image-Edit-2509 backbone and its VAE. The loading code is in the repository below; these weights are not diffusers-loadable as-is.

• Code, training recipe, evaluation: https://github.com/VicsonPeng/GazeCtrl_training-based
• Labels used to train it: https://huggingface.co/datasets/baki0115/GazeCtrl_dataset

How it works

        target gaze (dx,dy,dz)
                 │
                 ▼
   flat RGB field   R=(dx+1)/2  G=(dy+1)/2  B=(dz+1)/2      512×512
                 │
              VAE encode
                 │
noisy latent ─► InstantX ControlNet Union (5 blocks) ── 5 residuals ─┐
                                                                     ▼
source image ──────────────────────► Qwen-Image-Edit-2509 fp8 (frozen) ──► output
   (in-context reference token — this is what carries identity)

The gaze vector is the control signal: no skeleton, no pose-prediction stage, and no text prompt (txt_emb is a zero tensor, so every bit of control arrives through the ControlNet). Identity is carried by the source image entering the backbone as an in-context reference, not by the ControlNet.

Convention — dx > 0 looks image-right, dy > 0 looks up, dz > 0 looks toward the camera. yaw = atan2(dx, dz), so ±180° means facing away.

Files

file	steps	note
cn_step085000.safetensors	85 000	best training-set metrics; the one used for the demo
cn_step090000.safetensors	90 000	final checkpoint of the run

3.54 GB each, bf16, 1768 M parameters.

Usage

import torch
from safetensors.torch import load_file
from qwen_models import (QwenBackbone, QwenControlNet, load_qwen_vae, vae_encode, vae_decode,
                         patchify, unpatchify, compute_rope_freqs_3d, compute_text_rope_freqs)

dev = "cuda"
vae = load_qwen_vae(VAE_PATH, device=dev)
bb  = QwenBackbone(QWEN_IMAGE_EDIT_2509_FP8_PATH, device=dev)   # frozen
cn  = QwenControlNet(); cn.load_state_dict(load_file("cn_step085000.safetensors"))
cn.to(dev, torch.bfloat16).eval()

The full denoising loop — flow matching, 16 steps, cn_scale=1.0 to match training — is demo/app.py in the code repository. cn_scale is the knob that trades obedience against fidelity; values away from 1.0 change the reported metrics.

Training

data	1864 frames / 183 identities (COCO + WiderFace → Wan2.2 turnarounds), 48 % human-labelled
init	InstantX ControlNet Union pretrained weights (181/181 keys)
objective	flow-matching velocity MSE, head-weighted w = 1 + 8·head_mask + 30·eye_mask
optimiser	AdamW, lr 1e-4, grad-clip 1.0, batch 1
schedule	90 000 steps, ~2 s/step on one A6000

Evaluation

	training identities	held-out 70k	held-out 90k
gaze_err ↓	19.11°	55.97°	57.28°
ctrl_slope (1.0 ideal)	+0.874	+0.642	+0.667
id_sim (ArcFace) ↑	—	+0.277	+0.282
face_det	—	70 %	75 %

Held-out = 8 identities that were never trained on, cn_scale=1.0, 16 steps, seed 0.

Limitations — please read these

• The generalisation gap is large: 19° on training identities against ~56° on held-out ones. The held-out split is small (8 identities) and carries only automatic labels while 48 % of training labels are human, so some of that gap is label noise — but not all of it.
• Training longer stopped helping. 70k → 90k improved the training-set metric and did nothing for held-out performance.
• Identity preservation is marginal. id_sim ≈ 0.28 sits on ArcFace's own same-person threshold, and no loss term currently targets identity. Expect drift, worst at large rotations.
• gaze_err is a proxy. It scores head pose (6DRepNet) against the requested gaze vector. Head pose and gaze differ by construction — the two estimators that produced the labels disagree by a median 26.7° on the same frame — so a floor of that order is expected.
• Back-facing outputs cannot be scored for identity — ArcFace needs a face, hence the separate face_det column.
• Trained on 512×512 pad-to-square inputs, single person, roughly upright. Crowds, heavy occlusion and unusual framing are out of distribution.
• Label semantics are not uniform across directions — see the dataset card; this is the clearest known defect in the training signal.

Licence

Apache-2.0 for these weights. The base model (Qwen/Qwen-Image-Edit-2509) and the InstantX ControlNet Union it was initialised from keep their own licences; this adapter is useless without the former, so its terms govern any deployment.