DeltaRegion: text-guided, region-disentangled FLAME shape editing

DeltaRegion edits the identity shape of a FLAME 2020 head model from a short natural-language prompt. You give it a source identity β₀ ∈ ℝ³⁰⁰ and a prompt such as "make the nose long". It returns an edited identity β₁ = β₀ + Δβ in a single deterministic forward pass. The change stays inside the facial region named in the prompt (face, nose, lips, ears, neck or eyes).

DeltaRegion is the shape-editing stage of ManifoldSplat: Language-Guided Semantic Shape Editing of 3D Gaussian Head Avatars (GCPR 2026). There, the edited FLAME identity drives the refinement of an animatable 3D Gaussian Splatting avatar.

Project page: https://a-canela.github.io/manifoldsplat/

Model details

Property Value
Authors Antonio Canela, Jordi Sànchez-Riera (UPC, IRI CSIC-UPC)
Model type Conditional VAE (text + shape → FLAME shape delta) with a per-region gated latent
Input English prompt + FLAME 2020 shape coefficients (300-D)
Output Edited FLAME 2020 shape coefficients (300-D) and region probabilities
Parameters 348.0 M in total: 341.2 M frozen Flan-T5-Large encoder + 6.8 M trained DeltaRegion heads
Weights model.safetensors (fp32, 1.39 GB; the T5 encoder is bundled, so nothing else is downloaded)
License CC BY-NC 4.0 (non-commercial; see License)

Architecture

  1. Text encoder. The prompt goes through a frozen Flan-T5-Large encoder (1024-D tokens, max 128 tokens). It also produces a masked-mean sentence embedding.
  2. Shape encoder. An MLP ([512, 512], LayerNorm, GELU) embeds β₀ into a single 512-D token.
  3. Cross-attention fusion. The shape token is the query and the projected text tokens are the keys/values (8 heads). A residual connection and LayerNorm follow.
  4. Region gate. A softmax over the 6 regions, predicted from the sentence embedding alone.
  5. Prior / posterior. Diagonal Gaussians over z ∈ ℝ^(6×8), one 8-D block per region. The posterior is used only during training.
  6. Decoder. Each latent block is multiplied by its gate value, concatenated with the shape embedding and passed through an MLP ([768, 512, 512]), which outputs Δβ ∈ ℝ³⁰⁰.

At inference the posterior is dropped and z = s · μprior with s = 1.2 (config.inference_prior_scale). You can change s (the prior_scale argument) to make the edit weaker or stronger.

How to use

pip install torch transformers safetensors sentencepiece
import torch
from transformers import AutoModel

# revision pins the release, so the downloaded weights and model code cannot change later
model = AutoModel.from_pretrained("a-canela/DeltaRegion", revision="v1.0", trust_remote_code=True).eval()

# 1) Edit the neutral (mean) FLAME identity
out = model.edit("make the nose long")
edited_betas = out.pred_shape            # (1, 300) FLAME 2020 shape coefficients
print(model.region_probabilities(out.gate))  # [{'face': ..., 'nose': 0.99, ...}]

# 2) Edit your own identity (e.g. FLAME shape from MICA / a face tracker)
source_betas = torch.zeros(300)          # replace with your subject's 300 shape coefficients
out = model.edit(
    ["make the lips thick", "make the ears very protruding"],
    source_shape=source_betas,           # (300,), (1, 300) or (B, 300)
)

# 3) Control the edit strength continuously
weaker = model.edit("make the nose long", prior_scale=0.6)
stronger = model.edit("make the nose long", prior_scale=1.8)

You can also tokenize the prompt yourself and call the forward pass directly. Pad to 128 tokens to reproduce the training inputs exactly:

from transformers import T5TokenizerFast

tokenizer = T5TokenizerFast.from_pretrained("a-canela/DeltaRegion", revision="v1.0")
tokens = tokenizer(["make the eyes wide-set"], padding="max_length", max_length=128,
                   truncation=True, return_tensors="pt")
with torch.no_grad():
    out = model(tokens["input_ids"], tokens["attention_mask"], torch.zeros(1, 300))

The output pred_shape holds FLAME 2020 identity coefficients. Expression, pose, eyelids and texture are not modified. To get a mesh, feed the coefficients to a FLAME 2020 layer. The FLAME model files are not included here: register and download them from https://flame.is.tue.mpg.de/. FLAME fits with fewer than 300 shape coefficients should be zero-padded to 300.

Prompting

The model learned prompts of the form "make the {region} {edit}", plus a few template variants such as "give the {region} a {qualifier} {edit} shape" or "change the {region} to be {qualifier} {edit}". Name the region explicitly: the gate uses it to decide where the edit goes.

Training covered three edit magnitudes, each tied to a set of qualifiers:

Training scale Qualifiers
0.5× subtly, slightly, a little, a bit, somewhat
1.0× (no qualifier)
1.5× very, significantly, much more
Edit prototypes seen in training (129 in total)
  • face (25): big, small, wide, narrow, long, short, big forehead, small forehead, with chubby cheeks, with lean cheeks, with pronounced cheekbones, with a prominent / recessed / long / short / wide / narrow / cleft / pointed chin, with a wide / narrow / squared / rounded jaw, with the jaw deviated to the left / right
  • nose (43): small, big, narrow, wide, short, long, narrow / wide / protruding / recessed at the radix, narrow / thick / protruding / recessed / deviated left / deviated right at the bridge, with a downturned / upturned / recessed / protruding tip, small / big / narrow / wide / deviated left / deviated right at the base, with downturned / upturned / flared / narrow nostrils, aquiline, hawk-like, Greek, flat, saddle-shaped, wavy in profile, linearly deviated to the left / right, C-shaped crooked to the left / right, S-shaped crooked to the left / right, and one composite prototype that will be removed in the next release
  • lips (26): small, big, wide, narrow, with a thin / thick upper lip, with a thin / thick lower lip, with thin / thick lips, protruding, recessed, with a protruding / recessed lower / upper lip, with upturned / downturned corners, with the left / right corner raised / lowered, horizontally / vertically left / right asymmetric
  • ears (14): small, big, short, long, narrow, wide, attached lobe, detached lobe, protruding, flat, pointed top, rounded top, left protruding, right protruding
  • neck (6): thick, thin, long, short, with double chin, with prominent adam's apple
  • eyes (15): small, big, wide, narrow, vertically tall, almond-shaped, deep-set, prominent, close-set, wide-set, upturned, downturned, with a raised / low / prominent brow ridge

Training data

No public corpus pairs prompts with identity-preserving FLAME shape deltas, so the training set is synthetic:

  • Source identities: 300 FLAME 2020 identities fitted by MICA to the Liverpool-York Head Model (LYHM) dataset.
  • Edit prototypes: 129 hand-authored edits over 6 regions. Each one is a sparse set of FLAME vertex indices with 3D displacement vectors and a canonical prompt fragment.
  • Targets: for every (subject, edit, scale ∈ {0.5, 1.0, 1.5}) tuple, Δβ comes from an iterative projection onto the FLAME shape basis (Adam, 1500 steps, lr 1e-2, ℓ2 weight 1e-6). An outside-region penalty (weight 1e2) stops the deformation from leaking into other regions.
  • Prompts: 5 template-based variants per tuple, with scale-dependent qualifiers. This gives 580,500 triplets (β₀, t, β₁): 300 × 129 × 3 × 5.
  • Split: 85% train / 15% validation, random over triplets with seed 42. Validation shares identities and edit prototypes with training, so the validation loss does not measure generalisation to unseen identities or edits.

The training data, LYHM and FLAME are not redistributed with this model.

Training procedure

  • Objective: region-masked vertex losses inside the edited region (λ = 5·10⁵) and outside it (λ = 5·10⁵), ℓ2 on shape parameters (λ = 10), gate cross-entropy (λ = 10) and a free-bits KL term (λ = 0.05, free bits 0.03, 10-epoch linear warm-up).
  • Optimisation: AdamW (lr 1e-4, weight decay 0.01, gradient clipping 1.0), batch size 256, 100 epochs on a single NVIDIA RTX 5090. Training took about 40 h; data generation took about 70 h.
  • Checkpoint: best validation loss, reached at epoch 71 of 100. Only the DeltaRegion heads were trained; the Flan-T5 encoder stayed frozen.
  • Precision: fp32.

Limitations

  • Bounded edit space. The model only knows the 129 prototypes listed above and their interpolations. Structural changes outside that set are not produced reliably, and prompts far from the training phrasing may be mapped to the wrong region or produce weak edits. Check gate / region_probabilities to see which region was selected.
  • One region per prompt. The gate is trained with single-region labels. Compound prompts ("big nose and thin lips") are not supported; apply the edits one after another instead.
  • Shape only. Only FLAME identity geometry changes. Hair, accessories, skin, eye colour and texture are not modelled, and expression/pose coefficients are untouched.
  • Magnitude. Edit strength was only supervised at 0.5×, 1.0× and 1.5×. Values of prior_scale far outside about [0.5, 2] may produce implausible shapes.
  • Source distribution. Training identities are MICA fits to LYHM scans. Identities far from that distribution, or from other FLAME fitting pipelines with a different coefficient distribution, may behave worse.
  • English only, with short imperative prompts.

Bias, risks and ethical considerations

  • Demographic coverage. LYHM was collected in the United Kingdom and does not represent the global population evenly. The model has seen a limited variety of age, ethnicity and face morphology, and may perform worse on under-represented groups.
  • Language bias. Words such as "big" or "prominent" were mapped to shapes chosen by the authors. They are not objective or culturally neutral.
  • Misuse. Editing a real person's facial geometry can be used to create misleading or non-consensual depictions. Only edit identities of people who have consented, and disclose that avatars are synthetic.
  • No biometric use. The model is not intended for face recognition, identity verification, or inferring any personal attribute.

License

The DeltaRegion weights and code are released under CC BY-NC-SA 4.0 (non-commercial use only), because the model is derived from:

  • FLAME 2020 (Max Planck Institute for Intelligent Systems): non-commercial scientific research licence. You need your own FLAME license to turn the outputs into meshes. Commercial, military, surveillance, pornographic, and defamatory uses are strictly prohibited.
  • LYHM identities, fitted with MICA: academic, non-commercial use.
  • Flan-T5-Large encoder weights and tokenizer from google/flan-t5-large: Apache-2.0. They are bundled here unchanged.

Make sure your use also complies with the FLAME and LYHM licence terms.

Citation

@inproceedings{canela2026manifoldsplat,
  title     = {ManifoldSplat: Language-Guided Semantic Shape Editing of 3D Gaussian Head Avatars},
  author    = {Canela, Antonio and Sanchez-Riera, Jordi},
  booktitle = {DAGM German Conference on Pattern Recognition (GCPR)},
  year      = {2026}
}

Please also cite FLAME (Li et al., SIGGRAPH Asia 2017), MICA (Zielonka et al., ECCV 2022), LYHM (Dai et al., IJCV 2020) and Flan-T5 (Chung et al., JMLR 2024).

Downloads last month
92
Safetensors
Model size
0.3B params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support