creative-graphic-design/structure-diffusion

Model Card for StructureDiffusion

This package implements StructureDiffusion, a training-free text-to-image method from ICLR 2023. It parses a prompt into noun phrases, encodes the full prompt and each phrase, then combines cross-attention keys and values to improve object and attribute binding. The 🧨diffusers pipeline uses the Stable Diffusion 1.4 checkpoint.

Model Details

Model Description

StructureDiffusionPipeline subclasses diffusers.StableDiffusionPipeline, so it uses the framework's standard loading, device placement, callbacks, scheduler, and image output behavior. parser_type selects constituency parsing or optional scene-graph parsing. conjunction=True uses phrase-specific keys with one combined value; the default uses the full prompt as the key and the full prompt plus phrase encodings as values.

• Developed by: Weixi Feng et al.
• Shared by: creative-graphic-design.
• Model type: image generation; task: text-to-image; conditioning: text.
• Language(s): English prompts.
• Code license: Apache-2.0.
• Reference implementation license: MIT.
• Base checkpoint: CompVis Stable Diffusion v1.4; its weights have separate CreativeML OpenRAIL-M terms.

Model Sources

• Repository: official Structured Diffusion Guidance implementation
• Paper: arXiv:2212.05032
• Released base weights: CompVis/stable-diffusion-v1-4

Supported Checkpoints

Checkpoint	Hub ID	Status
Stable Diffusion v1.4	CompVis/stable-diffusion-v1-4	Gated base checkpoint; load with from_pretrained after access is approved.
Structure Diffusion package artifact	creative-graphic-design/structure-diffusion	Published code only; this package does not upload Structure Diffusion weights.

Uses

Direct Use

Use the pipeline for research inference on English prompts that describe multiple objects, attributes, or relations. The method changes cross-attention conditioning and does not train or fine-tune Stable Diffusion weights.

Downstream Use

Use generated images as draft visual material for creative-design workflows after inspecting them for prompt adherence and content suitability.

Out-of-Scope Use

Do not treat generated images as verified depictions of real people or events, or use them as evidence that prompt attributes were rendered correctly. The pipeline does not provide an image safety guarantee.

Bias, Risks, and Limitations

Generation behavior inherits the training data, biases, and restrictions of the selected Stable Diffusion checkpoint. The default constituency parser is English-only and requires the Stanza English tokenizer, POS, and constituency models to be installed separately. Scene-graph parsing is an optional extra with its own spaCy English model prerequisite. Outputs vary with the checkpoint, parser, scheduler, device, precision, and library versions.

Recommendations

Inspect generated images before downstream use, and record the checkpoint revision, parser, scheduler, dimensions, seed, and inference steps when comparing results.

How to Get Started with the Model

Install the package from this repository archive and download the English parser models before the first inference:

pip install "structure-diffusion @ https://github.com/creative-graphic-design/design-generators/archive/c2b5a63ce2c2e0247925f621a0787f71f36f78d0.zip#subdirectory=models/structure-diffusion"
python -c "import stanza; stanza.download('en', processors='tokenize,pos,constituency')"

CompVis/stable-diffusion-v1-4 is a gated model. Request access and accept its CreativeML OpenRAIL-M terms on the model page before downloading its weights. Set HF_TOKEN_FILE to a private file containing a token that can access the checkpoint.

import os
from pathlib import Path

import torch
from diffusers import DiffusionPipeline

token = Path(os.environ["HF_TOKEN_FILE"]).read_text().strip()
pipe = DiffusionPipeline.from_pretrained(
    "CompVis/stable-diffusion-v1-4",
    revision="133a221b8aa7292a167afc5127cb63fb5005638b",
    custom_pipeline="creative-graphic-design/structure-diffusion",
    torch_dtype=torch.float32,
    token=token,
    trust_remote_code=True,
)
pipe.to("cpu")
result = pipe(
    "A red car and a white sheep",
    parser_type="constituency",
    height=512,
    width=512,
    num_inference_steps=10,
    generator=torch.Generator(device="cpu").manual_seed(42),
)
image = result.images[0]
print("image size: {}x{}".format(image.size[0], image.size[1]))

image size: 512x512

The default constituency parser needs the English Stanza model assets downloaded separately with the command above. The optional scene-graph parser is available through the scene-graph extra and requires its spaCy English model. See REPRODUCING.md for local testing and CPU comparison instructions.

Examples

The gallery compares diffusers.StableDiffusionPipeline with StructureDiffusionPipeline using the same prompt and seed. It includes every output for seeds 42, 0, 1, and 2 for each prompt. The results vary by seed and do not show a consistent compositional improvement from structured guidance. For the car and sheep prompt, the sheep is absent from both seed-42 images; at seed 0 the structured result shows only part of a sheep, at seed 1 the baseline shows a sheep while the structured result does not, and both show a sheep at seed 2. For the backpack and elephant prompt, the baseline shows both objects while the structured result omits the elephant at seed 42; both omit the backpack at seed 0 and the elephant at seed 1; at seed 2 both show the objects, with the elephant more fully visible in the structured result. For the fruit prompt, neither pipeline clearly shows a banana at seeds 42, 0, or 1; at seed 2 both show several apples and banana-like shapes. The prompts are taken from Figure 1 (car and sheep; backpack and elephant) and Figure 13 (fruit examples) of the paper. All outputs are shown without selecting among seeds.

A red car and a white sheep

Source: Figure 1, attribute leakage.

Seed	Stable Diffusion v1.4	StructureDiffusion
42
0
1
2

A blue backpack and a brown elephant

Source: Figure 1, missing objects.

Seed	Stable Diffusion v1.4	StructureDiffusion
42
0
1
2

A green apple and a red banana

Source: Figure 13, CC-500 qualitative results.

Seed	Stable Diffusion v1.4	StructureDiffusion
42
0
1
2

Every image uses CompVis/stable-diffusion-v1-4 at revision 133a221b8aa7292a167afc5127cb63fb5005638b, the checkpoint's PNDM scheduler configuration (skip_prk_steps=True), 30 inference steps, 512×512 resolution, guidance scale 7.5, and CPU float32. The structured pipeline uses the constituency parser with conjunction=True; the baseline uses diffusers.StableDiffusionPipeline. The base checkpoint weights are licensed under CreativeML OpenRAIL-M. The image files are hosted in the public documentation-images dataset and are not committed to this repository.

This code recreates all displayed pairs with the same prompts and seeds. Set HF_TOKEN_FILE to a private file containing a Hugging Face token after accepting the checkpoint's access terms. Install the package and download the English Stanza parser models as shown above before running it.

import gc
import os
from pathlib import Path

import torch
from diffusers import StableDiffusionPipeline
from structure_diffusion import StructureDiffusionPipeline

MODEL_ID = "CompVis/stable-diffusion-v1-4"
MODEL_REVISION = "133a221b8aa7292a167afc5127cb63fb5005638b"
OUTPUT_DIR = Path("structure-diffusion-examples")
OUTPUT_DIR.mkdir(exist_ok=True)
EXAMPLES = (
    ("red-car-white-sheep", "A red car and a white sheep"),
    ("blue-backpack-brown-elephant", "A blue backpack and a brown elephant"),
    ("green-apple-red-banana", "A green apple and a red banana"),
)
SEEDS = (42, 0, 1, 2)

torch.set_num_threads(2)
token = Path(os.environ["HF_TOKEN_FILE"]).read_text().strip()


def make_generator(seed):
    return torch.Generator(device="cpu").manual_seed(seed)


baseline = StableDiffusionPipeline.from_pretrained(
    MODEL_ID,
    revision=MODEL_REVISION,
    torch_dtype=torch.float32,
    token=token,
)
baseline.to("cpu")
for slug, prompt in EXAMPLES:
    for seed in SEEDS:
        image = baseline(
            prompt,
            height=512,
            width=512,
            num_inference_steps=30,
            guidance_scale=7.5,
            generator=make_generator(seed),
        ).images[0]
        image.save(OUTPUT_DIR / f"stable-diffusion-v1-4-{slug}-seed{seed}.png")
del baseline

gc.collect()
structured = StructureDiffusionPipeline.from_pretrained(
    MODEL_ID,
    revision=MODEL_REVISION,
    torch_dtype=torch.float32,
    token=token,
)
structured.to("cpu")
for slug, prompt in EXAMPLES:
    for seed in SEEDS:
        image = structured(
            prompt,
            parser_type="constituency",
            conjunction=True,
            height=512,
            width=512,
            num_inference_steps=30,
            guidance_scale=7.5,
            generator=make_generator(seed),
        ).images[0]
        image.save(OUTPUT_DIR / f"structure-diffusion-{slug}-seed{seed}.png")

Training Details

Training Data

This method adds no training data or learned weights. The selected Stable Diffusion checkpoint retains its original data provenance and terms.

Training Procedure

No training or checkpoint conversion is performed by this package. It loads the released pipeline-format base checkpoint and changes the denoising pipeline's text-conditioning path.

Evaluation

Testing Data, Factors & Metrics

The independent CPU comparison covers 64×64 and 512×512 outputs for A red car and a white sheep, seed 42, 10 PNDM steps through the PLMS path in diffusers, guidance scale 7.5, and CPU float32 execution. Each reference/package pair receives the same precomputed starting latent. The vendor demo calls preprocess_prompts before parsing and token alignment, which lowercases the prompt; its get_token_alignment_map helper is case-sensitive only when called directly with mismatched prompt and tokenizer casing. The package also accepts capitalized input.

The test directly compares vendor-derived noun phrases and token spans, and compares independently assembled key/value conditioning with the package conditioning. For the image comparison, a separate base diffusers pipeline uses the pinned vendor attention processors to generate the reference image; the test compares that final image with the StructureDiffusionPipeline output. It does not capture or compare per-layer attention outputs. Both model loading and sampling use diffusers, so this checks structured conditioning and its end-to-end image result without reproducing the vendor checkpoint loader or sampler implementation.

Parity Results

Check	Cases	Criterion	Result
64×64 CPU image output against the reference image	1	Exact phrase and token-span match; K/V max difference ≤ 1e-6; image MAE < 1e-3	a red car / a white sheep; spans [1, 4) / [5, 8); 1 key / 3 values; K/V max differences 0.0 / 0.0; pixel max/mean/RMS differences 0.0 / 0.0 / 0.0; bit-exact, SHA-256 match.
512×512 CPU image output against the reference image	1	Exact phrase and token-span match; K/V max difference ≤ 1e-6; image MAE < 1e-3	a red car / a white sheep; spans [1, 4) / [5, 8); 1 key / 3 values; K/V max differences 0.0 / 0.0; pixel max/mean/RMS differences 0.0 / 0.0 / 0.0; bit-exact, SHA-256 match.

The independent comparison ran on commit 786fcf6cfbcfb68b9c9565d8ee275c32cd5a28ac; the reviewer recorded the full inputs, initial latents, measurements, and runtimes in the formal parity result. Both resolutions were bit-exact: the package and reference image SHA-256 values matched.

Reproducibility

Run the CPU comparison test in REPRODUCING.md to compare noun phrases, token spans, and K/V conditioning directly, then compare final generated images using the same scheduler, seed, and initial latent. A separate base diffusers pipeline runs the pinned reference attention processors to generate the reference image; the test does not compare per-layer attention outputs. diffusers supplies model loading and sampling in both paths, so the result does not claim parity with the reference checkpoint loader or sampler. The independently reviewed comparison on commit 786fcf6 matched phrase and span outputs exactly, had zero measured K/V differences, and produced bit-exact 64×64 and 512×512 images with identical SHA-256 values.

Environmental Impact

No model training is performed. The formal standard-CPU comparison took 126.777 seconds at 64×64 and 948.902 seconds at 512×512; no GPU comparison is included.

Technical Specifications

Model Architecture and Objective

The pipeline retains Stable Diffusion's VAE, CLIP text encoder, U-Net, and scheduler. It extracts phrase spans, encodes each phrase independently, aligns phrase tokens to the full prompt, and applies the official method's attention-map and value-composition rules in U-Net cross-attention layers.

Compute Infrastructure

The package supports CPU and accelerator execution with diffusers. The recorded comparison timings are test-process runtimes on a standard Colab CPU; setup time and GPU performance are not included.

Software

Runtime dependencies are declared in models/structure-diffusion/pyproject.toml. The default parser requires Stanza models; scene-graph parsing also requires the optional scene-graph extra and its spaCy model.

Citation

@inproceedings{feng2023trainingfree,
  title={Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis},
  author={Feng, Weixi and He, Xuehai and Fu, Tsu-Jui and Jampani, Varun and Akula, Arjun and Narayana, Pradyumna and Basu, Sugato and Wang, Xin Eric and Wang, William Yang},
  booktitle={International Conference on Learning Representations},
  year={2023},
  eprint={2212.05032},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url = "https://arxiv.org/abs/2212.05032"
}

Model Card Contact

Open an issue or pull request in the design-generators repository.