canter / API.md

Publish Canter 0.5.0 contrastive PDG support

13490a2 verified 3 days ago

preview code

Raw

History Blame Contribute Delete

20.7 kB

Canter API and inference parameters

Model card · Example gallery · Technical report

Canter exposes a high-level image pipeline and a lower-level latent inference engine. Configuration uses frozen dataclasses and enums.

The installed canter package always supplies the inference implementation. For remote model IDs, revision selects checkpoint artifacts only; selecting an older checkpoint does not load or execute the Python package bundled in that historical repository snapshot.

Minimal use

from canter import CanterPipeline

pipe = CanterPipeline.from_pretrained("data-archetype/canter")
output = pipe("A red rally car driving around a wet forest road")
image = output.image

Generation arguments

Parameter	Default	Description
`prompts`	required	One prompt string or a sequence containing one prompt per image.
`negative_prompts`	`None`	Optional negative prompt string or sequence for CFG and contrastive PDG. The count must match `prompts`. `None` uses the learned unconditional token.
`config`	`CanterPipelineConfig()`	Inference and output settings.
`initial_noise`	`None`	Optional float32 latent noise tensor with the configured batch and spatial shape.
`progress`	`None`	Optional callback receiving completed and total solver updates.
`preview`	`None`	Optional callback receiving asynchronous latent-RGB PIL images plus completed and total solver updates.

CanterPipeline loads the flow-matching denoiser and text tokenizer with the bundled SmolLM2-360M weights. It compiles the selected text backend, downloads DINAC-AE-D2, generates latents, and decodes them into images.

Loading the pipeline

from canter import CanterPipeline, TextAttentionBackend, WeightDType

pipe = CanterPipeline.from_pretrained(
    "data-archetype/canter",
    dtype=WeightDType.BFLOAT16,
    text_backend=TextAttentionBackend.DENSE,
    device="cuda",
    revision=None,
    cache_dir=None,
    compile_model=True,
)

`CanterPipeline.from_pretrained`

Parameter	Default	Description
`path_or_repo_id`	required	A Hugging Face repository ID or downloaded model repository directory.
`dtype`	`WeightDType.BFLOAT16`	Weight storage dtype. Compute still uses bfloat16 AMP with explicit float32 operations.
`text_backend`	`TextAttentionBackend.DENSE`	Bundled text-encoder, text-refinement, and cross-attention layout.
`device`	`"cuda"`	CUDA device string or `torch.device`.
`revision`	`None`	Hugging Face checkpoint branch, commit, or immutable release tag. `None` uses the installed package's pinned default checkpoint. An explicit tag such as `"v0001"` loads those weights with the currently installed code.
`cache_dir`	`None`	Optional Hugging Face cache directory.
`compile_model`	`True`	Compile the selected inference kernels. Compilation errors are reported directly.
`vae`	`None`	Optional compatible `CanterVae` instance. `None` downloads DINAC-AE-D2 automatically.

Weight dtypes

Enum	Value	Description
`WeightDType.BFLOAT16`	`"bfloat16"`	Default compact release with required float32 parameter islands retained.
`WeightDType.FLOAT32`	`"float32"`	Full-float32 weight storage release. Model compute remains bfloat16 AMP.

Text backends

Enum	Value	Description
`TextAttentionBackend.DENSE`	`"dense"`	Compatibility default using padded tensors and standard PyTorch scaled dot-product attention throughout text encoding and denoising.
`TextAttentionBackend.JAGGED`	`"jagged"`	Explicit optimization using packed text, CUDA FlashAttention, and jagged NestedTensor kernels throughout text encoding and denoising.

Only the selected backend is compiled. The backend cannot be changed after compilation.

Pipeline configuration

from canter import (
    CanterInferenceConfig,
    CanterOutputType,
    CanterPipelineConfig,
)

config = CanterPipelineConfig(
    inference=CanterInferenceConfig(),
    output_type=CanterOutputType.PIL,
)
output = pipe("A portrait lit by a large north-facing window", config=config)

`CanterPipelineConfig`

Field	Default	Description
`inference`	`CanterInferenceConfig()`	Latent generation settings.
`output_type`	`CanterOutputType.PIL`	Selected decoded or latent output representation.

Output types

Enum	Result
`CanterOutputType.PIL`	`output.images` contains one PIL image per prompt. `output.image` returns the sole image for batch size one.
`CanterOutputType.TENSOR`	`output.image_tensor` contains clamped float32 RGB pixels in `[-1, 1]`.
`CanterOutputType.LATENT`	VAE decoding is skipped. `output.latents` contains whitened float32 model latents.

Every output also contains the descending float32 solver schedule in output.schedule.

Live latent previews

def show_preview(images, completed, total):
    images[0].show()


output = pipe(
    "A lighthouse in a winter storm",
    preview=show_preview,
)

The bundled float32 linear projection maps the whitened 128-channel DINAC latent directly to RGB at one eighth of the requested image size. Every solver update is eligible by default. If the previous preview is still transferring, converting, or running the callback, the new update is dropped instead of blocking sampling. The callback receives this native one-eighth-size image; the web UI lets the browser scale it for display.

Accepted previews perform only a 1×1 projection and pixel shuffle on CUDA. The small RGB result is copied non-blockingly into pinned host memory. CUDA event waiting, PIL conversion, and the application callback all run on a single background worker. Every step is eligible; the one-worker busy-drop policy supplies backpressure without a separate callback cap.

Inference configuration

Defaults

from canter import (
    CanterInferenceConfig,
    CfgGuidance,
    PdgCurve,
    PdgGuidance,
    PdgMode,
    Schedule,
    Solver,
)

config = CanterInferenceConfig(
    height=1216,
    width=832,
    steps=50,
    solver=Solver.ABM2,
    schedule=Schedule.BETA,
    log_snr_shift=0.0,
    cfg=CfgGuidance(
        enabled=False,
        scale=None,
        start_step=0,
        stop_step=None,
    ),
    pdg=PdgGuidance(
        enabled=True,
        mode=PdgMode.FULL,
        curve=PdgCurve.CONSTANT,
        noisy_scale=2.5,
        clean_scale=2.5,
        power=3.0,
        start_step=0,
        stop_step=None,
    ),
    self_attention_gain=-0.03,
    sde_noise_multiplier=1.0,
    seed=42,
    generator=None,
)

`CanterInferenceConfig`

Field	Default	Description
`height`	`1216`	Output height in pixels. Must be positive and divisible by 16.
`width`	`832`	Output width in pixels. Must be positive and divisible by 16.
`steps`	`50`	Number of solver state updates. A run with 50 updates uses 51 schedule points.
`solver`	`Solver.ABM2`	Numerical solver.
`schedule`	`Schedule.BETA`	Timestep spacing.
`log_snr_shift`	`0.0`	Additive log-SNR schedule shift. Zero leaves the selected schedule unchanged.
`cfg`	disabled	Classifier-free guidance settings.
`pdg`	full-path PDG 2.5	Path-drop guidance settings.
`self_attention_gain`	`-0.03`	Gain applied exclusively to image self-attention on the main denoiser path.
`sde_noise_multiplier`	`1.0`	Shared non-negative stochastic noise multiplier used by Euler-Maruyama and ER-SDE.
`seed`	`42`	Random seed. Set to `None` when supplying `generator`.
`generator`	`None`	Optional CUDA `torch.Generator` on the same device as the model. Exactly one of `seed` and `generator` is required.

The seed controls latent noise, SPRINT routing, and stochastic solver operations. The VAE decoder retains its published seed behavior.

PNG metadata

Images downloaded from the Gradio interface contain a canter PNG text field with compact JSON. The object begins with the prompt, optional negative prompt, effective per-image seed, width, height, steps, solver, and schedule. It then records PDG, CFG, self-attention gain, logSNR shift, the Euler-Maruyama and ER-SDE noise multipliers, installed code version, numbered checkpoint release, and weight dtype.

Solvers

Enum	Value	Description
`Solver.EULER`	`"euler"`	First-order deterministic Euler updates.
`Solver.EULER_MARUYAMA`	`"euler_maruyama"`	Stochastic reverse-SDE updates. Uses `sde_noise_multiplier`.
`Solver.ER_SDE`	`"er_sde"`	Third-stage VP ER-SDE with 16-point Gauss-Legendre correction quadrature. Uses `sde_noise_multiplier`.
`Solver.DPMPP_2M`	`"dpmpp_2m"`	Flow-matching DPM++ 2M updates with finite pre-shift endpoint handling.
`Solver.ABM2`	`"abm2"`	Variable-step Adams-Bashforth-Moulton updates with corrected-state reevaluation.

ABM2 is the default and performs additional denoiser evaluations for its corrected states.

Schedules

Enum	Value	Description
`Schedule.LINEAR`	`"linear"`	Uniform spacing from noisy to clean.
`Schedule.BETA`	`"beta"`	Beta(0.6, 0.6) quantile spacing with more schedule density near the endpoints.

Schedules and solver state remain float32.

In the bundled web interface, selecting Beta(0.6, 0.6) resets the log-SNR shift to 0.0; selecting Linear resets it to -2.3. The shift remains independently editable after either preset is applied. Direct API configurations use the explicit log_snr_shift supplied by the caller.

CFG

from canter import (
    CanterInferenceConfig,
    CanterPipelineConfig,
    CfgGuidance,
    PdgGuidance,
    PdgMode,
)

cfg = CfgGuidance(
    enabled=True,
    scale=3.0,
    start_step=0,
    stop_step=29,
)

output = pipe(
    "A studio portrait with soft natural light",
    negative_prompts="oversaturated, harsh contrast",
    config=CanterPipelineConfig(
        inference=CanterInferenceConfig(
            cfg=cfg,
            pdg=PdgGuidance(mode=PdgMode.COMBINED_CFG_PDG),
        ),
    ),
)

`CfgGuidance`

Field	Default	Description
`enabled`	`False`	Enable classifier-free guidance.
`scale`	`None`	Non-negative guidance scale. Required when CFG is enabled or when the selected PDG mode uses CFG.
`start_step`	`0`	First active solver update, inclusive.
`stop_step`	`None`	Last active solver update, inclusive. `None` selects the final update.

Step indices run from 0 through steps - 1.

An explicit negative prompt replaces the learned unconditional token on CFG and contrastive PDG branches. Negative prompts require enabled CFG or a PDG mode that consumes contrastive text. For prompt batches, supply one negative prompt per positive prompt.

PDG

from canter import PdgCurve, PdgGuidance, PdgMode

pdg = PdgGuidance(
    enabled=True,
    mode=PdgMode.FULL,
    curve=PdgCurve.POWER,
    noisy_scale=2.0,
    clean_scale=2.5,
    power=3.0,
    start_step=0,
    stop_step=None,
)

`PdgGuidance`

Field	Default	Description
`enabled`	`True`	Enable path-drop guidance.
`mode`	`PdgMode.FULL`	Path and CFG interaction policy.
`curve`	`PdgCurve.CONSTANT`	Scale interpolation from noisy to clean.
`noisy_scale`	`2.5`	PDG scale at the first noisy schedule point.
`clean_scale`	`2.5`	PDG scale at the final clean schedule point.
`power`	`3.0`	Positive exponent used by the power curve.
`start_step`	`0`	First active solver update, inclusive.
`stop_step`	`None`	Last active solver update, inclusive. `None` selects the final update.

Constant PDG requires equal noisy_scale and clean_scale. Disabled PDG requires enabled=False and mode=PdgMode.NONE.

Increasing the PDG scale sharpens the generated distribution. Moderate values improve image structure and fine detail at the cost of reduced variety. Pushing the scale too far can introduce structural defects, excessive contrast, and oversaturation.

PDG curves

Enum	Value	Scale behavior
`PdgCurve.CONSTANT`	`"constant"`	Uses one scale throughout inference.
`PdgCurve.LINEAR`	`"linear"`	Interpolates linearly from `noisy_scale` to `clean_scale`.
`PdgCurve.POWER`	`"power"`	Interpolates with `position ** power`.

PDG modes

Enum	Value	Behavior
`PdgMode.NONE`	`"none"`	No PDG branch. Required when PDG is disabled.
`PdgMode.FULL`	`"full"`	Guides from the middle-skipped path toward the full main path.
`PdgMode.FULL_CONTRASTIVE`	`"full_contrastive"`	Uses negative text on the middle-skipped path, or learned unconditional text when no negative prompt is supplied.
`PdgMode.THREE_QUARTER`	`"three_quarter"`	Guides from the 75 percent SPRINT path toward the full main path.
`PdgMode.ALTERNATE_PDG_FIRST`	`"alternate_pdg_first"`	Alternates PDG on even updates and CFG on odd updates within the PDG window.
`PdgMode.ALTERNATE_CFG_FIRST`	`"alternate_cfg_first"`	Alternates CFG on even updates and PDG on odd updates within the PDG window.
`PdgMode.COMBINED_CFG_PDG`	`"combined_cfg_pdg"`	Evaluates CFG and PDG together and averages their guidance deltas.
`PdgMode.PDG_WITH_ALTERNATING_CFG`	`"pdg_with_alternating_cfg"`	Applies PDG on every active update and adds CFG on odd updates.
`PdgMode.CFG_TO_PDG`	`"cfg_to_pdg"`	Uses CFG before the PDG window, then uses full-path PDG inside the window.

The five compound modes require cfg.scale. PdgMode.FULL_CONTRASTIVE uses the PDG scale only. start_step=0 with PdgMode.CFG_TO_PDG begins directly with PDG.

pdg_branch_conditioning(mode) returns PdgBranchConditioning.POSITIVE or PdgBranchConditioning.CONTRASTIVE for the selected PDG alternative path.

Self-attention gain

self_attention_gain changes image self-attention only on the full main path. The model multiplies main-path attention queries by exp(self_attention_gain). Negative values reduce the attention-logit scale and therefore increase the effective softmax temperature. This softens the main prediction and helps moderate PDG oversaturation. Text self-attention, cross-attention, and weak guidance paths retain their trained scales.

The default is -0.03. A value of 0.0 uses the trained main-path self-attention scale without adjustment. Smaller images generally benefit from more negative values. As image size increases, the gain should move closer to zero.

Custom configuration example

from canter import (
    CanterInferenceConfig,
    CanterOutputType,
    CanterPipelineConfig,
    CfgGuidance,
    PdgCurve,
    PdgGuidance,
    PdgMode,
    Schedule,
    Solver,
)

inference = CanterInferenceConfig(
    height=1024,
    width=1024,
    steps=40,
    solver=Solver.DPMPP_2M,
    schedule=Schedule.LINEAR,
    log_snr_shift=0.5,
    cfg=CfgGuidance(
        enabled=True,
        scale=3.0,
        start_step=0,
        stop_step=11,
    ),
    pdg=PdgGuidance(
        enabled=True,
        mode=PdgMode.CFG_TO_PDG,
        curve=PdgCurve.POWER,
        noisy_scale=2.0,
        clean_scale=2.5,
        power=3.0,
        start_step=12,
        stop_step=39,
    ),
    self_attention_gain=-0.03,
    sde_noise_multiplier=1.0,
    seed=123,
    generator=None,
)

config = CanterPipelineConfig(
    inference=inference,
    output_type=CanterOutputType.PIL,
)
image = pipe("A glass greenhouse during heavy rain", config=config).image

Prompt batches

A sequence of prompts generates one image per prompt:

output = pipe(
    [
        "A windswept beach under dark clouds",
        "A sunlit kitchen with white tiled walls",
    ]
)

first, second = output.images

Batch size one is the primary inference path. The jagged backend packs active prompt tokens without padding them through text refinement and cross-attention. Prompts longer than 512 tokens are truncated with a warning.

Latent generation

Use CanterInferenceEngine to generate latents without loading or calling the VAE:

from canter import CanterComponents, CanterInferenceEngine

components = CanterComponents.from_pretrained("data-archetype/canter")
engine = CanterInferenceEngine(components)
output = engine.generate("A mountain road in winter")

latents = output.latents
schedule = output.schedule

CanterInferenceEngine.generate accepts:

Parameter	Default	Description
`prompts`	required	One string or a sequence of strings.
`negative_prompts`	`None`	Optional negative prompt string or sequence for CFG and contrastive PDG branches. The count must match `prompts`; blank text uses learned unconditional conditioning.
`config`	`CanterInferenceConfig()`	Latent inference settings.
`initial_noise`	`None`	Optional float32 noise with shape `[batch, 128, height / 16, width / 16]`.
`progress`	`None`	Optional callback receiving `(completed_updates, total_updates)`.
`state_callback`	`None`	Optional synchronous callback receiving `(state, completed_updates, total_updates)` after every solver update. Prefer pipeline previews when full CUDA latent access is not required.

An empty or whitespace-only prompt uses the learned unconditional token without running the text encoder. Every prompt in a batch must be either blank or nonblank.

The returned latents use float32 and channels-last memory format.

Direct VAE encoding and decoding

CanterVae exposes both directions of the pinned DINAC-AE latent interface:

import torch
from canter import CanterVae

vae = CanterVae.from_pretrained(
    "data-archetype/dinac_ae_d2",
    device="cuda",
)
images = torch.zeros((1, 3, 1216, 832), device="cuda", dtype=torch.float32)
latents = vae.encode(images)
reconstructed = vae.decode(latents, height=1216, width=832)

Encoder inputs and decoder outputs are float32 BCHW tensors in the DINAC image range [-1, 1]. Encoded latents are whitened float32 tensors with 128 channels and a spatial stride of 16.

Conditioning, guidance, and solver APIs

The public inference API exposes text conditioning, guidance, schedules, solvers, and the latent-RGB preview projection as separate components:

import torch
from canter import (
    LOGSNR_SOLVER_START_EPS,
    CanterGuidanceConfig,
    CanterGuidedVelocity,
    LatentRgbProjection,
    Schedule,
    Solver,
    build_schedule,
    solve,
)

conditional = engine.prepare_conditioning("A mountain road in winter")
unconditional = engine.prepare_unconditional(batch=1)
guidance = CanterGuidanceConfig()
generator = torch.Generator(device=engine.device).manual_seed(42)
velocity = CanterGuidedVelocity(
    components.model,
    conditional,
    unconditional,
    guidance,
    generator,
)
schedule = build_schedule(
    Schedule.BETA,
    steps=guidance.steps,
    log_snr_shift=0.0,
    finite_noisy_endpoint_epsilon=None,
    device=engine.device,
)
latents = solve(
    velocity,
    initial_noise,
    schedule,
    solver=Solver.ABM2,
    generator=generator,
)
preview_projection = LatentRgbProjection.bundled()

prepare_conditioning validates a string or prompt sequence and performs the bundled text-encoder/refinement pass. An entirely blank batch returns learned unconditional conditioning; mixed blank and nonblank batches are rejected. prepare_unconditional creates learned conditioning for an explicit positive batch size.

CanterGuidanceConfig contains the solver-step guidance state: step count, CFG, PDG, and self-attention gain. CanterGuidedVelocity, build_schedule, and solve preserve the same PDG, CFG, self-attention gain, stochastic generator, and float32 solver behavior as CanterInferenceEngine.generate.

For Solver.DPMPP_2M and Solver.ER_SDE, pass LOGSNR_SOLVER_START_EPS as finite_noisy_endpoint_epsilon; other solvers use None. LatentRgbProjection.bundled() returns the validated CPU float32 1×1 projection weight and bias used by native previews.

Pipeline metadata

pipe.metadata records the installed Canter code version, resolved checkpoint release, weight dtype, source digests, text-encoder revision, VAE repository, and resolved immutable VAE revision. Applications that require reproducibility should store this metadata and pin both the package version and checkpoint tag.