SAR2EO-CDiffSET

A conditional latent diffusion model that translates Sentinel-1 SAR imagery into Sentinel-2-style EO (optical) imagery. Full training code, experiment log, and architecture details: github.com/AyaanZ30/SAR_2_EO_CrossModal_Translation.

Model Description

CDiffSETUNet is a custom U-Net-based conditional diffusion model operating in stabilityai/sd-vae-ft-mse's 4-channel latent space. SAR input (VV/VH polarizations) is converted to a 3-channel pseudo-RGB representation before encoding, then injected as conditioning into the denoising U-Net at every resolution via a dedicated multi-scale SAR encoder. Conditioning is fused via direct element-wise addition of SAR features into the main path at each of the four resolutions (input scale through the bottleneck), not concatenation — this keeps channel dimensions unchanged through the network and adds negligible parameter overhead for the fusion itself.

• Architecture: Custom U-Net, base channel width 64, 3 down/up stages, 3 residual blocks per stage
• Bottleneck: Dilated convolution block (DilatedBottleneck) at 4×4 spatial resolution
• Diffusion: Epsilon-prediction, DDPM (1000 training timesteps), Min-SNR-γ weighted MSE loss (γ=5)
• Sampling: DDIM, 250 inference steps, no classifier-free guidance (guidance_scale=1)
• Latent space: stabilityai/sd-vae-ft-mse (frozen, not finetuned)
• Note: the model also outputs a per-pixel confidence map (confidence_head) alongside the noise prediction. This output exists in the architecture and checkpoint but is currently unused downstream — it's not factored into the loss or sampling process in this release.

Intended Uses & Limitations

Intended use: research and experimentation with SAR-to-optical image translation, particularly as a reference point for latent-diffusion approaches to this task. Not validated for operational/production remote sensing use.

Known limitations:

• Trained on a 25-ROI-scene subset of SEN1-2 — a small dataset by diffusion-model standards. Generalization outside similar geography/season/sensor conditions is untested.
• Fine spatial detail (individual buildings, sharp field/road boundaries) is not reliably preserved — output tends toward a smoothed, painterly texture rather than crisp structure, likely a combination of the 8x-compressive VAE latent space and limited training data/budget.
• Flat, low-texture, or low-light scenes (open water, desert, night imagery) show the least reliable color accuracy, plausibly because SAR backscatter provides little disambiguating signal in these regions.
• Validation Image L1 was still improving at the final checkpoint (epoch 75) — this model may not represent full convergence for this architecture/dataset combination.
• This is a research artifact from an iterative debugging project, not a benchmarked, peer-reviewed model. See the GitHub repo's "What Didn't Work" section for approaches that were tried and abandoned (domain-specific VAE, hand-crafted auxiliary losses, v-prediction, CFG>1).

Training Data

SEN1-2 — paired Sentinel-1 SAR / Sentinel-2 EO patches. This model used a 25-ROI-scene subset (ROIs1868_summer), split 85/15 train/val.

Training Procedure

Hyperparameter	Value
Epochs	75
Batch size	16
Optimizer	AdamW
Learning rate	5e-4 (cosine schedule, 5-epoch warmup)
Weight decay	1e-2
EMA decay	0.999
Precision	fp16 (mixed)
Hardware	2x Kaggle T4 GPUs (DDP)

Evaluation Results

Metric	Epoch 75 (final)
Val L1 Error (noise-prediction space)	0.3493
Image-space L1 (VAE-decoded, vs. ground truth)	0.3771

Full per-checkpoint trend (epochs 25/50/75) and best/worst-case qualitative comparison images are in the GitHub repo README — not duplicated here to keep this card focused.

How to Use

import torch
from huggingface_hub import hf_hub_download

weights_path = hf_hub_download(repo_id="AyaanZ30/sar2eo-cdiff",
                                 filename="cdiffset_final_weights.pt")
ckpt = torch.load(weights_path, map_location="cpu")

# Requires CDiffSETUNet from the SAR_2_EO_CrossModal_Translation repo's C_Diff/model.py
from C_Diff.model import CDiffSETUNet

model = CDiffSETUNet(
    latent_ch=ckpt["config"]["latent_channels"],
    base_ch=ckpt["config"]["base_channels"],
)
model.load_state_dict(ckpt["model_state_dict"])
model.eval()

print(f"Loaded weights from epoch {ckpt['epoch']}")

Full inference pipeline (SAR preprocessing → SD-VAE encode → diffusion sampling → decode) requires the diffusion/sampling code in the GitHub repo's C_Diff/diffusion.py and C_Diff/utils/vae_ops.py — this card only hosts the model weights themselves.

Citation

If referencing this work, please link to the GitHub repository, which contains the full experimental methodology:

@misc{sar2eo-cdiffset,
  author = {Ayaan},
  title = {SAR2EO-CDiffSET: Conditional Latent Diffusion for SAR-to-EO Translation},
  year = {2026},
  url = {https://github.com/AyaanZ30/SAR_2_EO_CrossModal_Translation}
}

Acknowledgements

Architecture and SAR pseudo-RGB conditioning approach adapted from the C-DiffSET paper's methodology. VAE: stabilityai/sd-vae-ft-mse.