SAR-to-EO Image Translation using Progressive Generative Models

This repository contains my GalaxEye technical assignment project on SAR-to-EO image translation using paired Sentinel-1/Sentinel-2 image patches.

The goal is to study how different generative models translate SAR images into optical-like EO images, and to understand their strengths, failures, and trade-offs. The project is structured as a progressive research workflow rather than a single-model experiment.

Project Overview

Synthetic Aperture Radar (SAR) provides all-weather and day-night imaging, but SAR images are difficult to interpret visually because of speckle noise, geometric effects, and backscatter-based appearance. Optical EO images are easier to interpret but are affected by cloud cover, illumination, and acquisition conditions.

This project explores SAR-to-EO translation using generative models. Since SAR-to-EO translation is underdetermined, the generated EO image is treated as a plausible optical prediction, not a true observation.

Dataset

Dataset used:

Sentinel-1&2 Image Pairs (SAR & Optical)

The dataset contains:

• 16,000 paired SAR/EO patches
• 4 terrain classes: agriculture, urban, barrenland, grassland
• 256 x 256 images
• scene-disjoint train/validation/test split

The dataset itself is not included in this repository due to size. The scripts expect the Kaggle dataset folder structure or a local equivalent.

Model Progression

Model	Architecture	Objective
M1	U-Net	L1 reconstruction baseline
M2	Pix2Pix	BCE GAN + L1
M3	Pix2Pix + perceptual loss	BCE GAN + L1 + perceptual + feature matching
M4	Attention ResUNet GAN	Hinge GAN + L1 + perceptual + feature matching
M5	Final Fusion GAN	Hinge GAN + L1 + perceptual + feature matching + gradient consistency + EMA

The aim was to understand how each modelling improvement changes performance, not just to select a model blindly.

Repository Structure

.
├── 01_eda_dataset_exploration.py
├── 02_dataset_dataloader.py
├── 03_train_unet_l1.py
├── 04_train_pix2pix.py
├── 05_train_pix2pix_perceptual.py
├── 06_train_attention_resunet.py
├── 07_train_final_fusion_gan.py
├── 08_evaluate_all_models.py
├── 09_ablation_and_failure_analysis.py
├── 10_generate_report_results.py
├── infer.py
├── sar2eo_common.py
├── requirements_kaggle.txt
├── README_RUN_ORDER.md
├── Report.pdf
└── sar2eo_outputs/

Installation

Recommended environment: Kaggle GPU or local Python environment with PyTorch.

pip install torch torchvision pillow numpy matplotlib tqdm scikit-image lpips torchmetrics torch-fidelity

For Kaggle:

pip install -q lpips scikit-image "torchmetrics[image]" torch-fidelity

Run Order

1. EDA and Manifest Creation

python 01_eda_dataset_exploration.py

2. DataLoader Validation

python 02_dataset_dataloader.py

3. Train Models

python 03_train_unet_l1.py --epochs 30 --train_limit 8000 --val_limit 1000 --batch_size 4
python 04_train_pix2pix.py --epochs 30 --train_limit 8000 --val_limit 1000 --batch_size 4
python 05_train_pix2pix_perceptual.py --epochs 30 --train_limit 8000 --val_limit 1000 --batch_size 4
python 06_train_attention_resunet.py --epochs 30 --train_limit 8000 --val_limit 1000 --batch_size 4
python 07_train_final_fusion_gan.py --epochs 30 --train_limit 8000 --val_limit 1000 --batch_size 4

4. Evaluate Models

python 08_evaluate_all_models.py --limit 1000

For slower local machines:

python 08_evaluate_all_models.py --limit 200 --batch_size 1 --num_workers 0 --skip_fid

5. Failure Analysis

python 09_ablation_and_failure_analysis.py

Do not run --train_ablations unless additional training time is available.

6. Generate Report Artifacts

python 10_generate_report_results.py

Inference

Use infer.py to generate an EO-like RGB image from a SAR grayscale PNG.

python infer.py \
  --checkpoint sar2eo_outputs/model_5_final_fusion_gan/checkpoints/best.pt \
  --input path/to/sar_image.png \
  --output generated_eo.png

Expected input:

• SAR image: grayscale PNG or any image readable by Pillow
• Output: RGB PNG

Final Evaluation Summary

The final local evaluation was run on 200 held-out test samples with LPIPS enabled and FID skipped.

Model	PSNR ↑	SSIM ↑	MAE ↓	LPIPS ↓
M1 U-Net L1	13.987	0.2115	0.1645	0.8579
M2 Pix2Pix	12.833	0.1113	0.1825	0.6151
M3 Pix2Pix + perceptual/FM	13.149	0.1517	0.1817	0.7087
M4 Attention ResUNet	13.994	0.1915	0.1656	0.9268
M5 Final Fusion GAN	13.963	0.2041	0.1643	0.9166

Key findings:

• M4 achieved the best PSNR.
• M1 achieved the best SSIM and fastest inference.
• M5 achieved the best MAE.
• M2 achieved the best LPIPS but weaker structural fidelity.
• No single model dominated all metrics.

Main Observations

The models learned SAR-conditioned optical priors, but the task remains difficult.

Common failure modes:

• over-smoothed EO outputs
• green/gray colour bias
• weak urban structure reconstruction
• missing fine boundaries and high-frequency details
• poor performance on dense edge regions

Failure analysis showed that urban scenes were the most difficult terrain class across all models. EO edge density had a strong negative correlation with SSIM, indicating that fine structure is the dominant failure mode.

Report

The final thesis-style report is included as:

Report.pdf

It contains literature review, EDA, methodology, model progression, quantitative results, qualitative comparisons, failure analysis, limitations, and future work.

Checkpoints

Model checkpoints may be excluded from GitHub if they exceed size limits.

Recommended checkpoint to share:

model_5_final_fusion_gan/checkpoints/best.pt

If possible, also share all model checkpoints through Google Drive, Kaggle Dataset, GitHub Releases, or Hugging Face.

Limitations

• The dataset uses PNG display images, not calibrated Sentinel-1 backscatter.
• SAR-to-EO translation is one-to-many and inherently uncertain.
• Generated EO images should not be treated as real optical observations.
• FID was skipped in local evaluation due to compute limitations.
• Full-scale evaluation on all test samples should be run on GPU for final benchmarking.

Future Work

Recommended next steps:

• evaluate on the full test split using GPU
• add calibrated SAR VV/VH preprocessing
• train controlled ablations for the final model
• add structure-aware or segmentation-guided losses
• condition generation on terrain metadata
• explore conditional or latent diffusion models
• generate multiple plausible EO outputs with uncertainty estimation

Conclusion

This project shows that SAR-to-EO translation is feasible but highly underdetermined. The progressive model study demonstrates that different objectives improve different aspects of the output. Pixel metrics, perceptual metrics, and qualitative realism do not always agree. The strongest contribution of this work is therefore not only the final model, but the complete research workflow: data understanding, reproducible implementation, model progression, evaluation, and failure analysis.