Dhruv1000/TSRDA

T-SRDA — Temporal Spatial-Reduction Dual-Attention for Crop Mapping

T-SRDA swaps the Swin/DaViT windowed temporal encoder of an AD-STCLN-style crop-mapping pipeline for a Spatial-Reduction Dual-Attention (SRDA) encoder — adapted from PDAViT (Zhou et al., Neurocomputing 2026) — applied along the temporal axis instead of the spatial axis it was designed for. The idea: reduce a satellite image time series to a small set of "key-value" summary tokens per pixel, then let every timestep attend to that reduced set in a single global cross-attention layer, instead of needing a stack of shifted local windows to approximate a global receptive field.

The pipeline is masked-reconstruction pretrained on unlabelled Sentinel-2 time series, then finetuned end-to-end for 18-class crop-type semantic segmentation on PASTIS, following the official STCLN protocol byte-for-byte (same seed, same folds, same steps/epoch, same hyperparameters) so the temporal encoder is the only intentionally-varied component.

Headline result: T-SRDA does not beat the baseline. Test mIoU 0.4124 vs. A_linear 0.4747 at the same seed (Δ = −0.0314, ≈4.4× the baseline's own seed-to-seed std of 0.0071). This is reported as a negative result — see § Results for the full picture, including three caveats that keep it from being conclusive (one seed, two residual architectural confounds, a checkpoint-selection mismatch).

This repository contains the full project: source code, training/eval scripts, every checkpoint from pretraining and finetuning, and the raw logs for every run.

---

Table of contents

1. Architecture
2. Dataset
3. Training configuration
4. Hardware & environment
5. Results
6. Training dynamics
7. Repository contents
8. Reproducing this run
9. Honest summary & next steps
10. Citation

---

Architecture

Input: (B, T, 10, H, W) Sentinel-2 time series + (B, T) acquisition-day offsets
        │
        ▼
┌───────────────────────────────────────────────────────────────┐
│  AtrousSpatialEncoder   (per-frame, shared across T)           │
│    dilated conv stem (d=64) → ASPP (dilations 1,2,4) → SE gate │
│    (B·T, 10, H, W) ──────────────────────────► (B·T, 256, H, W)│
└───────────────────────────────────────────────────────────────┘
        │  reshape to per-pixel sequences (B·H·W, T, 256)
        ▼
┌───────────────────────────────────────────────────────────────┐
│  TemporalSRDAEncoder   ← the architectural variable (T-SRDA)   │
│  3 × TemporalSRDABlock, R_schedule = [4, 4, 4]                 │
│                                                                  │
│    date PE added to x (sin/cos over real acquisition offsets)  │
│    for each block:                                              │
│      KV path : LayerNorm → Linear(d→d/R) → unfold R steps      │
│                → (T/R, d) → self-attention  ("self-checking")  │
│      Q  path : full-resolution T query tokens                  │
│      2nd attn: Q(T) × reduced-KV(T/R) cross-attention           │
│                → global temporal receptive field in ONE layer  │
│      scMLP   : FC → GELU + depthwise Conv1d(k=3, over T) → FC  │
└───────────────────────────────────────────────────────────────┘
        │  (B, H, W, T, 256)
        ▼
   ┌─────────────────────┐        ┌──────────────────────────────┐
   │  Pretrain: UTAEPrediction  │  │  Finetune: UTAEClassificationDual │
   │  NDVI-gated masked recon  │  │  STA temporal-attention pool      │
   │  Linear(256→10), MSE loss │  │  → semantic head (18+bg classes)  │
   │  4.01M params              │  │  → boundary head (morph. gradient)│
   │                             │  │  → sigmoid-gated residual refine  │
   │                             │  │  6.02M params                    │
   └─────────────────────┘        └──────────────────────────────┘

Why the temporal axis, not spatial? PDAViT's spatial-reduction dual-attention was designed to cut the quadratic cost of attention over image patches. Applied here to time instead of space, the same mechanism turns an irregular, variable-length acquisition sequence (33–61 Sentinel-2 revisits) into a small set of date-aware summary tokens, then gives every timestep a genuinely global view of the whole series in one layer — the Swin/DaViT baseline instead approximates this with a [4, 8, 16]-window + shift schedule.

Parameter count: temporal encoder alone 3.22M; full pretrain graph (spatial encoder + T-SRDA + reconstruction heads) 4.01M; full finetune graph (spatial encoder + T-SRDA + STA + dual decoder + gated refinement) 6.02M. For reference, the Swin variant of the same pipeline is ~2.71M and the DaViT variant ~3.50M (temporal encoder only).

What's held constant vs. the baseline (A_linear): everything except the temporal encoder was intended to be held constant, but two confounds remain in this arm — the decoder is dual (semantic + boundary + gated refinement) rather than a single linear head, and the spatial encoder is ASPP+SE rather than UTAE. Both are called out explicitly in § Results as caveats.

---

Dataset

PASTIS — Panoptic Agricultural Satellite TIme Series. Sentinel-2 optical time series over 2,433 patches (128×128 px, 10 spectral bands) in France, annotated with 18 crop types + background + void, at parcel-level panoptic granularity (only semantic segmentation is used here).

All four data roles are fold-disjoint (official 5-fold split, ~1 km spatial buffer between folds):

Role	Fold(s)	Units	Batching	Steps / epoch
Pretrain (unlabelled, masked recon)	5	496 patches	4 patches × 16 crops (4×4 grid)	1,984
Finetune train	1	76 IDs (72 unique)	2 patches × 2 crops	76
Finetune val	2	76 IDs (71 unique)	2 patches × 2 crops	76
Test	4	482 patches	full 128×128, no cropping	121 batches (batch 4)
unused	3	474 patches	—	—

• Crop size for pretrain/finetune: 32×32 (PATCH_SIZE // 4); test runs on the full 128×128 patch with no cropping and no TTA.
• The model is fed torch.arange(T) index positions, not real day offsets (USE_INDEX_POSITIONS = True), matching the official reference implementation exactly — real acquisition days are still carried by the dataset loader for analysis.
• Train/val IDs are hardcoded lists, not a fold filter, ported verbatim from the official script — duplicates are intentional (some patches count twice per epoch).
• Pretext task: per-frame NDVI-gated masking (STCLN.py:193-202, ported bit-identical — verified max|Δ| = 0.0 across 3 seeds). NDVI = (NIR − Red)/(NIR + Red); a frame is exempted from masking entirely if ≤90% of its pixels are vegetated. Measured on 20 fold-5 patches: the gate fires on 90.6% of frames, leaving 96.2% of all values visible.

---

Training configuration

	Pretrain	Finetune
Epochs	100	100
Optimizer	AdamW (wd=0, ≡ Adam)	AdamW (wd=0, ≡ Adam)
LR schedule	flat 1e-4	flat 1e-4, no scheduler
Grad clip	5.0	—
Batch	4 patches × 16 crops/patch	2 patches × 2 crops/patch
Loss	MSE reconstruction (masked)	segmentation loss on dual decoder
Mask ratio	0.4	—
Augmentation	—	none
Early stopping	—	disabled (patience 0)
Deep supervision	off	off
Mixed precision	AMP (torch.cuda.amp)	AMP (torch.cuda.amp)
Seed	3407 (official)	3407 (official)
Checkpoint cadence	every epoch (latest.tar) + milestone every 20	best-val + latest.tar every epoch

The fixed-epoch-99 checkpoint, not best-validation, is the reported primary result — see § Results for why.

T-SRDA-specific: KV reduction schedule R = [4, 4, 4] across 3 temporal blocks; 8 attention heads; d_model = 256; date positional encoding added before KV reduction (mandatory for irregular acquisition spacing); no internal gradient checkpointing in the temporal encoder (checkpointing is done at the chunk level in the encoder wrapper to avoid ~30% wasted backward compute from double-checkpointing).

---

Hardware & environment

GPU	1 × NVIDIA L4, 23.6 GB (23,034 MiB)
Driver / CUDA	550.127.08 / CUDA 12.4
PyTorch	2.5.1
Peak VRAM — pretrain	2.46 GB
Peak VRAM — finetune	1.21 GB
Peak VRAM — eval	16.86 GB at EVAL_BATCH=4 (121 batches); 9.50 GB at batch 2; 5.05 GB at batch 1 — metrics are identical at any eval batch size

Wall-clock time

Stage	Epochs	Total wall time	Per epoch
Pretraining	100	43.0 h	25.8 min (1,984 crop-steps/epoch)
Finetuning	100	60.3 min	~36 s
Full run (incl. preflight/data checks/eval)	—	44.3 h wall clock (2026-08-23 20:40 → 2026-08-25 16:57)	—

Pretraining dominates the budget by >40×; finetuning is nearly free once a pretrained encoder exists (relevant for the seed-variance runs recommended in § Next steps — they reuse the same pretrained encoder and cost ~1 h each, not another 43 h).

---

Results

Headline (test set: PASTIS fold 4, 482 full 128×128 patches, 3,961,114 scored pixels, no TTA)

Metric	T-SRDA (ep99, primary)	T-SRDA (ep37, reference)
mIoU	0.4124	0.4433
OA	0.7831	0.7720
mF1	0.5406	0.5704
Kappa	0.7319	0.7224

The primary number is the fixed-epoch-99 checkpoint (checkpoints/finetune/latest.tar), a deliberate protocol choice: best-checkpoint selection on only 152 validation crops was previously shown to manufacture a 147× variance artifact that vanished at fixed epoch, and the official reference implementation has no early stopping either. model_best.tar (epoch 37) is kept only as a reference point.

Comparison to baselines

System	mIoU	Selection	Seeds
Published STCLN	0.4843	—	—
E_s3m_nosemi	0.4835 ± 0.0006	best-val	3
A_linear	0.4805 ± 0.0071	best-val	3
A_linear (seed 3407)	0.4747	best-val	1
T-SRDA (ep37, best-val)	0.4433	best-val	1
T-SRDA (ep99, fixed — primary)	0.4124	fixed epoch	1

The only like-for-like comparison is T-SRDA's own best-val checkpoint against A_linear at the same seed: 0.4433 vs. 0.4747 (Δ = −0.0314) — about 4.4× A_linear's own seed-to-seed standard deviation, so unlikely to be pure seed noise, but not quantifiable with a single T-SRDA seed.

Per-class IoU / F1 — epoch 99 (primary)

Cls	Class	IoU	F1	Support (px)
1	Meadow	0.8057	0.8924	1,466,003
2	Soft winter wheat	0.7175	0.8355	560,680
3	Corn	0.8502	0.9190	688,289
4	Winter barley	0.4542	0.6247	163,392
5	Winter rapeseed	0.7637	0.8660	129,119
6	Spring barley	0.1772	0.3010	53,762
7	Sunflower	0.3169	0.4813	81,933
8	Grapevine	0.4765	0.6455	187,477
9	Beet	0.5828	0.7364	71,047
10	Winter triticale	0.0500	0.0953	58,362
11	Winter durum wheat	0.5220	0.6860	69,960
12	Fruits, vegetables, flowers	0.1914	0.3213	60,715
13	Potatoes	0.2199	0.3605	22,613
14	Leguminous fodder	0.2212	0.3623	127,794
15	Soybeans	0.5059	0.6719	68,316
16	Orchard	0.2841	0.4424	76,859
17	Mixed cereal	0.0873	0.1606	42,769
18	Sorghum	0.1964	0.3283	32,024

No dead classes — all 18 scored classes produce non-zero IoU. The four large, well-represented classes (Meadow, Corn, Soft winter wheat, Winter rapeseed) all land at 0.72–0.85 IoU; the deficit vs. baseline concentrates almost entirely in the confusable cereals (Spring barley, Mixed cereal, Winter triticale) — exactly the classes the baseline itself is least stable on across seeds.

---

Training dynamics

Pretraining — 100 epochs, 43.0 h — masked-reconstruction MSE

Epoch	0	19	39	59	79	99
loss	0.0246	0.0027	0.0017	0.0014	0.0012	0.0010

Converged cleanly and monotonically. No non-finite losses, no NaN-recovery events across all 198,400 optimizer steps.

Finetuning — 100 epochs, 60.3 min — validation mIoU

Epoch	0	10	20	30	37	40	50	60	70	80	90	99
val mIoU	0.1995	0.4259	0.4384	0.4319	0.4759	0.4546	0.4333	0.4185	0.4246	0.4395	0.4195	0.4207

Validation mIoU peaks early (epoch 37, 0.4759) and then declines — consistent with overfitting to the 152 training crops. The last 20 epochs plateau at mean 0.4299 (sd 0.0156), and test mIoU tracks the same direction (0.4433 at ep37 → 0.4124 at ep99), though n=2 is too few points to draw a general conclusion from.

---

Repository contents

tsrda/
├── README.md                    this file
├── RESULTS.md                   full write-up this card is derived from
├── PATCH_model.md                notes on the 3-edit swap from DaViT → T-SRDA
├── config.py                     all hyperparameters, paths, PASTIS class table
├── model.py                      spatial encoder, T-SRDA wrapper, pretrain/finetune heads
├── temporal_srda.py              TemporalSRDAEncoder / TemporalSRDABlock
├── dataset.py                    PASTIS loading, cropping, collation
├── losses.py                     segmentation loss(es)
├── pretrain.py / finetune.py     training loops
├── evaluate.py                   test-set evaluation (mIoU/OA/F1/Kappa, per-class)
├── check_data.py / check_split.py / preflight.py   protocol/verification gates
├── smoke_model.py                shape + VRAM smoke test
├── run.sh / run_all.sh / launch.sh / status.sh      pipeline orchestration (tmux, resumable)
│
├── checkpoints/
│   ├── pretrain/                 checkpoint_{0,19,39,59,79,99}.tar (+ .utae.tar encoder-only)
│   │                              latest.tar / latest.utae.tar      (~476 MB total)
│   └── finetune/
│       ├── latest.tar             epoch 99 — PRIMARY reported checkpoint
│       └── model_best.tar         epoch 37 — best-val, reference only
│
├── runs/20260823_204044/         raw logs for this run
│   ├── preflight_log.txt          protocol gates
│   ├── check_data_log.txt         data assertions
│   ├── smoke_log.txt              VRAM + shape checks
│   ├── pretrain_log.txt           100 epochs
│   ├── finetune_log.txt           100 epochs, per-epoch metrics
│   ├── eval_ep99_PRIMARY_log.txt  the reported test numbers
│   └── eval_best_REFERENCE_log.txt
│
├── logs_prev_machine/            pre-port logs, kept for audit trail
└── run_all_console.txt           full console transcript of the run

Checkpoint files

File	Stage	Epoch	Params	Notes
checkpoints/pretrain/checkpoint_{0,19,39,59,79,99}.tar	pretrain	milestone	4.01M	full pretrain graph (encoder + recon heads) + optimizer/scaler/RNG state
checkpoints/pretrain/checkpoint_*.utae.tar	pretrain	milestone	—	encoder-only weights, for loading into finetune
checkpoints/pretrain/latest.tar / .utae.tar	pretrain	99 (resumable)	4.01M	written every epoch
checkpoints/finetune/latest.tar	finetune	99 — primary	6.02M	the checkpoint RESULTS.md reports
checkpoints/finetune/model_best.tar	finetune	37 — reference	6.02M	best validation mIoU, kept for comparison only

---

Reproducing this run

Requires the PASTIS dataset locally (PASTIS_ROOT env var or a sibling PASTIS/ directory containing metadata.geojson).

git clone https://huggingface.co/Dhruv1000/TSRDA
cd TSRDA

# gates only
python3 preflight.py

# full pipeline (pretrain → finetune → eval), tmux-backed and resumable
bash launch.sh
bash status.sh          # check progress without attaching

# evaluate an existing checkpoint
python3 evaluate.py --ckpt checkpoints/finetune/latest.tar

# additional finetune seeds from the already-pretrained encoder (~1h each)
python3 finetune.py --pretrain_pth checkpoints/pretrain/checkpoint_99.utae.tar \
  --seed 42   --tag seed42
python3 finetune.py --pretrain_pth checkpoints/pretrain/checkpoint_99.utae.tar \
  --seed 1234 --tag seed1234

If evaluation OOMs on a smaller/shared GPU, rerun with EVAL_BATCH=2 (9.5 GB peak) or EVAL_BATCH=1 (5.05 GB peak) — metrics are numerically identical at any batch size (full-patch inference has no batch-dependent ops in eval mode).

Verification gates (all passed, re-runnable via preflight.py)

Check	Result
Class nomenclature vs. official PASTIS list	PASS — 20/20
Cross-check vs. PhenoProto splits.PASTIS_CLASSES	PASS
Installed masker vs. STCLN.py:193-202	bit-identical, max|Δ|=0 across 3 seeds
Eval chunking vs. unchunked	bit-identical, max|Δlogit|=0 on 5 patches
All submodules in eval mode after .eval()	PASS
Protocol assertions (folds, IDs, step counts, disjointness)	PASS — all

---

Honest summary & next steps

The port to the official protocol worked cleanly: fold-disjoint splits, exact step counts, a bit-identical pretext task, no dead classes, stable convergence, and a fixed-epoch primary metric that doesn't depend on validation noise. The measurement is sound.

The measurement says T-SRDA is 0.031 mIoU behind the simplest correct baseline at matched checkpoint selection and one seed — concentrated almost entirely in three cereal classes that the baseline itself is least stable on across seeds. This is a negative result, reported as measured, not a claim that the architecture is broken: it trains stably and lands in the same 0.41–0.48 mIoU band as every other arm of this project, it just doesn't win.

Ranked next steps (by information gained per GPU-hour, cheapest first):

1. Two more finetune seeds (42, 1234), ~2 GPU-hours — reuses the already-pretrained encoder. Without this, the −0.031 gap can't be told apart from a bad draw on unstable classes. Highest-value next run.
2. Remove the decoder confound, ~1 GPU-hour — finetune the same encoder with a plain linear head instead of the dual+boundary decoder.
3. Earlier-epoch encoders, ~1 GPU-hour each — milestones at epochs 19, 39, 59, 79 are already on disk; tests whether 43h of pretraining was even necessary.
4. Real date positions (USE_INDEX_POSITIONS=False) — another full 43h pretrain, so only worth it after 1–3 narrow things down. This is the one protocol choice that arguably handicaps T-SRDA specifically, since its design assumes date PE is available before KV reduction.

Citation

This repository ports and evaluates the SRDA attention mechanism from:

@article{zhou2026pdavit,
  title   = {PDAViT: Spatial-Reduction Dual-Attention Vision Transformer},
  author  = {Zhou, et al.},
  journal = {Neurocomputing},
  year    = {2026}
}

against the official STCLN PASTIS protocol and baselines:

@misc{stcln,
  title = {STCLN},
  howpublished = {\url{https://github.com/XiaoleiQinn/STCLN}}
}

on the PASTIS benchmark:

@inproceedings{garnot2021pastis,
  title     = {Panoptic Segmentation of Satellite Image Time Series with
               Convolutional Temporal Attention Networks},
  author    = {Sainte Fare Garnot, Vivien and Landrieu, Loic},
  booktitle = {ICCV},
  year      = {2021}
}