DUNIA: Pixel-Sized Embeddings via Cross-Modal Alignment for Earth Observation Applications
Authors: Ibrahim Fayad
Max Zimmer
Martin Schwartz
Fabian Gieseke
Philippe CIAIS
Gabriel Belouze
Sarah Brood
Aurélien de Truchis
Alexandre d'Aspremont
Venue: ICML 2025 poster (2025)
Keywords: earth observation
multi-modality
self-supervised learning
cross-modal retrieval
Primary area: applications->chemistry_physics_and_earth_sciences

Abstract: Significant efforts have been directed towards adapting self-supervised multimodal learning for Earth observation applications. However, most current methods produce coarse patch-sized embeddings, limiting their effectiveness and integration with other modalities like LiDAR. To close this gap, we present DUNIA, an approach to learn pixel-sized embeddings through cross-modal alignment between images and full-waveform LiDAR data. As the model is trained in a contrastive manner, the embeddings can be directly leveraged in the context of a variety of environmental monitoring tasks in a zero-shot setting. In our experiments, we demonstrate the effectiveness of the embeddings for seven such tasks: canopy height mapping, fractional canopy cover, land cover mapping, tree species identification, plant area index, crop type classification, and per-pixel waveform-based vertical structure mapping. The results show that the embeddings, along with zero-shot classifiers, often outperform specialized supervised models, even in low-data regimes. In the fine-tuning setting, we show strong performances near or better than the state-of-the-art on five out of six tasks.

DUNIA: Pixel-Sized Embeddings via Cross-Modal Alignment for Earth Observation Applications
Ibrahim Fayad 12 Max 3 Martín Schwartz'
Fabian Gieseke* Philippe Ciais
Gabriel Belouze
Sarah Brood
Aurelien De Truchis? Alexandre d' Aspremont
Abstract
Model
Inference
Land Cover
Tree Species
Significant efforts have been directed towards adapting self-supervised multimodal learning for Earth observation applications. However, most current methods produce coarse patch-sized em- beddings, limiting their effectiveness and inte- gration with other modalities like LiDAR. To close this gap, present DUNIA, ap- proach to learn pixel-sized embeddings through cross-modal alignment between images and full- waveform LiDAR data. As the model is trained in a contrastive manner, the embeddings can be di- rectly leveraged in the context of variety of envi- ronmental monitoring tasks in zero-shot setting. In our experiments, we demonstrate the effec- tiveness of the embeddings for seven such tasks: canopy height mapping, fractional canopy cover. land cover mapping, tree species identification, plant area index, crop type classification, and per- pixel waveform- vertical structure mapping. The results show that the embeddings, along with zero-shot classifiers, often outperform specialized supervised models, even in low-data regimes. In the fine-tuning setting, we show strong perfor- mances near or better than the state-of-the-art on five out of six tasks.
Timeseries
Decoder
Heighre
Cover
Encoder
Singl image
Decoder
Encoder
Decoder
Embedding Databases
VVI
Vertical information (VI)
Aligument
Embedding
Dense
Query
Embeddings
Figure f. DUNIA'S alignment strategy leverages temporal and tical information at the pixel level, enabling the model to develop a cross mossl understanding from a single satellite image. This ap- proach serves as foundation for cross-modal retrieval tasks across diverse EO applications. The pre-trained model represented navy bluc, the multi nulti-temporal autoencoder (AE) in yellow, and the waveform AE in green.
for key EO applications, ranging from monitoring natural resources (Chen al., 2023; Fayad et al., 2024; Li et al., 2023; Liu al., 2022; 2023; Tolan al., 2024; Lang et al.,
2023; Pauls et al., 2024; Schwartz et al., 2023; Pauls et al.,
2025) and assessing environmental impacts (Dalagnol et al., 2023; Wagner et al., 2023), to climate monitoring and fore- cast (Andrychowicz et al., 2023; Schultz et al., 2021), I), as well as agricultural and land resource management (Ienco et al., 2019; Kussul et al., 2017: Zhang et al., 2019).
1. Introduction
While research of deep learning architectures and up- coming remote satellite missions is expected to enhance ex- isting BO applications, supervised deep learning approaches have key limitations. First, these models depend on labeled data, often requiring large annotated datasets. For exam- ple, the estimation of above -hove-ground biomass (AGB) still heavily relies on traditional methods due to limited ground- based measurements and the need to use other variables as proxies (Morin et al., 2023; Schwartz et al., 2023).
With the rapid expansion of Earth Observation (EO) satellite missions, deep learning has emerged as a powerful solution
'Laboratoire des Sciences du Climat et de l'Environnement,, LSCE/IPSL, France *Kayrros SAS, Paris 75009, France 2 'Department for in Society, Science, and Technology, Zuse In- stitute Berlin, Germany 'Department of Information Systems, Uni- versity of Münster, Germany Department of Computer Science, CNRS, INRIA & École Normale Supéricure, Paris 75230, France. Correspondence to: Ibrahim Fayad lpp..iisi.......r.........
Second, EO models are often tailored to specific tasks, lim- iting their adaptability and reusability, even when using similar input data sources. For instance, a land cover classi- fication model cannot easily predict canopy height due to differences in input-output relationships, as the network may
Proceedings of the 4nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

DUNIA: Pixel-Sized Embeddings via Cross-Modal Alignment for Earth Observation Applications
prioritize learning features that are highly discriminative for given task but not the other.
2. Background & Contribution
We sketch the related works along with our contributions.
Third, EO models often use optical or radar imagery to predict single-valued targets like canopy height or land cover classes. However, they struggle with more complex outputs like the full vertical structure of vegetation."
2.1. Contrastive Learning
Contrastive learning trains networks to produce similar em- beddings for semantically related inputs and dissimilar em- beddings for unrelated ones. Related inputs may include different augmentations of the same sample or paired exam- ples from different modalities. In this context, one of the main challenges is to avoid model collapse, where model outputs become constant. This is typically addressed by: 1) architectural designs or 2) specialized objective functions.
Foundation models (FMs) offer promising solution to the aforementioned limitations by leveraging self-supervised pre-training on vast amounts of unlabeled data (Brown et al., 2020; Devlin, 2018; Kirillov et al., 2023; Minderer et al., 2022; Radford et al., 2021). For EO applications, such models are being developed to process diverse data types, including time series processing (Yuan et al., 2022), adapta- tion to different input satellite sensors (Xiong et al., 2024) or varying Ground Sample Distances (GSD) (Reed al., 2023), and multi-modal data fusion (Astruc et al., 2025; 2024; Fuller et al., 2024). Most of current foundation mod- els primarily address the adaptability issue, but not label scarcity. In fact, most EO-focused FMs are pre-trained masked auto-encoders (MAEs), which require extensive fine-tuning (Lehner et al., 2024; Singh et al., 2023). More- over, existing FMs, even multi-modal ones, only leverage image-based modalities. As such, they excel in EO tasks re- quiring horizontal structural understanding (e.g., land cover mapping, tree species identification). but struggle with EO applications requiring vertical structure understanding (e.g.. canopy height estimation). Additionally, their patch-based approach limits direct pixel-level prediction capabilities. This hinders the integration of LiDAR data, such as full- waveform LiDAR data available labels at resolutions higher than the patch scale.
Prominent examples of architectural designs are BYOL (Grill et al., 2020) and SimSiam (Chen & He, 2021), which respectively use asymmetry to prevent collapse, BYOL employing a momentum encoder and SimSiam using stop-gradient operation with a predictor network. Specialized objective functions can be categorized into pair-based contrastive losses, clustering losses, and Segative--pir--ae losses. Here, pair-based losses, such as those presented in SimCLR (Chen et al., 2020) and MoCo (He et al., 2020), align positive pairs and separate negative pairs, but assume unique positive pairs within mini-batch, which is not always true for EO data, especially at high resolutions. Clustering methods like SeLa (Asano et al., 2019) and SwAV (Caron et al., 2020) relax this constraint by optimizing a contrastive objective over cluster assignments instead. Negative- pair-free methods, such as Barlow Twins (Zbontar et al., 2021) and VICReg (Bardes et al., 2021), focus on redundancy reduction in the feature dimension. Zero-CL (Zhang et al., 2021) uses whitening transformations on the embeddings to maximize the trace of the cross-covariance matrix, achieving alignment and redundancy reduction without negative pairs.
In this work, we propose Dense Unsupervised Nature Interpretation Algorithum (DUNIA) that learns to generate pixel-level embeddings by aligning vertical (from a space- borne full waveform LiDAR) and horizontal structure infor- mation with satellite imagery through contrastive learning. This dual alignment strategy enables understanding of both vertical and horizontal structures, supporting diverse EO ap- plications with minimal, if any, training. DUNIA achieves strong retrieval performance for several tasks and excels in forest structure benchmarks. In our experiments, we demon- strate the effectiveness of the resulting embeddings for seven key EO applications and show that zero-shot classification operating on these embeddings can outperform specialized supervised EO models in several cases. A simplified version of our approach is presented in Figure 1.
2.2. Self-Supervised Learning for EO
Inspired by masked image modeling (MIM) from vision transformers, several SSL frameworks have adapted MAE- style objectives to the remote sensing domain. These methods mainly focus on reconstructing masked inputs to learn spatial, spectral, and temporal representations. Sat- MAE (Cong al., 2022) introduces a spectral-tempprall aware masking scheme for Sentinel-2 (S-2) timeseries. Sat- MAE++ (Noman et al., 2024) extends SatMAE by incor- porating multi-scale feature extraction. Scale-MAE (Reed et al., 2023) embeds spatial resolution information into po- sitional encodings for multiscale imagery. DOFA (Xiong et al., 2024) extends masked autoencoding with dynamic, awere weighting mechanism, enabling the model to encode diverse sensor modalities through shared encoder.
'Capturing such vertical structures is essential for understand- ing biomass allocation and species diversity. Predicting vertical structure from EO imagery is challenging due to its complexity and limited sensor data, requiring cross-modal understanding over direct prediction (Tan et al., 2024).
Other related works on SSL for EO applications relies on
2

---

DUNIA: Pixel-Sized Embeddings via Cross-Modal Alignment for Earth Observation Applications
contrastive objective to capture the relationships within the data. CROMA (Fuller et al., 2023) aligns radar and optical modalities using cross-modal contrastive loss, sup- ported by an auxiliary reconstruction head. DeCUR (Wang et al., 2024) explicitly disentangles intra- and inter-modal contrastive signals based on Barlow Twins (Zbontar et al., 2021). AnySat (Astruc et al., 2024) proposed a resolution- and temporal-adaptive framework capable of handling data from various sensors and resolutions, effectively combining multiple EO data sources into shared embedding space.
cover understanding, it performs competitively in zero-shot and low-shot evaluations. Additionally, pixel embeddings can directly generate waveforms representing forest vertical structures - a task not possible with existing methods.
3. Approach
Our objective is to learn two distinct pixel-level embed- ding spaces corresponding to two EO data modalities vertical and horizontal using a single pre-trained model. This enables simultaneous understanding of both horizon- tal and vertical structures. Vertical information is derived from the Global Ecosystem Dynamics Investigation (GEDI) instrument (Dubayah et al., 2020), a spaceborne LiDAR that provides sparse measurements of 3D structures via 1D waveform signals. Horizontal information comes from 10 m resolution imagery acquired by the S-1 & 2 missions.
2.3. Multi-Modal Learning
Multi-modal SSL approaches can be categorized into three main categories. (I) agdalitya models use shared network for different modalities (Carreira et al., 2022; Gird- har et al., 2022; Jaegle et al., 2021), but are typically limited to one modality at a time during training, restricting cross- modal knowledge sharing (Srivastava & Sharma, 2024). For EO applications, several models follow this approach. Scale- MAE (Reed et al., 2023) and DOFA (Xiong al., 2024) both process imagery across varying resolutions, the lat- ter also processes different wavelengths using a common encoder. (2) Fusion-encode, models integrate data from multiple modalities through cross-modal attention, effec- tively combining information from each modality (Bach- mann al., 2022; Bao et al., 2021; Singh et al., 2022). Om- niSat (Astruc al., 2025) and AnySat (Astruc et al., 2024) exemplify this by integrating data from diverse EO sources into a unified representation.
While our framework can be implemented in various ways, we adopt a modular design centered around a transformer encoder and two independent convolutional decoders. This encoder-decoder architecture enables us to generate pixel- sized embeddings for each input. The two decoders are designed to disentangle horizontal and vertical structure understanding: one focuses spatial (horizontal) relation- ships, the other on vertical structure.
For horizontal structure alignment, we use a multi-temporal autoencoder (AB) that encodes a sequence of satellite im- ages and produces temporally informed, per-pixel embed- dings. This design achieves two goals: 1) it provides an augmented input to the pre-trained model, and 2) it allows the model to implicitly capture phenological patterns with- out requiring a time series at inference time.
(3) Finally, multi-encoder ap- proaches use separate encoders for each modality, which are then aligned in a shared latent space. This strategy is highly effective for tasks such as cross- modal retrieval and zero-shot classification across different modalities: im- age/text (Radford et al., 2021; Jia al., 2021; Yuan al., 2021; Yu et al., 2022), audio/text (Guzhov et al., 2022), and video/text (Luo et al., 2022; Pei et al., 2023).
For vertical structure alignment, we also use an AE, this time encoding GEDI waveforms. The waveform encoder projects the 1D input into a latent space, which is aligned with the corresponding pixel embedding from the pre-trained model. While the decoder is not needed for the aligmment itself, it is necessary for waveform generation, ensuring that the latent space remains semantically and structurally meaningful.
2.4. Contribution
Our work aligns closely with the abovementioned third cat- egory, focusing on mono- and modal retrieval within the multi-encoder approach. Unlike prior models that align paired data at the instance level (e.g., matching an image to its textual description), here we perform pixel-level align- ment. This is necessary to preserve the spatial resolution required to mathh with full LiDAR waveform footprints.
Figure illustrates the main components of our approach, in- cluding the inputs to each model, their architectural building blocks, and the layers used for alignment. Below, we de- scribe the main modules in more detail, followed by our pre- training objectives and the latent diffusion model for wave- form generation. Full details are provided in Appendix A.
Specifically, our approach aligns sparse LiDAR waveforms with high-resolution imagery through contrastive learning to understand the vertical structure (i.e., pixel- waveform align- ment) and simultaneously performs pixel-pixel alignment for horizontal structure understanding. DUNIA enables per- pixel vertical and horizontal structure retrieval and surpasses specialized models in forest structure benchmarks. For land
3.1. Pre-trained Model
The pre-trained model in DUNIA (Figure 2) takes an im- age I E R and produces pixel level embeddings in # X,, C' is the number of input channels, (H, W) is the image resolution, and D, is the target projection di-
3

---

DUNIA: Pixel-Sized Embeddings via Cross-Modal Alignment for Earth Observation Applications
x
HxW W Dr
O7
GxL Hx XxHxWxO
HxWx x Dy
M
0
ON
HXW Dr
I
H D
O
0
Location to waveform mapper
H W x DD
Legend
encoderidecoder blocks
Three dates 5-182 Imput mosatcs
GxD,
Block transfommer layers
Single date 3-182 inpuri mossic
Transforeer decoder blocks
GEDI waveform location
Resnet encoder block
Alggmmrt
Figure 2. Simplified architectural overview of the proposed framework. The figure illustrates the key encoder and decoder components, input modalities, and the layers where alignment is performed (red dashed arrows). The pre-trained model is shaded blue. The multi-temporal image AE, is shaded in yellow. The waveform encoder is shaded in green. Additional components, losses and training objectives are omitted for clarity; see Figure Al for full details.
mension. is median composite image generated from S-1 & observations over several dates. The pre-trained model comprises five building blocks: patch embedding, shared encoder, two similar decoders, neighborhood at- tention layers and several projection heads. Following is brief description of each of these blocks.
samples features while combining information from differ- ent encoder layers to maintain both fine and coarse details, Each decoder block E {1, 3, 4), with d represent- ing the final decoder layer) produces image-like feature H map of shape प.- X z X Dpe. where Dp 2d D, is the feature dimension out of block d. We refer to the out- put at these stages for given decoder as Og. For simplicity, we will henceforth refer to O as O.
W
Patch Embedding. The 2D composite image I split into N HW/P² patches, each of dimension using convolutional layer, with P and D) the patch size and the patch embedding dimension respectively.
Neighborhood Attention (NA). To enhance local spatial relationships, we add two NA layers (Hassani et al., 2023) for each decoder block the highest resolution (d 1), where each pixel attends to its surrounding window size This complements the global attention from the transformer layers with explicit element relation modeling mechanism.
Encoder Architecture. The encoder consists of 16 stan- dard transformer layers (Vaswani et al., 2017), organized into four Transformer Blocks, of four layers each. Each layer inputs/outputs sequence of image tokens in N)
Projection Head. For the pixel-pixel alignment objective, we follow standard practice and append projection head for each output of OH As the projection head becomes more specialized towards the training objective (Xue et al., 2024), its output becomes less generalizable when training and downstream objectives are misaligned, and as such
The two decoders transform the encoder outputs into two sets of pixel-sized embeddings (OV, OH E #x DD) for wixel-waveform and pixel- pixel alignment respectively. They use a hierarchical struc- ture inspired by Ranftl et al. (2021) that progressively up-
Decoder Architecture.

---

DUNIA: Pixel-Sized Embeddings via Cross-Modal Alignment for Earth Observation Applications
discarded after training. For the pixel-waveform alignment Oy no projection head is used the downstream tasks (e.g.. waveform generation) align with the training objective.
inference, or mono-temporal models limited single date, is trade-off that enables the model to capture some phenological traits without needing time series data during inference. Indeed, while image composites such as a median composite may be richer than single-date images, they are still less informative than a full time series, as they only provide a median reflectance value over a given period. Conversely, even this simple form of aggregation preserves significantly more information than a single-date image.
3.2. Waveform Model
The waveform AE is based VQ-VAE (Vector Quan- tized Variational Autocncoder) (van den Oord et al., 2017) architecture. It consists of three primary components: waveform encoder (Figure 2), a residual vector quantizer (RVQ), and a waveform decoder. The input to this model are GEDI LiDAR waveforms (W) available only for certain pixel locations in the composite image I.
Multi-temporal Image AE. This AE follows a UNet structure, but replaces conventional convolutional blocks with ConvLSTM (Shi al., 2015) layers to explicidl model temporal correlations. The model takes a multi- temporal input image M and produces a feature map which is then reconstructed back to a tensor M of the same shape as the inputs. Each de- coder block produces an image-like feature map of shape H W denoted as Xa for respective decoder stage. 1- 15
Waveform Encoder. The waveform encoder (Ew) uses ResNet-based architecture to process ID waveform inputs through multiple stages, progressively reducing their length while increasing feature representation capacity. & trans- forms a waveform (W) shape 4x1 into a latent represen- tation 2e E L/16116 To align ze with its corresponding pixel embedding from OV we perform a channel-wise aver- age pooling followed by linear projection. This process transforms into OW E Dp.
X' E
To align the output embeddings from the multi-temporal image AE with those from the pre- training model, we first perform temporal average pool-
Temporal Pooling.
Pooling
Residual Vector Quantizer (RVQ). For regularization by enforcing discrete representations, we use an RVQ layer with Q quantizers each containing V codebook vectors. The RVQ iteratively quantizes the latent waveform repre- sentation z. through a sequence of quantizers, with each processing the residual from the previous one to produce the final quantized representation 2.
ing. This transforms Xss E K x²'xd"-y H x "d= xD, to OJ E R. x zd' xDp which then passes through a projection head. For the output with the highest resolu- tion (i.e., of) we also append two NA layers before the projection head. For simplicity refer to Of as OT.
3.4. Pre- Pre-Training Objective
Waveform Decoder. The waveform decoder Dw mirrors the structure of the encoder but operates in reverse order to reconstruct the waveform from its quantized latent rep- resentation z. In essence, (z) = W. While waveform reconstruction is not necessary for the alignment, recon- structing directly from aligned encoded waveforms ensures that the future generation process operates on waveform latent space that is already structured based on the shared semantic and structural alignment with the pixel embed- dings. This approach is beneficial as we are interested in generating waveforms given pixel inputs.
DUNIA is pre-trained pixel-waveform and pixel-pixel alignment, alongside modality reconstruction for the modality-specific AFs. For the alignment task, we rely on two, similarly performing, non-negative-pair contrastive losses, namely VICReg (Bardes et al., 2021) and Zero-CL (Zhang et al., 2021). The selection between either loss func- tions determined by the number of available elements within mini-batch and the requirements of each loss func- tion. VICReg requires large batch sizes to perform well (Bardes et al., 2021), which is a non-issue for the pixel-pixel contrastive objective given the high number of pixels avail- able in each mini-batch of images. However, this is not suitable for the waveforms which are few in number within the mini- On the other hand, Zero-CL performs well even for small batch sizes but faces computational bottle- necks with very large batches.
3.3. Multitemporal Image Processing Model
The multitemporal image model (Figure 2) is designed to process temporal sequences of median composite images, following an autoencoder (AE) architecture. The input to this model is a sequence of images M E RT**>xXxx X covering the same area as the input to the pre-training model. These images come from T S-1 & 2 acquisition dates that overlap with the dates used to generate the composite image pre- pre-training
1.
ALIGNMENT
Zero-CL replaces the alignment and uniformity terms in negative-paii pair-based contrastive losses (Arora et al., 2019) with, respectively, an instance-wise contrastive loss (Cmss and a feature- wise contrastive loss (Lre)). The overall
I. This design choice, unlike models that require time series data during training and
5

---

DUNIA: Pixel-Sized Embeddings via Cross-Modal Alignment for Earth Observation Applications
loss Ce CL is Leea + Liss- Le is applied on ZA E RG the L22 normalized pixel embeddings from O², and = E R their corresponding L2 normalized waveform embeddings from OW, where G is the number of available GEDI samples in given mini-batch. The formu- lation for this loss can be found in Appendix B.
of the quantized waveform obtained from the RVQ layer (Section 3.2). During inference, the waveform generation follows two-step process. First we sample representation given condition Oy using the LDM, and then the frozen Dar (z) yields a waveform.
For a detailed description of the diffusion process, including the denoising objective, noise schedule, and latent variable sampling, we refer the reader Appendix C.
3.4.2. PIXEI--PIXES ALIGNMENT
VICReg formulated as three loss terms: variance (Lvar), in- variance (C and covariance (Ceor). Let Zz E R Maxax Pau represent the L2 normalized pixel embeddings from OH,, Ma- = BBx x 2) x W 2(d-1) with B the mini-batch size, 1 and 2 the height and width of decoder's out- put feature map. zī= E Max is their corresponding T2 normalized pixel embeddings from OT- The overall hierarchical pixel-pixel alignment loss is expressed as:
4. Experimental Evaluation
H
We provide a detailed experimental evaluation showing that DUNIA is able to achieve high performance across a vari- ety of tasks in zero-shot and fine-tuned settings, including land cover classification, crop mapping, and vertical for- est structure analysis (e.g., canopy cover, height and GEDI waveform retrieval). We leverage diverse datasets to inform the model on horizontal structures - S-1 & 2, and vertical structures with GEDI waveforms.
M
W
LVICReg
Σ d=1
Cv ( ( k3)
+ B. (2 7/z)
The following section contains a summary of the experimen- tal setup, with full details in the appendix. Model config- uration and optimisation can be found in Appendix E. For clarity, during the pre-training phase, we set the input image size to 64 x 64 pixels, with 14 channels from stacked S-1 & 2 image composites. The embedding dimension is set to 64 (i.e., D, = 64). During inference, the input image can be of any size, but we inferred on 256 x 256 pixel images. Our code is available at github. com / AI /oresst/DONIA.
Yc (2 233
(1)
Following Bardes (2021) we set an and B; to 25.0 and To to
1.0. VICReg formulation can be found in Appendix B.
3.4.3. RECONSTRUCTION LOSSES
In addition to the previously defined contrastive losses, our pre-training objective also has three additional reconstruc- tion losses for the modality-speciif AEs (i.e., the waveform AB and the multi-temppral image AE) and on the outputs of OH for regularization. The reconstruction losses are simply the mean squared error loss (MSE) on the reconstructed waveform W RIXL given input waveform W. the recon- structed multi-temporal image M E RT*H>W> xC given input image M, and the reconstructed mono-temporal im- age I E R. ff x w x given input image I. The reconstruction
4.1. Experimental Setup
For our evaluation, we resort to various datasets and tasks. The datasets and experimental details are described next.
4.1.1. DATASETS
Pre-training Datasets. We used S-2 Level-2A surface reflectance data from Google Earth Engine, including 10 m and 20 m spatial resolution bands with the latter upscaled to 10 m. Two sets of mosaics were created over the entire metropolitan French territory: a single leaf-on season mo- saic (April-
term is formulated as:
Sree = MSEWW, + MM
+ MSE(I,
2020) for the pre-trained model, and three four-month mosaics (October 1019-September 2020) for the multitemporal AE. Cloud filtering was applied using the S-2 Cloud Probability dataset.
3.5. Waveform Generation
To generate a full GEDI waveform given a pixel input, we leverage latent diffusion models (LDMs) (Rombach et al., 2022). LDMs are generative models that iteratively refine a latent representation, starting from normally distributed prior in a learned latent space. These models can also be conditioned on auxiliary inputs y model conditional dis- tributions p(zly). In our case, the conditioning variable con- sists of pixel embedding OY a.a at coordinate (ø, A), while
S-1 data were obtained from the S-1A and S- satellites op- erating at C-band, collected in Interferometric Wide swath mode with VV and VH polarizations. The data was cal- ibrated, geometrically corrected, and resampled to 10 m resolution. Similar S-2, we created two sets of mosaies with normalized backscattering coefficients.
GEDI is full-waveform LiDAR sensor on the International Space Station (ISS), operational between 51.6 N and 51.6*
the latent variable 2 E T L/16x16 represents the embedding
6

---

DUNIA: Pixel-Sized Embeddings via Cross-Modal Alignment for Earth Observation Applications
S from 2019-2023. We used Level IB, 2A, and 2B data from April 2019 to December 2021, extracting waveforms, geolocation, height metrics (Wra), canopy cover (Wo), and Plant Area Index par)). After quality filtering following Fayad et al. (2024) and seasonal selection, the dataset con- tained RV 19 million waveforms, covering less than I 1% of the total surface area of France.
as queries, creating a separate database for each target as not all targets were available all pixels simultaneously. Here, keys are L2 normalized pixel embeddings from O with targets being a land cover class (i.e., CLC+), crop type (i.e., PASTTS), or tree species (i.e., PF). For tree species identification, the labels cover a x 5 pixel area, so queries and keys are the averaged embeddings over this window. Next, given an input image and its Le normalized pixel embeddings from O² or OH,, we retrieve the k nearest neighbors (KNN) for each pixel based on cosine similarity voting. For
Overall, our pre-training dataset consisted of 836K 64 x 64 pixels S-1 & 2 images with, on average, 26 GEDI wave- forms per image. Additional details on the pre-training datasets can be found in Appendix F.
and assign the target class by distance- the KNN retrieval, keys were obtained from the training split while the queries were obtained from the test split.
Evaluation Datasets. We evaluated our model on seven downstream tasks using labels from various data products at resolutions matching or lower than our model's s output (10 m). Porest (PF) provides a benchmark dataset of ground truth patches for classifying mono-specific forests in France, featuring high-resolution imagery and annotations for over 135K 50x x 50 m (i.e., 5x5 pixels) patches across Backbone (CI.S+) J is a pan-Furopean land cover inventory for 2021, utilizing S-2 time series (2020-2022) and a TempCNN clas- sifier (Pelletier et al., 2019) to produce a 10 m raster indi- cating the dominant land cover among 11 classes. PASTIS is crop mapping dataset by Garnot et al. (2021), covering 18 crop classes and I background class with 2433 densely annotated 128 x 128 pixels images at 10 m resolution. The vertical structure dataset assesses model performance in mapping forest heights, canopy cover, plant area index, and waveform retrieval at 10 m resolution, relying on GEDI- derived products as presented earlier. When available, we used the train/val/test split used by the references (i.e., PF and PASTIS). For the CLS+ dataset, we used the same split as the unsupervised dataset, which followed a 65/10/25 split.
For the low-shot fine-tuning, we froze the entire pre-trained network except for the last NA layer in each decoder, which were appended with an output head consisting of two se- quential 1 x 1 convolutional layers. The first layer halves the input channels, and the second projects the reduced representation to the desired output size.
13 tree species (Gaydon & Roche, 2024).
We evaluated zero-shot classification and low-shot fine- tuning in a low-data regime. We define a dataset with a low number of labels based on the type of labels usually available for this dataset. For CLC+ and PASTIS, la- beled data are available as densely annotated images. For PF, W. Wrr. We and Wpus, labeled data are available as single annotated pixels.
4.1.3. COMPETING MODELS.
We compared DUNIA in the fine-tuned setting to five cur- rent state-of-the-arr Earth Observation FMs: SatMAE (Cong et al., 2022), DOFA (Xiong et al., 2024), DeCUR (Wang et al., 2024), CROMA (Fuller et al., 2023) and AnySat (Astruc et al., 2024). SatMAE takes as input S-2 imagery, DOFA, DeCUR and CROMA take as input S-1 & 2 im- agery, while Anysat is pre-trained using S-1 & 2 times series as well as very high-resolution imagery. For a fair comparison, we pre-trained all competing models (using pre- trained weights when available) for 250K steps using our datasets and the training details from the respective papers. For AnySat, we used multi-temporal S-1 & 2 mosaics with three time steps and included SPOT images at 1.5 m reso- lution as an additional input modality during pre-training but fine-tuned using only S-1 & 2. All competing models were evaluated on all downstream tasks except waveform generation, they do not support this task.
4. 1.2. PERFORMANCE EVALUATION
We evaluated our model's 's retrieval capacities for both zero- shot dense prediction tasks and its performance in the fine- tuned setting. For both settings, we used the weighted F1 score (wF1) for classification tasks, root mean squared error (RMSE) and Pearson's correlation coefficient (r) for re- gression tasks. To evaluate the similarity between acquired and retrieveddgenerrated waveforms, we used Pearson's cor- relation coefficient, computed between the time-aligned retrieved/generated waveforms and the acquired waveforms.
For zero-shot classification, we constructed retrieval databases based on the downstream tasks. For vertical structure tasks, outputs from OV served as queries, and we created single database with L2 normalized waveform embeddings from OW as keys, paired with four target la- bels: Wr., We. Wpass and the complete waveform (W). For horizontal structure tasks, outputs from OH were used
4.2. Results
Our evaluation shows that embeddings from our proposed framework, combined with simple zero-shot classifiers, sur- pass specialized supervised models in several tasks. In the fine-tuned setting, our model demonstrates strong low-shot performance, rivaling or exceeding state-of-the-a methods.
7

---

DUNIA: Pixel-Sized Embeddings via Cross-Modal Alignment for Earth Observation Applications
Table 1. Top-1 retrieval-based zero-shot classification performance of DUNTA for different retrieval settings. SOTA results, when available, represent the best- performie supervised models for any given dataset. are retrained models for a particular dataset. and are DUNIA query embeddings from O and O* respectively. represents the number of samples in retrieval database, with meaning a 64 64 pixels fully annotated image, and / meaning single annotated pixel. KNN represents the k nearest neighbors used in the wesgttee-weighted voting. W represents performance results for vertical structures higher than 5 m. Best scores are bold.
DATASET
METRIC
SOTA
DUNTA
KNN 50
KNN =
KNN
SCHWARTZ ET AL. (2023)
5.2
LIU ET AL. (2023)
5.2
WeA
RMSE (r)
50K 1
2.0 (.93)
2.1 (.93)
2.1 (.91)
TOLAN ET AL. (2024)
(.59)
LANG ET AL. (2023)
5.6
We
RMSE (T)
SCHWARTZ ET AL. (2023))
22.1 (.54)
50K /
11.7 (.89)
12.1 (.84)
12.3 (.86)
Wpns CLC-
RMSE (r)
SCHWARTZ ET AL. (2023)²
1.5 (.35)
50K /
0.71 (.75)
0.74 (.73)
0.74 (.72)
wFI
500 im
80.1
74.3
69.5
PASTIS
OA
GARNOT & LANDRIEU (2021)
84.2
500
56.2
54.1
49.5
PF
WFI
GAYDON & ROCHE (2024)
74.6
50K |
73.8
76.0
75.8
w"
50K /
.70
This number of samples is considered as a low data regime due to: 1) a total coverage area of R 31Km² 2) only xs 0.25% of data required compared to some supervised models.
4.2.1. ZERO-SHOT CLASSIFICATION PERFORMANCE
retrieval database sizes (Table G1). This is to be expected given the small subset of waveforms in the retrieval database that might not reflect all possible variations for particular height class. Retrieved vs. reference waveforms can be found in Figures 15 to 18.
Zero-shot results in Table 1 demonstrate that for verti- cal structure products (Wyh, Wc. Wpas), DUNIA with KNN=50 outperforms specialist models. It achieves RMSE (r) improvements of 3.2 m (.16), 10.4% (.35), and 0.79 (.4) for, respectively, canopy height, cover, and plant area index, and, matching the mapping quality of supervised methods (Figures and 12). For tree species identification (PF), DUNIA slightly underperforms by 0.8% KNN=50 but ex- ceeds the baseline by 1.4% when decreasing the number of neighbors (KNN=5) used for the distance-weighted voting. For land cover mapping (CLC+), DUNIA achieves strong performance with a wF1 score of 80.1% %. However, for crop type mapping (P ASTIS), DUNIA significantly underper- forms compared to the state-of-the-an which likely due to the variability of the phenological cycles of crops that cannot be well captured by single median composite.
4.2.2. FINE-TUNING PERFORMANCE
line-tuning results in Table demonstrate that DUNIA out- performs the other models in vertical structure-related prod- ucts, with all models except AnySat trailing significantly. In comparison to DUNIA's zero-shot results, all the models underperformed in the finetuning setting for W. and Wpa due to the long-tailed distribution of these two products that are not well modeled with the simple L1 loss that we used. Qualitatively, as shown in Figures I1 and 12, DUNIA and AnySat show similar level of detail; however, DUNIA is capable of producing higher values, which we attribute to the alignment with the vertical structure in the pre-training stage. For CROMA, Figures Il and 12 shows that this model produces much smoother maps than the two other models.
For lower KNN values, the performance of vertical structure- related products and tree species identification remains sta- ble. In contrast, land cover and crop type mapping expe- rience significant performance drops, indicating that the model needs more samples for accurate classification. For retrieval databases with fewer samples (Table GI), verti- cal structure-related products (Wrhs We. Wpas) maintain their performance and map qualities (Figure 14), while other products (CLC4, PASTIS, PF) show a substantial degra- dation in performance.
For CLC, and PF, DUNIA's performance in the fine- tuned setting increased in comparison the zero-shot set- ting with results par with the much more data-heavy AnySat model (Table 2). However, DUNIA underperformed by significant margin (1 4.1%) in comparison to AnySat for the P PASTIS dataset. Qualitatively (Figure 13) both DUNIA and AnySat show similar level of detail for the CT.C+ dataset, in contrast to CROMA which also showed very smooth maps with fewer details.
Regarding waveform (W) retrieval performance, we observe that our model able to retrieve relevant waveforms given a pixel embedding query, with correlation coefficient of .70 for S=50k 1, and decreases by small factor with smaller
In the novel task of waveform generation, observed correlation (r) increase of .08 between reference and gener-
8

---

DUNIA: Pixel-Sized Embeddings via Cross-Modal Alignment for Earth Observation Applications
Table 2. Fine-tuning performance of DUNIA and the five competing models. and are DUNIA's embeddings from O and O respectively. represents the number of samples used for the fine-tuning, with im meaning a 64 pixels fully annotated image, and meaning single annotated pixel. W represents performance results for vertical structures higher than 5 m. Best scores are in bold.
DATASET
METRIC
SAMPLES (S)
DUNIA
ANYSAT
CROMA
DOFA
DECUR
SATMAE
WAB We Wpas CLC+
RMSE (T)
5K im
1.3 (.95)
2.8 (.89)
3.5 (.78)
(.51)
11.0 (.55)
10.5 (.52)
RMSE (T)
5K im
9.8 (.85)
12.1 (.79)
14.2 (.73)
29.2 (.50)
28.5 (.48)
30.2 (.48)
RMSE (n)
5K im
0.62 (.71)
0.95 (.67)
(.61)
(.38)
1.38)88)
(.39)
wFI
5K im
90.3
90.1
86.4
72.0
75.1
75.0
PASTIS
wFI
1.5K
77.0
81.1
73.3
54.5
57.3
55.2
PF
wF1
50K I
82.2
82.3
80.5
79.8
78.9
78.8
W
19MM
.78
ated waveforms compared to retrieval (r=.70 with S=50K I, KNN=1). This correlation dropped to .75 when the diffusion model was trained on just 20% of the available waveforms (Table G2). Retrieved vs. generated waveforms can found in Figures 15 to 18.
less, the model was designed to be pre-trained with limited computational resources.
A Acknowledgements
This work is supported via the A14Forest project, which is funded by the French National Research Agency (ANR; grant number ANR-22-FAII-0NN Ministry of Education and Research (BMBF; grant number 011823025A..
To further analyze the contributions of different architectural choices, including the use of separate decoders and the impact of our alignment strategies, we provide additional ablation studies Appendix H.
and the German Federal
5. Discussion
Impact Statement
We introduced DUNIA, novel framework for Earth Obser- vation applications that learns dense representations through mono-modal and cross-modal alignment. This strategy en- ables our model to perform mono-modal and cross-modal retrieval tasks on several datasets with high precision. It also allows novel capabilities like retrieving or generating highly complex outputs from pixel inputs. To the best of knowledge, this is the only model capable of generating GEDI waveforms at this scale.
Forests are the third-most important terrestrial carbon sink for climate change mitigation by absorbing about one- third of anthropogenic CO2 emissions (Friedlingstein et al., 2022). However, these ecosystems are under increas- ing threat from deforestation, degradation, and climate- induced disturbances such as wildfires, insect outbreaks, and droughts (Anderegg al., 2022). Global initiatives, including the Glasgow Declaration and the Sustainable De- velopment Goals² emphasize the need for accurate forest monitoring to support conservation, restoration, and carbon sequestration projects. Unfortunately, despite these high praises, many other major initiatives have struggled due to a lack of reliable, high-resolution data on forest carbon stocks and how they change over time.
Our main contributions lie in several design choices at the decoding stage. The proposed pre-trained model encoder is flexible and can be adapted to components from other ex- isting approaches; for instance, modules that support multi- temporal input, handle images with varying resolutions. Nonetheless, in this work focused input data $-1 & 2, alongside GEDI that are freely accessible globally to promote easier adoption.
Current forest monitoring systems heavily rely ground- based inventories, which provide robust statistical estimates of biomass and carbon at national or regional scales but lack the spatial granularity needed for tracking localized carbon gains and losses. Additionally, the largest forested cesions-particullrae boreal and tropical areas-remain vastly under-sampled. While satellite data offers global coverage, traditional approaches often constrained by their reliance extensive labeled datasets, limiting their ability to map forest properties comprehensively.
One of the main limitations of this work is also coinciden- tally the reliance on static inputs from S-1 & data, which has limitations for datasets requiring multi-tempprra images, such as crop type mapping. However, our design choices pri- oritize nomo-temppral data to reduce storage requirements and enable applications in areas where time series data are not readily available. Another limitation, common to all EO models, is that the current pre-trained model is tailored to the area and year on which it was trained on; we expect that will require pre-training when used elsewhere. Nonethe-
DUNIA introduces a novel self-supervised learning ap-
Pnttps://sdgs. un. wrg.:///..../r///////e.e/ org/20 22//2000000ggaaa

---

DUNIA: Pixel-Sized Embeddings via Cross-Modal Alignment for Earth Observation Applications
proach that overcomes these limitations by generating pixel- level embeddings from freely accessible satellite data, in- cluding optical and radar imagery. By aligning these embed- dings with sparse but highly informative LiDAR waveforms, our approach enables the estimation of multiple forest at- tributes-includingg canopy height, fractional cover, land cover, tree species, and vertical structure-without requir- ing task-specific labels. This method allows for zero-shot and low-shot predictions of key forest variables, paving the way for scalable monitoring of global ecosystems.
Bao, H., Wang, W., Dong, L., Liu, Q. Mohammed, O. K., Aggarwal, K., Som, S., and Wei, F. Vlmo: Unified vision- language pre-training with mixture-of-moddliy--ypee arXiv preprint mwXiv:2111.022 0.2358, 2021.
Bardes, A., Ponce, J., and LeCun, Y. Viereg: Variance- invariance-covariance regularization for self-supervised learning. arXiv preprint arXiv:2105. 2021.
Bardes, A., Ponce, J., and LeCun, Y. Vicregl: Self- supervised learning of local visual features. Advances in Neural Information Processing Systems, 35:8799-8810, 2022.
By providing unified, multimodal representation of forests, DUNIA democratizes access to advanced Earth Observation tools, enabling researchers, policymakers, and conservation- ists to make informed decisions with minimal reliance on expensive field data collection. This work lays the founda- tion for high-resolution, large-scale ecosystem monitoring, offering a transformative approach to quantifying forest structure, biomass, and biodiversity in support of climate change mitigation and sustainable land management.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., and Joulin, A. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems,
References
2020.
Anderegg, W. R., Wu, C., Acil, N., Carvalhais, N., Pugh, T. A., Sudler, J. P., and Seidl, R. A climate risk analysis of Earth's 's forests in the 21st century. Science, 377(6610); 1099-1103, 2022.
Carreira, J., Koppula, S., Zoran, D., Recasens, A., Ionescu, C.,, Henafff, O., Shelhamer, E., Arandjelovic, R., Botvinick, M., Vinyals, O., et al. Hip: Hierarchical perceiver. arXiv preprint ar arXiv:2202.1080
2022.
Andrychowicz, M., Espeholt, L., Li, D.,, Merchant, S., Merose, A., Zyda, F., Agrawal, S., and Kalchbrenner, N. Deep learning for day forecasts from sparse observa- tions. arXiv preprint wrXiv:2306.06079 2023.
Chen, C., Jing, L., Li, H., Tang, Y., and Chen, F. Individual tree species identification based on a combination of deep learning and traditional features. Remote Sensing, 15(9): 2301, 2023.
Arora, S.. Khandeparkar, H., Khodak, M., Plevrakis, O., and Saunshi, N. A theoretical analysis of contrastive unsupervised representation learning. arXiv preprint 1rXiv:190..009229, 2019.
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual rep- resentations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Asano, Y. M., Rupprecht, C., and Vedaldi, A. Self-labelling via simultancous clustering and representation learning. arXiv preprint arXiv: 1wI.::::91.... 1911.5337, 2019.
Chen, X. and He, K. Exploring simple siamese represen- tation learning. Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp. 15750-15758, 2021.
Astruc, G., Gonthier, N., Mallet, C., and Landrieu, L. AnySat: An Earth observation model for any resolutions, scales, and modalities. arXiv preprint 1rXiv:2412.14123, 2024.
Chen, X., Liang, C., Huang. D.,, Real, E.,, Wang, K., Pham, H., Dong, X., Luong, T., Hsieh, C.-J., Y., et al. Sym- bolic discovery of optimization algorithms. Advances in neural information processing systems, 36, 2024.
Astruc, G., Gonthier, N., Mallet, C.,, and Landrieu, I. Om- nisat: Self-supervised modality fusion for earth observa- tion. In European Conference on Computer Vision, pp. 409-427. Springer, 2025.
Cong. Y., Khanna, S., Meng, C., Liu, P., Rozi, E., He, Y., Burke, M., Lobell, D., and Ermon, S. Satmae: Pre- training transformers for temporal and multi-spectral satellite imagery. Advances in Neural Information Pro- cessing Systems, 35:197-211, 2022.
Bachmann, R., Mizrahi, D., Atanov, A., and Zamir, A. Mul- timae: Multi-modal multi-task task masked autoencoders. In European Conference on Computer Vision, pp. 348-367. Springer, 2022.
Dalagnol, R., Wagner, F. H., Galvão, L. S., Braga, D., Os- born, F,, da Conceição Bispo, P., Payne, M., Junior, C. S.,
10

---

DUNIA: Pixel-Sized Embeddings via Cross-Modal Alignment for Earth Observation Applications
Favrichon, S., Silgueiro, V., et al. Mapping tropical for- degradation with deep learning and planet nicfi data. Remote Sensing of Environment, 298: 108:11332 13798, 2023.
Fuller, A., Millard, K., and Green, J. Croma: Remote sens- ing representations with contrastive radar-optical masked autoencoders. Advances in Neural Information Process- ing Systems, 36, 2024.
Devlin, J. Bert: Pre-training of deep bidirectional trans-
formers for language understanding. arXiv preprint 101.v00810.04805, 2018.
Garnot, V. S. F. and Landrieu, L. Panoptic segmentation of satellite image time series with convolutional temporal attention networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4872- 4881, 2021.
Dubayah, R., Blair, J. B., Goetz, S., Fatoyinbo, L., Hansen, M., Healey, S., Hofton, M., Hurtt, G., Kellner, J., Luthcke, S., et al. The global ecosystem dynamies investigation: High-resolution laser ranging of the earth's 's forests and topography. Science of remote sensing, 1:100002, 2020.
Gaydon, C. and Roche, F. Pureforest: A large-scale aerial lidar and aerial imagery dataset for tree species classification in monospecific forests. arXiv preprint arXiv:2404.12064 2024.
European Environment Agency.
Clo+backbone 2021 (raster 10 m), europe, 3-yearly, jun. URL
2024, 2024.
Girdhar, R., Singh, M., Ravi, N., Van Der Maaten, L., Joulin, A., and Misra, I. Omnivore: A single model for many visual modalities. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp. 16102-16112, 2022.
europa wwr.pa.eu/cataalcoe
71fc9d1b- 479-----79---11--1----- 4da1-aa666 aa66- 662=2fff2cf7? lanquage=al1.
Fayad, I., Ciais, P., Schwartz, M., Wigneron, J.-P., Baghdadi, N., de Truchis, A., A'prrmoon,, A., Frappart, F., Saatchi, S., Sean, E., et al. Hy-tec: hybrid vision transformer model for high-resolution and large-scale mapping of canopy height. Remote Sensing of Environment, 302: 113945, 2024.
Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., et al. Bootstrap your own latent-a approach self-supervised learning. Advances in neural information processing systems, 33:21271-21284, 2020.
Friedlingstein, P., Jones, M. W., O'Sullivan, M., Andrew, R. M., Bakker, D. C. E., Hauck, J., Le Quéré, C., Pe- ters, G. P., Peters, W., Pongratz, J., Sitch, S., Canadell, G., Ciais, P., Jackson, R. B., Alin, S. R., Anthoni, P., Bates, N. R., Becker, M., Bellouin, N., Bopp, L., Chau, T. T. T., Chevallier, F., Chini, L. P., Cronin, M., Currie, K. I., Decharme, B., Djeutchouang. L. M., Dou, X., Evans, W., Feely, R. A., Feng, L., Gasser, T., Gilfillan, D., Gkritzalis, T., Grassi, G., Gregor, L., Gruber, N., Gürses, O., Har- ris, I., Houghton, R. A., Hurtt, G. C., lida, Y., Ilyina, T., Luijkx. I. T., Jain, A., Jones, S. D., Kato, E., Kennedy, D., Klein Goldewijk, K., Knauer, J., Korsbakken, I., Körtzinger, A.. Landschüttzrr, P.. Lauvset, S. K., Lefévre, N., Lienert, S., Liu, J., Marland, G., McGuire, P. C., Melton, J. R., Munro, D. R., Nabel, J. E. M. S., Nakaoka, S.-I., Niwa, Y., Ono, T., Pierrot, D., Poulter, B., Rehder, G. Resplandy, L., Robertson, E., Rödenbeck, C., Rosan, M.M Schwinger, J., Schwingshackl, C., Séférian, R. Sut- ton, A. J., Sweeney, C., Tanhua, T., Tans, P. P., Tian, H., Tilbrook, B., Tubiello, F., van der Werf, G. R., Vuichard, N., Wada, C., Wanninkhof, R., Watson, A. J., Willis, D., Wiltshire, A. J., Yuan, W., Yue, C., Yue, X., Zachle, S., and Zeng, J. Global carbon budget 2021. Earth System Science Data, 14(4):1917-2005, 2022.
Guzhov, A., Raue, F., Hees, J., and Dengel, A. Audioclip: Extending clip to image, text and audio. In ICASSP 2022 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 976-980. IEFE, 2022.
Hassani, A., Walton, S., Li, J., Li, S., and Shi, H. Neigh- borhood attention transformer. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6185-6194, 2023.
He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. Mo- mentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-99777 9738, 2020.
Ienco, D., Interdonato, R., Gaetano, R., and Minh. D. H. Combining sentinel-1 and sentinel-2 satellite image time series for land cover mapping via multi-source deep learning architecture. ISPRS Journal of Photogrammetry and Remote Sensing, 158:1 1-22, 2019.
Fuller, A., Millard, K., and Green, J. R. Croma: Remote sensing representations with contrastive radar-optical masked autoencoders. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Jaegle, A., Gimeno, F., Brock, A., Vinyals, O., Zisserman, A., and Carreira, J. Perceiver: General perception with it- erative attention. In International conference on machine learning, pp. 4651 1661-666. PMLR, 2021.

---

DUNIA: Pixel-Sized Embeddings via Cross-Modal Alignment for Earth Observation Applications
Jia, C., Yang, Y., Xia, Y., Chen, Y.TT., Parekh, Z.,, Pham, H., Le, Q., Sung, Y.-H., Li, Z.., and Duerig, T. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pp. 4904-4916. PMLR, 2021.
Luo, H., Ji, L.,, Zhong, M., Chen, Y., Lei, W., Duan, N., and Li, T. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomputing, 508:293-380 -049303-442000 2022.
Minderer, M., Gritsenko, A., Stone, A., Neumann, M., Weis- senborn, D., Dosovitskiy, A., Mahendran, A., Arnab, A., Dehghani, M., Shen, Z., et al. Simple open-vocabulary object detection. In European Conference on Computer Vision, pp. 728-755. Springer, 2022.
Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems, 35: 06565-26577) 2022.
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision pp. 11154406,,20 2023.
Morin, D., Planells, M., Mermoz, S., and Mouret, F. Es- timation of forest height and biomass from open-access multi-sensor satellite imagery and gedi lidar data: high- resolution maps of metropolitan france. arXiv preprint 0rXiv:2310.14662 2023.
Kussul, N., Lavreniuk, M., Skakun, S., and Shelestov, A. Deep learning classification of land cover and crop types using remote sensing data. IEEE Geoscience and Remote Sensing Letters, 14(5):77--782, 2017.
N. Baghdadi, M. Bernier, R. G. and Neeson, I. Eval- uation of c-band sar data for wetlands mapping. In- ternational Journal of Remote Sensing, 22(1):71-88, 2001. doi: 10. 00011000001411160070000003333880
URL https: 1080 11w.1000/00111111100000000001000 //t.aoooooo/....../////011111110
Lang, N., Jetz, W., Schindler, K., and Wegner, J. D. A high-resolution canopy height model the earth. Nature Ecology & Evolution, 7(11):1778-177 1789, 2023.
Noman, M., Naseer, M., Cholakkal, H., Anwer, R. M., Khan, S., and Khan, F. S. Rethinking transformers pre-training for multi-spectral satellite imagery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 27811 127819, 2024.
Lehner, J., Alkin, B., Fürst, A., Rumetshofer, E., Miklautz, L., and Hochreiter, S. Contrastive tuning: A little help to make masked autoencoders forget. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 2965-2973. 2024.
Pauls, J., Zimmer, M., Kelly, U. M., Schwartz, M., Saatchi, S., Ciais, P., Pokutta, S., Brandt, M., and Gieseke, F. Esti- mating canopy height at scale. In Forty-frrt International Conference Machine Learning, 2024. URL https: / //openevieww net/ / forumf wtt.mmmdddd22202ooeeemem
Li, S., Brandt, M., Fensholt, R., Kariryaa, A., Igel, C., Gieseke, F., Nord-Larsen, T., Oehmeke, S., Carlsen, A. H., Junttila, S., et al. Deep learning enables image- based tree counting, crown segmentation, and height pre- diction at national scale. PNAS nexus, pga):ga,, 2023.
Pauls, J., Zimmer, M., Turan, B., Saatchi, S., Ciais, P., Pokutta, S., and Gieseke, F. Capturing temporal dynamics in large-scale canopy tree height estimation. In Forty- second International Conference on Machine Learning, 2025. URL https: /twpn//eeeeeeeeeeeeeeeeeeeeeeeee / id=rilcs3vtXX.
Li, S., Liu, Z., Tian, J., Wang. G.. Wang, Z., Jin, W., Wu, D., Tan, C., Lin, T., Liu, Y., et al. Switch ema: A free lunch for better flatness and sharpness. arXiv preprint arXiv:2402.092200 2024.
net/ forum?
Pei, R., Liu, J., Li, W., Shao, B., Xu, S., Dai, P., Lu, J., and Yan, Y. Clipping: Distilling clip-based models with a student base for video-language retrieval. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1 18983-18992, 2023.
Liu, S., Brandt, M., Nord-Larsen, T., Chave, J., Reiner, F., Lang, N., Tong, X., Ciais, P., Igel. C., Pascual, A., et al. The overlooked contribution of trees outside forests to tree cover and woody biomass across europe. Science Advances, :3a)xee444 2023.
Pelletier, C., Webb, G., and Petitjean, F. Temporal convo- lutional neural network for the classification of satellite image time series. Remote Sensing, 1 155):523, March 2019. ISSN 2072-4292. doi: 10.3390/rs1100052 URL http:///ds doi http///xxddoooo////./o//oo////// 33
Liu, X., Su, Y., Hu, T., Yang, Q. Liu, B., Deng, Y., Tang, H., Tang, 7., Fang, J., and Guo, Q. Neural network guided interpolation for mapping canopy height of china's 's forests by integrating gedi and icesat-2 data. Remote Sensing of Environment, 269:1 12844, 2022.
Press, O., Smith, N. A., and Levy, O. Train short, test long: Attention with linear biases enables input length extrapolation. Proceedings of the NeurIPS, 2022. doi: 10.485550/aXxvv:100..222
Loshchilov, L. Decoupled weight decay regularization. arXiv preprint 171 2017.
12

---

DUNIA: Pixel-Sized Embeddings via Cross-Modal Alignment for Earth Observation Applications
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Singh, A., Hu, R., Goswami, V., Couairon, G., Galuba, W., Rohrbach, M., and Kiela, D. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15638-15650, 2022.
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text- conticondiiional image generation with clip latents. arXiv preprint arXiv:2204.00 06/25, 12):3, 2022.
Singh, M., Duval, Q., Alwala, K. V., Fan, H., Aggarwal, V. Adcock, A., Joulin, A., Dollár, P., Feichtenhofer, C., Girshick, R., et al. The effectiveness of mae pre- pretraining for billion-scale pretraining. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 5484-5494 2023.
Ranftl, R., Bochkovskiy, A., and Koltun, V. Vision trans- formers for dense prediction. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 22179-12188, 2021.
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, Deep unsupervised learning using nonequi- librium thermodynamics. In International conference on machine learning, pp. 2256-2265. PMLR, 2015.
Reed, C. J., Gupta, R., Li, S., Brockman, S., Funk, C.,, Clipp, B., Keutzer, K., Candido, S., Uyttendacle, M., and Dar- rell, T. Scale-mae: A scale-aware masked autoencoder for multiscale geospatial representation learning. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4088-4099, 2023.
Srivastava, S. and Sharma, G. Omnivec: Learning robust representations with cross modal sharing. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 233--148, 2024.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF con- ference computer vision and pattern recognition, pp. 10684-10695, 2022.
Su, J., Lu, Y., Pan, S., et al. Roformer: Enhanced trans- former with rotary position embedding. Proceedings of the ICLR, 2021. doi: 10.48555/anXiv.2104..00
Tan, X., Qin, T., Bian, J., Liu, T.-Y., and Bengio, Y. Regen- eration learning: A learning paradigm for data generation. In Proceedings of the AAAI Conference on Artificial In- telligence, volume 38, pp. 22614-22622, 2024.
Schultz, M. G., Betancourt, C., Gong, B., Kleinert, F., Langguth, M., Leufen, L. H., Mozaffari, A., and Stadtler, S. Can deep learning beat numerical weather prediction? Philosophical Transactions of the Royal Society A, 379 22200070 2021.
Tolan, J., Yang, H.-I., Nosarzewski, B., Couairon, G., Vo. H. V., Brandt, J., Spore, J., Majumdar, S., Haziza, D., Vamaraju, J., al. Very high resolution canopy height maps from rgb imagery using self-supervised vision trans- former and convolutional decoder trained on aerial lidar. Remote Sensing of Environment, 300:1 13888, 2024.
Schwartz, M., Ciais, P., De Truchis, A., Chave, J., Ottlé, C., Vega, C., Wigneron, J.-P., Nicolas, M., Jouaber, S., Liu, S., al. Forms: Forest multiple source height, wood volume, and biomass maps in france at 10 to 30 m resolution based on sentinel-1, sentinel-2, and global ecosystem dynamics investigation (gedi) data with a deep learning approach. Earth System Science Data, 15(11): 4927-4945 2023.
Topouzelis, K., Singha, S., and Kitsiou, D. Incidence angle normalization of wide swath sar data for oceanographic applications. Open Geosciences, 8(1):450 464, 2016. doi: doi: 1001::00..515/geo2200660000 URI. https:://////ccc //doi. doi.
Shaw, P., Uszkoreit, J., and Vaswani, Self-attention with relative position representations. In Proceedings of the NAACL-HIL,, 2018. doi: 10. N0.1865//1//11-2200 2074.
van den Oord. A., Vinyals. O., and Kavukcuoglu, K. Neural discrete representation learning. CoRR, abs/1711.009377, 2017. URL http://arxiv.orr
Shen, K., Ju, 7., Tan, X., Liu, Y., Leng, Y., He, L., Qin, T., Zhao, S., and Bian, J. Naturalspeech 2: Latent diffu- sion models are natural and zero-shot speech and singing synthesizers. arXiv preprint arXiv:2304.00 06, 2023.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need, 2017. URL https:// http::///axvv.ort org/ abs 110/1706.000 03766........
Shi, X., Chen, Z., Wang. H., Yeung, D.-Y., Wong, W.-K., and Woo, W.c. Convolutional Istm network: A machine learning approach for precipitation nowcasting. Advances in neural information processing systems, 28, 2015.
Wagner, F. H., Dalagnol, R., Silva-Junior, C. H., Carter, G., Ritz, A. L., Hirye, M. C., Ometto, J. P., and Saatchi, S. Mapping tropical forest cover and deforestation with
13

---

DUNIA: Pixel-Sized Embeddings via Cross-Modal Alignment for Earth Observation Applications
planet niefi satellite images and deep learning in mato grosso state (brazil) from 2015 to 2021. Remote Sensing, 15(2):521, 2023.
Wang, Y. Albrecht, C. M., Braham, N. A., Liu, C., Xiong, Z., and Zhu, X. X. Decoupling common and unique representations for multimodal self-supervised learning. In European Conference on Computer Vision, pp. 286-303. Springer, 2024.
Xiong, Z., Wang, Y., Zhang, F., Stewart, A. J., Hanna, J., Borth, D., Papoutsis, I., Le Saux, B., Camps-Vale Valls, G., and Zhu, X. X. Neural plasticity-insple model for observing the earth crossing modalities. arXiv e-prints, pp. arxiv-2403, 2024.
foundation
Xue, Y., Gan, E., Ni, J., Joshi, S., and Mirzasoleiman, B. Investigating the benefits of projection head for represen- tation learning. arXiv preprint arXiv:2403.1 1391, 2024.
Yu, J., Wang, Z., Vasudevan, V., Yeung. L., Seyedhosseini, M., and Wu, Y. Coca: Contrastive captioners are image- text foundation models. arXiv preprint 2rX11:205.00191,, 2022.
Yuan, L., Chen, Chen, Y.-L., Codella, N., Dai, X., Gao, J., Hu, H., Huang, X., Li, B., Li, C., al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11433 2021.
Yuan, Y., Lin, L., Liu, Q., Hang. R., and Zhou, Z.-G. Sits- former: pre-trained
cetiloo-spetrrattmppppee represen- tation model for sentinel-2 time series classification. In- ternational Journal of Applied Earth Observation and Geoinformation, 106:102651, 2022.
Zbontar, J., Jing, L., Misra, LeCun, Y., and Deny, S. Barlow twins: Self-supervised learning via redundancy reduction. In International conference machine learn- ing. pp. 12310- 12320. PMLR, 2021.
Zhang, C., Sargent, I., Pan, X., Li, H., Gardiner, A., Hare, J., and Atkinson, P. M. Joint deep learning for land cover and land use classification. Remote sensing of environment, 221:173-187, 2019.
Zhang, S., Zhu, F., Yan, J., Zhao, R., and Yang. X. Zero-cl: Instance and feature decorrelation for negative-free sym- metric contrastive learning. In International Conference on Learning Representations, 2021.
14

---

DUNIA: Pixel-Sized Embeddings via Cross-Modal Alignment for Earth Observation Applications
A. Implementation Details
In the main text, we provided a high-level overview of the architecture and training setup. Here, we include additional implementation details for key modules and components of our framework. These details were omitted from the main body for brevity and are provided here for completeness and reproducibility.
M
R2
X
NA
ViCReg
VICReg
VICReg
VICReg
M
O
O
NA
0
TB
TB #2
O
NA
x2
Zero-CL low
Location to waveform mapper
Legend
Neighboorthood attention (NA)
imput
Block transformer btyers
Temporal avg pooling
Singls data 9-1&2 input mosaic
RVQ
Transformer decoder tnyers
Channel wine avg pooting
GEDI weveform location
w
W
Upsample
Linear adapter layer
GEDI waveform
Resnet encoder block
Convolutional adapter layer
Skip connection
D
Reanet decoder brock
Projection head
..... Only during Training
MSE
Recursive vector quantizer
Dense features
Figure AI. An overview of our framework showcasing the main blocks and connections. The pre- pre-trained model, which take input a single date S-1 & image mosaic, is shaded blue. The multi-temporal image AE, which take as input multi-date S 11 2 image mosaic, is shaded in yellow. The vertical structure AE, which take as input the GEDI waveforms, is shaded in green.
A.1. Fre-Traiedd Model
Patch Embedding. To generate patch embeddings, the 2D composite image E H XC is processed using a convolutional layer with kernel size (P, P) and stride P. This operation partitions into N HW/P² non-overlapping patches, which are reshaped into sequence embeddings NxD Here, C represents the number of input channels, (H, W) denotes the resolution of the input image, P is the patch size, and D is the patch embedding dimension.
Unlike tokenization strategies that rely fixed-size input grids, the use of convolutions allows the model to adapt to varying input resolutions. This is particularly usefuul in EO applications, where spatial resolutions can vary across datasets, While transformer based architectures address sequence length variability through techniques like rotary embeddings (Su et al., 2021), ALiBi (Press et al., 2022), and relative position encodings (Shaw et al., 2018), convolutional patch embedding provides a structured inductive bias that benefits spatial feature extraction while maintaining translational invariance.
15

---

DUNIA: Pixel-Sized Embeddings via Cross-Modal Alignment for Earth Observation Applications
The decoding process begins by reshaping the encoded token sequence from RR into an image-like feature representation different resolutions. At each decoder stage d E 1,2,3, 4, tokens are rescaled to 2 Dpa. where Dp2- = 2d-1 Dp represents the channel dimension at each stage. This transformation involves an initial 1 x 1 convolution to adjust channel dimensions, followed by strided 3 x 3 convolutions for downsampling when 2d-1 2 P, or transpose convolutions when 2d-1 < P.
Decoder Architecture
H 2
W
To reconstruct high-resolution embeddings, feature maps from deeper layers (d = 4) are progressively upsampled using bilinear interpolation and refined through two 3 3 Nonv-BatchNorm- ReLU operations before being concatenated with corresponding features from earlier transformer layers. A final 1 convolution ensures that the final output dimension matches Dp when required. This architecture is instantiated two separate decoders, outputting OV and OH
Projection Head. For the pixel-pixel alignment objective, we apply a lightweight two-layer 1 x convolutional projection head to each output O.H. The first layer expands the feature dimension by a factor of two, followed by a second layer that projects it back to Dp. This design enhances feature expressiveness during training but, as observed in prior work (Xue et al., 2024), projection heads tend to overfit to the pretraining task, reducing their transferability to downstream applications. Consequently, we discard the projection head after training. In contrast, the wixel-waveform alignment output Oy is directly used without an additional projection head, as its representation remains well-aligned with downstream tasks such waveform retrieval and generation.
A.2. Waveform Model
Waveform Encoder. The waveform encoder consists of an input stem and three stages designed to progressively reduce the ID waveform of length L (i.e., W E Rxx1) while increasing its feature representation capacity. First, the input stem processes the waveform using a 3 x 1 convolutional layer with stride of 2. This initial operation halves the length of the waveform to L/2 and increases the channel dimension to 2. Following the input stem, the encoder comprises three stages, each containing three ResNet blocks. Each ResNet block is designed with two 3 x convolutional layers, batch normalization, ReLU activations, and a skip connection. At the beginning of cach stage, the first ResNet block incorporates a strided convolution its initial layer to halve the waveform's 's length and double the channel dimension. The subsequent blocks within the same stage operate with stride of 1. By the end of the three stages, the waveform encoder reduces the input waveform's 's length to L/16 while progressively increasing the channel dimension to 16. Thus, the waveform encoder E transforms waveform of shape R into a latent representation E L/16616
The alignment of the waveforms and the pixels is performed on this latent representation 7n = E., (WW). However, to adapt them to the outputs of OV, we first apply an average pooling along the channel dimension, followed by a projection to the OV projection dimension (i.e., Dp) using a linear layer. In essence, this process transforms Ze into OW E RD,
Residual Vector Quantizer (RVQ). The RVQ layer takes a segment from the latent waveform representation of R//1 where zt is the 5th row of the waveform latent representation, and iteratively quantizes it. It operates by first mapping each = to the closest codebook vector in the first quantizer, then computing the residual (i.e., the difference between the input vector and the quantized approximation). This residual is passed to the next quantizer in the sequence, and the process is repeated for all Q quantizers. At the end of the process, the latent waveform representation ze is converted into a discrete series of quantized residual vectors, which are summed to produce the final quantized waveform representation 2.
Waveform Decoder. The waveform decoder (D,r) mirrors the structure of the encoder but operates in reverse order to reconstruct the waveform from its quantized latent representation z. Similar the encoder, the decoder consists of three stages and output stem. However, instead of using strided convolutions for downsampling, the decoder replaces them with transpose convolutions to progressively upsample the waveform. Fach stage in the decoder increases the length of the waveform while halving the channel dimension, reversing the transformations applied by the encoder. By the end of the decoding process, the latent representation is transformed back into a waveform W with length L.
A.3. Multitemporal Image Processing Model
Multitemporal Image AE. The AE has four processing and downsampling stages. In each stage we replace the conventional double Conn2DD-BtchNomee layer and ReLU activation layer. This design choice is to explicitly model temporal correlations across time steps as mentioned earlier. Downsampling is performed using a strided Conv3I layer applied on the spatial dimension, while
layers in UNets with a two-layer ConvLSTM followed by a BatchNorm
16

---

DUNIA: Pixel-Sized Embeddings via Cross-Modal Alignment for Earth Observation Applications
upsampling is performed using trilinear upsampling operation also the spatial dimension, followed by Conv3D layer. As such, the model takes multi-temporal input image M IRTxHxWxx where 7 is the temporal dimension, and produces feature map X' E xT*HxWxD, Finally, as this is an autoencoder, the original input shape is recovered using Conv3D layer producing M E RT All Conv3D layers in the AE have kernel of 3 x 3 on the spatial dimension.
B. Objective Functions.
Zero-CL Loss. Let A RxD represent a Le normalized pixel embeddings from Oy, where is the number of waveforms in minibatch and D is the feature dimension, and 7B E RR denote their corresponding L2 normalized waveform embeddings from OW The instance-wise contrastive loss is defined
Dr Σ d=1
Lins Σ
1-
i.d
Ims
H
Ins
Hi.a represents the t feature value of the 2t instance. Im is zero-phase component analysis (ZCA) whitened embedding matrix defined as:
H Whis Z, Wins Elus A 1/2 A 1 E
(4)
where Elns RR,G and As are the eigenmatrix and diagonal of the eigenvalue matrix of the asfinity matrix $ = D5p respectively. Similar to the instance-wise contrastive objective, the feature-wise contrastive loss is formulated as:
12T
D2
G
2
Lra
Σ
1
Σ
A.F.Fa i,d
B.Fea
(5)
d
Where Hea is defined as:
Fea Wrec TT
Pea FAA-1/237
(6)
where E' E Pp and Ac are the eigenmatrix and diagonal of the eigenvalue matrix of the covariance matrix C = 2 Z, respectively.
The overall loss Lzero-CL CL Lrea + Lins-
VICReg Loss. Let N E MxD, represent an L2 normalized pixel embedding from 2 and ZT E MxD, its corresponding Le normalized pixel embeddings from OT, with B x x W the number of pixels in the mini-batch of size B. VICReg expressed sum of three losses: a variance loss, an invariance loss, and covariance loss. The variance loss is formulated
1 (((()*) r((T)),
Lvrr /
D2
M
(7)
u( = Dp d
Σ
max
0,1
Σ
(Zid Zz) +
M
where Zi, represents the th feature value of the tth instance.
The invariance loss is formulated as:
M
Linv
Σ
2
(8)
M
i=l
17

---

DUNIA: Pixel-Sized Embeddings via Cross-Modal Alignment for Earth Observation Applications
The covariance loss is formulated as:
Loor = c(ZH)) + =zz²).
c() 1
Σ 1000| (2)li.t
Dp
lept
(9)
M
1
cav( 2) = M
Zi 2) Σ
The overall loss term LVicReg is then:
EVicReg agC + Bi + Yeho Following Bardes (2021) we set on and B3 to 25.0 and Ye to 1.0.
(10)
C. Waveform Generation
Denoising diffusion models (DM) (Sehl-Diekstein et al., 2015) are generative models that learn a data distribution Pdata by gradually denoising a noisy variable following a predefined noise schedule. Let ca be a neural network. Its goal to the original data x from noisy version 4y generated by H, 04200 ote, where NO, I) is Gaussian noise, Or and a parameters of noise scheduling function, and is time step from {I....,}}.
predict
The corresponding objective of a DM can then be written as:
LDM E [1 - c ]]]
(11)
where Pdota denotes the data distribution over the clean inputs zo, and the target y can be the input noise the original input x variable, or the velocity = a,c o,z. DMs can also be parametrized by condition c E R in addition to the diffusion time f, and the corresponding objective becomes:
LCDM -
[1l Cov-co t,
(12)
Another class of diffusion models, known as Latent Diffusion Models (LDMs) (Rombach et al., 2022; Shen et al., 2023; Ramesh et al., 2022), operate in a latent space rather than directly in the high-dimensional data space. By operating in a lower-dimensional latent space, LDMs significantly reduce the computational cost of training and inference compared to DMs, which operate in the original data space. LDMs leverage the latent representation of data obtained by training an autoencoder, defined by encoder & and a decoder D. The autoencoder is trained such that è = D(E(x)) is reconstruction of the original data After training the autoencoder, diffusion model is trained directly on the encoded data samples 2 E {z). A new sample is then obtained by first sampling a representation 2, and then D(2) yields x.
Following the LDM formulation, we learn a network co conditioned on O% 2. where ø and À are the waveform coordinates, that directly predicts denoised waveform latent žy E Ih x16 where 2g = E(W). by minimizing:
1, =0l12]
LLDM- E
(13)
In our formulation, the network E0(2,, OK is ID UNet model which takes the current noisy latent 2t, the time step as input, and the condition OY is integrated into the UNet via cross-attention at each layer. To sample 2u from the diffusion 0.2 model, we start from a Gaussian noise 2r and gradually denoise into samples 27-1, 20 using the ordinary differential equation (ODE) solver proposed by Karras al. (2022), as we found that produces high-quality samples with fewer denoising steps. To make the sampled latent waveform 2n better match its conditioning Specifically, during training we randomly drop the conditioning for 50% of the examples to make the model capable of conditional and unconditional denoising. In practice, we replace the conditioning signal with a random vector. Within the classifier-free free guidance formulation, model predictions can be written as:
classifier-free guidance.
1% co(2), + s. (co(2e, 1, 0.2) ())
(14)
18

---

DUNIA: Pixel-Sized Embeddings via Cross-Modal Alignment for Earth Observation Applications
where the guidance scale. Setting s to 0 is equivalent unconditional sampling, while setting it 2 1 has shown produce more coherent but less diverse results. Here set to 3. Finally, synthesize waveform any location (ø, A) given pixel embedding Oy 0. as condition, use eq. 14 to get 2o, and leverage the frozen waveform decoder Dw, where (w)) is generated waveform.
D. Visual Comparison: Pixel vs. Patch Embeddings
To highlight the difference in spatial granularity between our proposed pixel-level embedding model and a patch-based baseline (e.g., CROMA), present side-by-side qualitative visualization in Figure D1. Both models are applied the same S-1 & 2 imput, and their embeddings projected to RGB using t-SNE. The pixel-based model captures finer spatial structures and preserves object boundaries more effectively than the patch-based baseline, yielding lower-resolution outputs.
DUNIA
CROMA
Figure DI. Comparison between pixel-level embeddings from proposed model (top row) and patch-based embeddings from one of the baseline models (bottom row), both generated from the same S-1 2 inputs (left). Embeddings are visualized using I-SNE.
E. Model Configuration and Optimization
E.1. Model Configuration
The image model pre-trained on 64 64 pixel sized images with 14 bands, 16-layer transformer with attention heads, patch size of eight, 512 embedding dimensions, feedforward layer with two linear layers with a hidden dimension of 2048, GeGLU activation, and 0.1 dropout and attention dropout rates. The image decoders are identical and project the decoded image feature map to 64 embedding dimensions (i.e. Dp 64), Finally, we set the window size of the NA layers to 19 pixels. The waveform AE is trained on 256 bin-long waveforms, with RVQ layers composed of eight quantizers, and each quantizer containing 512 codebook entries. As such, the waveform encoder encodes the waveforms into a 16 x 16 latent representation which average pooled along the channel dimension and projected to 64 embeddings dimension for pixel-waveform alignment. The multi-temppoaal image AE trained on three x 64 pixel sized input images, each with 14 bands, and produces similarly shaped feature map on the temporal and spatial dimensions, and 64 embeddings dimension. This output is then average-pooled the temporal dimension for the pixel-pixel alignment. Finally, the diffusion model is trained on the 16 x 16 quantized waveform latents, which are first projected to 128 channels before being passed to the UNet model.
E.2. Training, Inference, and Generation
DUNIA pre-trained single NVIDIA A6000 48GB GPU with batch size of 60 for 250K steps using the Lion optimizer (Chen et al., 2024), learning rate of 5e-5, weight decay of 0.4, 5K warmup steps, and cosine annealing schedule. For finetuning, we used AdamW (Loshchilov, 2017) with 2e-4 learning rate, a cosine annealing schedule, a batch size of 20, and trained until plateau. The diffusion model was trained with a batch size of 4096 for 100K steps using AdamW W. a le-4 learning rate, 5K warmup steps, and a cosine annealing schedule. During inference, diffusion steps were set to 30. Both DUNTA and the diffusion model were regularized with the Switch EMA (SEMA) technique (Li et al., 2024).
19

---

DUNIA: Pixel-Sized Embeddings via Cross-Modal Alignment for Earth Observation Applications
maintaining an EMA with a decay rate of 0.9, updating every steps, and replacing online model parameters every 1K steps with SEMA coefficient of 0.9.
F. Experimental Settings
Sentinel-2 Data. We used Level-2A surface reflectance data (S2-L2A) from Google Earth Engine (GEE). The dataset includes bands at 10m (Blue, Green, Red, and NIR) and 20m (Red Edge 1-4, SWIRI, and SWIR2) spatial resolutions, all upscaled to 10m ground sampling distance (GSD) using cubic interpolation. We created two sets of mosaies: a single mosaic during the leaf-on season using the median of all available images from April September 2020. This mosaic is used as input for the pre-traiee pre-trained model. The other set is used input for the multitemporal AE and is comprised of three mosaics, each representing the median acquisitions of all images in four-month span from October 2019 until September 2020. Cloudy pixels were filtered out using the S-2 Cloud Probability dataset provided by SentinelHub in GEE.
Sentinel-1 Data. Sentinel-1 data were obtained from the Sentinel-1A and Sentinel-1B B satellites, operating at the C-band (6 cm wavelength). Images were collected in Interfcrometric Wide swath mode (IW) with VV and VH polarizations, derived from the high-resolution Level-1 1 ground range detected (GRD) product. The original 20 m x 22 m resolution was resampled to 10 m x 10 m GSD. Similar to S-2, we also created two sets of mosaics. The S-1 data were calibrated using the Sentinel SNAP toolbox, converting pixel values to backscattering coefficients (a") in linear units and applying geometric correction using the 30m Shuttle Radar Topography Mission (SRTM) Digital Elevation Model (DEM). Finally, the backscattering coefficients of acquired at difference incidence angles were normalized a common reference incidence angle set at Fref -0 using the cosine correction equation (N. Baghdadi & Neeson (2001); Topouzelis et al. (2016)):
"gcos" cos2(0)
0
(15)
GEDI Data. GEDI measures the vertical structure of objects (e.g., vegetation) using three lasers emitting near-infrared light (1064 nm), one of the laser beams is split in two, and after applying optical dithering, this results in eight ground tracks spaced 600 m apart, with each shot having a 25 m footprint. For this study, we used GEDI Level 1B (LIB), Level 2A (L.2A), and Level 2B (L2B) data from April 2019 to December 2021. From the LIB product we extracted the waveforms and their geolocation (i.e., latitude, longitude, and elevation), and the geolocation of the GEDI instrument for each shot. Finally, as the waveforms are stored as vectors of 1420 bins bin 0.15 m), we also extracted for each waveform the bin count, the signal start (Wstart) and signal end (Wend). The latter two indicate the location of the canopy top and the end position of the waveform before the noise, respectively. From the L2A data product, we extracted the Rifss (referred to Wrb),, which represents the estimated height of the tallest object within the waveform footprint, the canopy cover (W.). Finally, from the L2B data product, we extracted the Plant Area Index (Mpat).
As the waveforms are affected by the atmospheric conditions at acquisition time, we followed the filtering scheme of Fayad et al. (2024) to remove low-quality waveforms. Finally, due to the different waveform shapes between the leaf-off and leaf-on seasons, especially for those acquired over deciduous forests, we only considered waveforms acquired during the leaf-on season. This resulted in GEDI dataset of 119 million waveforms and their associated metrics.
PureForest (PF). The PureForest dataset was designed as a benchmark and provides ground truth patches for the classification of mono-specific forests in France (Gaydon Roche, 2024). It includes high-resolution imagery and corresponding annotations for more than 13K 50 x 50 forest patches for 13 tree species.
CLC+Backbone (CLS.). The CLS,, A pan-European wall-to-wall land cover inventory for the 2021 reference year. The product based on Sentinel (i.e., optical) time series from 2020 to 2022 and a temporal convolutional neural network (TempCNN) (Pelletier et al., 2019) serving as a classifier. The product available as 10 m raster and shows for each pixel the dominant land cover among the 11 basic land cover classes.
PASTIS dataset. A crop mapping dataset by Garnot (2021) for 18 crop classes and 1 background class from the French Land Parcel Information System. The dataset contains 2433 128 x 128 pixels densely annotated patches at 10 resolution.
20

---

DUNIA: Pixel-Sized Embeddings via Cross-Modal Alignment for Earth Observation Applications
Vertical Structure dataset. For vertical structure evaluation, evaluated model its ability to map 10 resolution: (1) forest heights (Wra), forest fractional canopy cover (We), plant index (Mpae), and complete waveform (W) retrieval generation. For these products, we rely on the products derived from the GEDI dataset presented earlier.
G. Impact of Label Quantity on Performance
Table GI. Top-1 aetrieval-basee zero-shot classification performance of DUNIA for different database sizes. and are DUNLA query embeddings from Oy and O respectively. represents the number of samples in the retrieval database, with im meaning 64 x 64 pixels fully annotated image, and meaning a single annotated pixel. KNN, represents the best KNN value for a given dataset. w represents performance results for vertical structures higher than 5 m.
DATASET
METRIC
$
KNN;
100%
10% S
5% S
Wr We Wpat
RMSE (r)
50K 1
50
2.0 (.93)
2.1 (.92)
2.1 (.92)
RMSE (r)
50K I
50
11.7 (.89)
12.4 (.86)
12.0 (.84)
RMSE (T)
50K
0.71 (.75)
0.72 (.75)
0.72 (.74)
CLC+
wF1
500 im
50
80.1
75.2
70.2
PASTIS
OA
500 im
50
56.2
52.4
48.3
PF
wF1
50K I
76.0
73.5
70.9
W
r
50K
.70
.67
Table G2. Fine-tuning performance of DUNIA and the five competing models. and
are DUNIA's embeddings from OV and O respectively. represents the number of samples used for the fine-tuning, with im meaning 61 64 pixels fully annotated image, and / meaning a single annotated pixel. W represents performance results for vertical structures higher than 5 m. Best scores are in bold.
DATASET
METRIC
SAMPLES (S)
DUNTA
ANYSAT
CROMA
DOFA
DECUR
SATMAF
Wrh We Wpat CLC+
RMSE (r)
1.4 (.93)
(.89)
3.6 (.76)
11.2 (.50)
11.1 (.52)
10.5 (.52)
RMSE (r)
1K im
10.0 (.83)
12.4 (.79)
14.5 (.72)
30.1 (.48)
29.4 (.47)
30.7 (.47)
RMSE (r)
1K im
0.61 (.70)
0.94 (.67)
1.5 (.60)
1.7
(.36)
1.9 (.37)
wF1
1K im
89.4
89.5
85.9
71.1
74.6
73.8
PASTIS
wF1
300K im
75.3
80.2
71.0
52.2
56.4
54.8
PF
wF1
10K
80.1
80.0
80.2
79.8
78.5
78.7
-
R338MM
.75
H. Ablation Studies
H.1. Loss Choice
We used different loss functions for the Pixel-Pixel alignment and the cosine similarity (CS) using the VICReg loss for the pixel-wavee waveform alignment only reaches a maximum of 0.56 for the positive pairs and 0.10 for the negative pairs, indicating poor alignment. On the other hand, using the ZERO-CL loss, the CS between the positive pairs reaches 0.86, and 0.35 between the negative pairs. We attribute the poor results for Pixel- Woee-oreooorm alignment using the VICReg loss to the low number of available waveforms within a mini-batch. In contrast, for the Fixel-Pixcl alignment, the CS for the positive pairs 0.99, and 0.04 for the negative pairs using the VICReg loss. decreases to 0.98 for the positive pairs and 0.45 for the negative pairs with ZERO-CL. Nevertheless, using ZERO-CL for both modalities increased training times exponentially due the whitening transformation and the high number of available pixels within the mini-batch.
alignment. Table HI shows that
Table HI. Cosine similarity between positive pairs (+) and negative pairs (-) during pre-training. P represents pixel emhedding, W represents waveform embedding. Positive Pairs consist of pixel embedding and its corresponding pixel/waveform embedding, while negative pairs are computed between pixel embedding and other pixel/waveform embeddings within the mini-batch.
Loss
+P++P
-Pe+P
+Pe>
-Pe>
VICReg
0.99
0.04
0.56
0.10
ZERO-CL
0.98
0.45
0.86
0.35
21

---

DUNIA: Pixel-Sized Embeddings via Cross-Modal Alignment for Earth Observation Applications
H.2. Using Shared Decoder
In our formulation, we used two different decoders for horizontal and vertical structural understanding. We hypothesized that single real-valued embedding cannot simultaneously encode contrasting information, such as trees of the same species with different vertical structures. Table H2 Shows that in the retrieval case, training single decoder produces pixel embeddings with no semantic overlap with the waveforms with an average CS of -0.42, while the retrieved pixel embeddings given pixel input are still highly similar. In the case of training separate decoders, retrieval performance is drastically different, with high correlations (0.99) between pixels and the retrieved waveforms.
Table H2. Average cosine similarity (CS) between retrieved pixels and pixel queries (PtP) and between retrieved waveforms and pixel queries (P-W) for KNN=1.
PtP PCS
P-W
CS
Shared decoder
0.97
-0.42
Separate decoders
0.98
0.99
H.3. Hierarchical VICReg Loss
The original VICReg loss proposed by Bardes et al. (2021) was applied between two views of the same input image. They later extended their work and applied to local image features (Bardes et al., 2022). Their results showed that better results were obtained when applying the VICReg loss on local features and at the instance level. Here, we further extend this work and apply it hierarchically to the decoder outputs between the pre-train pre-trained model and the multi-temporal AE, Results in Table H3 show significant improvements for both datasets when using this formulation. These results are also in line with the findings in Bardes et al. (2022).
Table H3. Zero-shot classification performance (wF1) on the PF and CLC+ datasets using a model pre-trained with multiple VICReg losses at different decoder levels (VICReg,) against single loss applied on the embeddings from the last decoder layer (VICReg.).
PF
CLC+
VICReg,
76.0
80.1
VICReg,
67.6
74.2
H.4. Neighborhood Attention
The output of our pre-trained model is composed of two neighborhood attention layers per output head, instead of the standard convolutional layers. This design choice allows each embedding to be modeled based on its local neighborhood instead of relying on small and fixed receptive fields. Table H4 shows an increase of 2. 1 (wF1) for the CLC, dataset and an RMSE decrease of 0.4 m for the Wia. dataset using this new configuration.
Table H4. Performance differences between convolutional output layer (CNN) NA layer the CLC_ and W.s datasets.
CLC+
W.
CNN
72.1 (wF1)
2.4 (RMSE)
NA
74.2 (wF1)
2.0 (RMSE)
H.5. Embedding sensitivity
Our results, either in the zero-shot setting the fine-tuned setting used embeddings from HH for horizontal structure-related products (e.g., tree species identification, land cover mapping) and embeddings from OV for vertical structure-related products (c.g., canopy height mapping, waveform retrieval). We perform the same tests in the zero-shot setting, but reversing the query and retrieval embeddings for the other structure direction. Table H5 shows that relying on vertical structure data for products requiring horizontal understanding and vice versa performs poorly for the six tested products. This validates our design choice of cross-modal alignment with vertical structure data for EO products relying on this type of information.
22

---

DUNIA: Pixel-Sized Embeddings via Cross-Modal Alignment for Earth Observation Applications
Table H5. Top-1 retrieval-based zero-shot classification performance DUNTA with opposite direction query and retrieval embeddings. are DUNIA query embeddings from O² and O respectively.
OG represents the original results from Table for KNN=50.
and
DATASET
METRIC
OG
DUNIA
Wr We Wpat CLC+
RMSE (r)
2.0 (.93)
4.3 (.63)
RMSE
11.7 (.89)
(.53)
RMSE (T)
0.71 (.75)
1.6 (.42)
wF1
80.1
35.1
PASTIS
OA
56.2
0.0
PF
wF1
73.8
56.1
H.6. Inference Runtime (Zero-Shot)
Generating maps in the zero-shot setting requires: 1) forward pass through the pre-trained model (encoder-dual decoders) to obtain the embeddings. 2) for each pixel, retrieve the k nearest neighbors (KNN) from the retrieval database (DB), and 3) classify the queried pixel embedding based on wesighee voting. As such, generating a map over e.g. a 20.48 x 20.48 Km area at m resolution, requires querying and classifying 4, 4,194,304 pixels. However, Table H6 shows that the querying and classification times only add small overhead. Moreover, the time required for the three operations (forward pass, retrieval, and classification) is still faster than similarly capable model like AnySat.
Table H6. Inference runtime (in seconds) of DUNIA in the zero-shot setting, compared AnySat. DB denotes the retrieval database of N key/value pairs (embeddiingllahel) and KNN the numher nearest neighbors retrieved per pixel. All times are in seconds.
MODEL
DB
KNN
FORWARD PASS
RETRIEVAL
CLASSIFICATION
TOTAL
DUNIA
256K
100
2.52
0.36
1.34
4.22
DUNIA
512K
200
2.52
0.40
1.88
4.80
ANYSAT
177.37
177.37
H.7. Impact of Input Size
To assess the sensitivity DUNIA to the input image resolution, we conducted experiments using three different input sizes: 128 x 128, 256 256, and 512 x 512. As shown Table H7, results across datasets and metrics remain unchanged. This confirms that image size has a negligible impact on the final performance of the model in the zero-shot setting.
Table H7. Zero-shot performance of DUNIA at different input image sizes for several datasets.
DATASET
METRIC
128x 128
512x512 256x256
WA W. CLC+
RMSE (r)
2.2
2.0
2.1
RMSE (r)
11.6
11.7
11.7
wFL
80.2
80.1
80.2
H.8. Temporal Stability
To evaluate the temporal stability DUNIA, conducted two experiments on several vertical structural products across the years 2019, 2020, and 2021. Other products were excluded due to the unavailability of labels for different years.
1. Fine-tuning an MLP head on top of model pre-trained in 2020, and evaluating 2019 and 2021.
2. Pre-training DUNIA using combined data from 2019-2021 and evaluating it in the zero-shot setting.
Table H8 shows that fine-tuning on individual years yields consistent performance across time. Similarly, Table H9 demonstrates that zero-shot performance is also stable over several years.
23

---

DUNIA: Pixel-Sized Embeddings via Cross-Modal Alignment for Earth Observation Applications
Table H8. Fine-tuned performance (RMSE) vertical structure variables estimation several years.
DATASET
METRIC
2019
2020
2021
Wih We Wpoi
RMSE
1.35
1.34
1,40
RMSE
9.8
9.8
10.1
RMSE
0.65
0.62
0.63
Table H9. Zero-shot performance (RMSE) on vertical structure variables retrieval across years.
DATASET
METRIC
2019
2020
2021
Wrh We Wpai
RMSE
2.4
2.0
2.1
RMSE
11.6
11.7
11.9
RMSE
0.75
0.71
0.74
H.9. Impact of S-2 and S-2 Input Images
Table H10 shows that combining both modalities yields the best performance for height estimation and tree species identification, while land cover classification is relatively robust to the choice of modality.
Table HI0. Performance of DUNIA when using S-1 only, S-2 only, or both modalities, Results are reported for height prediction (RMSE), land cover classification (wFI) the CLC, dataset, and tree species identification (wF1) on the PF datasct. Performance results obtained in the fine-tuned setting.
DATASET
METRIC
S-1 ONLY
S-2 ONLY
S-2 &
We PF
RMSE
3.80
2.80
1.34
wF1
65.5%
81.2%
82.2%
CLC+
wF1
90.2%
90.0%
90.3%
H.10. Median Composites vs. Single-Date Images
Results on height estimation and crop classification (PASTIS) (Table HIl) show that median composites - even without full time series - significantly outperform single-date imagery by capturing more stable and representative reflectance patterns over time.
Table HII. Comparison between single-date median composite inputs products where temporal information is critical. Performance results obtained in the fine-tuned setting.
DATASET
METRIC
SINGLE-DATE IMAGE
MEDIAN COMPOSITE
WA PASTIS
RMSE WF1
1.9
1.3
42.3
77.0
24

---

DUNIA: Pixel-Sized Embeddings via Cross-Modal Alignment for Earth Observation Applications
I. Further Results
Baseline
DUNIA (ZS)
DUNIA (FT)
Any
CROMA
We
(%)%
Figure 11. Fractional canopy cover (We) maps produced with different models. Baseline represents reference maps from adapted FORMS (Schwartz et al., 2023). DUNIA (ZS) represents maps obtained the zero-shot setting, while DUNIA (FT) represents maps obtained the fined-tuned setting. Best viewed zoomed-in (3004%).
25

---

DUNIA: Pixel-Sized Embeddings via Cross-Modal Alignment for Earth Observation Applications
Baseline
DUNIA (ZS)
DUNIA (FT)
AnySat
CROMA
Wyh
(m) 30
Figure 12. Canopy height (W+x) maps produced different models. Baseline represents reference maps FORMS (Schwartz al., 2023). DUNIA (ZS) represents maps obtained in the zero-shot setting, while DUNIA (FT) represents maps obtained in the fined-tuned setting. Best viewed zoomed-in (300+%)
26

---

DUNIA: Pixel-Sized Embeddings via Cross-Modal Alignment for Earth Observation Applications
Baseline
DUNIA (ZS)
DUNIA (FT)
AnySat
CROMA
CLC.
Sealed
Bushes, shrubs
Non/sparsely-vegetann
Needle leaved
Permanent herbaceous Periodically herbaceous
Water
Broadleaved deciduous
Broadleaved evergreen
Lichens and mosses
Figure 13. Land classes (CTC4) maps produced with different models. Baseline represents reference maps from the European
Environment Agency (2024). DUNIA (ZS) represents maps obtained in the zero-shot setting, while DUNIA (FT) represents maps obtained in the fined-tuned setting. Best viewed zoomed-in in (300+%).
27

---

DUNIA: Pixel-Sized Embeddings via Cross-Modal Alignment for Earth Observation Applications
Wrh
0
30
Wrh (m)
Figure 14, Comparison of map qualities for the task of estimating the canopy height in the zero-shot setting using 50K GEDI samples queries (top row) vs. only 10K (bottom row).
28

---

DUNIA: Pixel-Sized Embeddings via Cross-Modal Alignment for Earth Observation Applications
Conmtatorr
Conslater
Conelatton 2:07
Combleteer 84
0.
2.4
Ratioeent Atvedare
The Deneeaate Wiseffon
Tue Patiiement Wovettrre
muree Wavefare
3.0
0:2
100
250
100
250
200
50
220
250
Comebater
Commeaatiors D:R7
Corelattoon 2.50
Conelitters
6.6
Toe Ratraneed deveram
Toe Gomeravet Wavefore
Toe Rastliaed Wiredamm
Ger BEONN Waveforr
3.0
0.0
100
1500
250
1000
144
2012
100
100
200
255
Comveetor: 6.70
Conveettor 0.54
Conslation
Comnletter: 8.53
2.-
Sia
Sira (9)
Grm In1
Tue Retroced Mtsedam
Tre Denerated Woudiinn
The Rethineed Wovedar
Brc ensred Wavebore
2.0
8.0
8.0
50
300
150
200
255
50
100
150
250
25.
52
100
200
252
52
100
250
255
Figure 15. Uncurated list of retrieved ( (~>) and generated waveforms ( (~^) overlayed reference waveform (^^). 'Correlation' Pearson's 's correlation coefficient (r) between the reference and the retrieved/generrtte waveforms. Best viewed zoomed-in (200+%).
29

---

DUNIA: Pixel-Sized Embeddings via Cross-Modal Alignment for Earth Observation Applications
Carsbros
Care wttan
conmaaaio
06 61:0
Max 1910
wovetfor
Combbrip
Commaatiopy
Como ammarttoo..oooo
Conelation:
2.6
2.
Omm 010
Dem 1910
Waedfars
T1a Wiwefoor
Waanfore
Cambbrtas
Commtationn
Ban-00
Nkm 194))
Dett Weedam
The Geme aha Movehber
Tuer Rentioved Wevehbrr
The Cemersted Meadfoom
122
Set Weaeffrre
derar snd Mouerberr
Tia Batlived Vowerborrs
Oesoonated Maanfoee
2.0
39
300
20C
250
50
U00
152
200
250
40
100
200
250
100
250
259
Netreved Weadhom
Gener Movefemn
Retroned Wevetor
Commuted Movetern
0.8
300
200
250
50
100
150
200
250
200
150
200
250
50
100
150
250
Figure 16. Uncurated list of retrieved (~n) and generated waveforms (<) overlayed on reference waveform (~n). 'Correlation' Pearson's 's correlation coefficient (r) between the reference and the retrieved/generated waveforms. Best viewed zoomed in (200+%).
30

---

DUNIA: Pixel-Sized Embeddings via Cross-Modal Alignment for Earth Observation Applications
Carrirru 3.88
Comsarries
Cara
Conmaato
06:01 01:0
Werehorr
Cormbrtonn/.ss
Commtation
Come
Conelation
2.6
0mm 010
Sra 9m0)
Dim 1910
Weedaar
Tue Oera Morefom
T1ee Wiweform
Waanforr
Carmerria
6km (91)
Weedam
Toe Geme shad Moverber
Toe Rattirved Wevehor
Tee Cenmrsted Meadform
Du
Sett Weaefarr
Settinadd soleeber
Tie Oeeeenntte Mownfooro
2.0
1.0
30
200
250
50
D00
151
200
150
250
100
255
Wevefam
Gener Movefeem
Seriived Meutforr
Govented Hoveter
300
200
250
50
100
150
200
250
20
100
150
252
30
100
150
202
250
Figure 17. Uncurated list of retrieved (~)) and generated waveforms (~n) overlayed on reference waveform (~n). 'Correlation' Pearson's 's correlation coefficient (r) between the reference and the retrieved/generated waveforms. Best viewed zoomed in (200+%).
31

---

DUNIA: Pixel-Sized Embeddings via Cross-Modal Alignment for Earth Observation Applications
Retrieudd Revaforo
Consrated Weaeforr
Waveffor
Ganer Vaseforr
0.2
200
1.00
302
250
200
Comberor
Commacos
Conelation: 2.00
Convlatice: 8.8
0.4
Vauchfer
Condlaber
Cordiation:
Condiatore 0.20
Comneatiors
81ox
Covemnted Weadfore
Ther's Ratr Mavelum
Tre Oreen Mounhber
1.0
320
200
50
Rchieved
Commente Mondber
Retreveed Mondlern
Domoreted Merderm
3.0
3.0
0.8
00
0.0
150
100
350
250
18C
100
200
100
756
Corebdiee
0.2
0.2
200
150
200
250
250
3ma
Figure 18. Uncurated list retrieved ( (^^) and generated waveforms ( (~) overlayed on a reference waveform (~^). 'Correlation' Pearson's 's correlation coefficient (r) between the reference and the retrieved/generated waveforms. Best viewed zoomed-in in (200+%).
32


Reviews:
[{"review_id": "5UvL13vKR6", "reviewer": "Reviewer_7y1C", "summary": "- The paper introduces DUNIA, a novel framework that generates pixel-level embeddings through cross-modal alignment between Sentinel-1 & 2 imagery and LiDAR waveform data.\n- The model incorporates several components, including a multi-modal pre-training model, two autoencoders, dual decoders with neighborhood attention, contrastive losses (Zero-CL and VICReg), and a latent diffusion model for waveform generation.\n- Extensive experimental evaluations demonstrate the framework's effectiveness in multiple downstream Earth observation tasks, achieving state-of-the-art results in zero-shot and low-shot learning scenarios.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First, we thank the reviewer for their feedback and their appreciation of our work. The following are our responses to your comments.\n\n    1. Questions For Authors\n> 1\\. Concern (C): The Methods section could be improved by adding a brief subsection that explains the overall flow of the proposed pipeline in a simplified manner. Currently, it relies on a complex figure and separate explanations of individual modules, leaving the actual flow of the method unclear.\n\nAnswer (A): This concern was also raised by **WDGb**.\n1. We simplified the Methods section and revised the first paragraph of Section 3 (Approach) to make it easier for the reader to follow.\n2. We have simplified Figure 2, only keeping the main blocks that aid in understanding the methodology. The original Figure 2 has been moved to the Appendix as a reference.\n\n> 2\\. C: Tables 2 and 3 could be refined; as they currently appear to present a large amount of information without clear organization. \n\nA: This comment was also raised by **WDGb**. Due to space limitations, we had to format them as they were so that they could be included in the main text. In the revised version, Table 1 has been split into two tables: the first table now includes only the default configuration for the zero-shot classifier (i.e., sample size S and different KNNs), while the second table includes the modifications to the zero-shot classifier. Table 2 has also been split in a similar fashion.\n\n> 3\\. C: It is not clear why the authors selected FMs AnySat and CROMA for comparison. Is it because these are the only two models that use - Sentinel-1 and Sentinel-2 data in their pre-training? If other models also utilize this data, an explanation for choosing only these two would be helpful.\n\nA: Although data requirements were a factor in selecting the models we compared against, the decision to include these two models was based on the type of EO applications they target, performance compared to other EO foundation models, and also their recency and novelty. However, following the comments of the reviewer **hPct**, we added three more models as baselines  namely: DOFA, SatMAE, and DeCUR. We invite reviewer **7y1C** to see our response to reviewer **hPct** for the comparison results.\n\nAgain, thank you for your time reviewing our paper.", "claims_and_evidence": "- The paper's claims about the efficiency and effectiveness of the proposed method are well-supported by extensive experiments. The authors provide detailed quantitative results across multiple datasets and tasks, clearly validating the benefits of their pixel-level embeddings and cross-modal alignment strategies.", "methods_and_evaluation_criteria": "The authors have carefully selected relevant datasets, performance metrics, and models to construct their framework, reinforcing the robustness and applicability of their approach.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimental design is valid, and the analysis is discussed in comprehensive detail, reinforcing the credibility of the approach.", "relation_to_broader_scientific_literature": "The paper not only delivers significant performance enhancements but also rigorously benchmarks against established literature baselines, underscoring its substantial potential to advance satellite-based tasks.", "essential_references_not_discussed": "The authors provide a comprehensive discussion of related work, supported by an extensive list of references.", "other_strengths_and_weaknesses": "**Strength*\n- Comprehensive combination of diverse, advanced modeling techniques including multi-modal pre-training, dual decoders, and latent diffusion models.\n- The paper is well-written and structured, making it easy to follow the detailed analyses and experimental evaluations.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "- The methods section could be improved by adding a brief subsection that explains the overall flow of the proposed pipeline in a simplified manner. Currently, it relies on a complex figure and separate explanations of individual modules, leaving the actual flow of the method unclear.\n- Tables 2 and 3 could be refined; as they currently appear to present a large amount of information without clear organization.\nIt is not clear why the authors selected FMs AnySat and CROMA for comparison. Is it because these are the only two models that use - Sentinel-1 and Sentinel-2 data in their pre-training? If other models also utilize this data, an explanation for choosing only these two would be helpful.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZSONxmkm3D", "reviewer": "Reviewer_WDGb", "summary": "The paper introduces DUNIA, a new approach to learn pixel-level embeddings of Earth observation images. DUNIA is trained contrastively, aligning satellite images with full-waveform LIDAR data to enable understanding of both “vertical” and “horizontal” structures. Experiments measure the effectiveness of the embeddings in seven environmental monitoring tasks. They find the embeddings enable zero-shot classifiers to perform comparably to or outperform supervised specialists, and strong fine-tuning performance with low amounts of data compared to state-of-the-art.\n\n## Update after rebuttal\nThe authors have addressed most of my concerns, so I have adjusted my score accordingly.\n\nI still think the paper has problems with methodological motivation and clarity, and presentation more broadly, which are why I haven't increased my score further. For example, why are two autoencoders used (encoder-decoder architectures) rather than simply two encoders with the third alignment model? Is there intuition for why the RVQ module is needed/helpful beyond what the authors state as \"regularization by enforcing discrete representations\" (which isn't clear to me why it's needed for this task)? These are just a few examples of design decisions that I don't think are clearly motivated, and there are many design decisions made in the paper. For decisions the authors are newly making, intuition should be provided. If designs are well-studied and motivated in prior work, that should be clearly stated and cited.\n\nI also think the Tables are still hard to read. For example there is inconsistent use of decimal formatting, inconsistent use of entries with values in parentheses, and lots of acronyms/shorthands which I don't think are necessary or could be made more clear.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank reviewer **WDGb** for the thorough review and the helpful comments.\n\n    1. Methods And Evaluation Criteria\n> 1\\. Concern (C): There may be at least one baseline that should be compared against for proper contextualization in the literature.\n\nAnswer (A): Thank you for this comment, also raised by **hPct**. We have added four additional baselines for the canopy height mapping task and one additional dataset (canopy height from ALS). Regarding the foundation model baselines (FMs), we have benchmarked against three new recently proposed FMs. We invite reviewer **WDGb** to read our responses to reviewer **hPct** regarding the performance results.\n\n    2. Essential References Not Discussed\n> 1\\. C: It does not seem like the paper has a “self-supervised learning for EO” related work section.\n\nA: We agree and thank you for raising this point. Originally, related works on self-supervised models for EO were cited based on their pre-training objectives. In the revised version, we have included a dedicated section for these models. This section now discusses the previously mentioned models, the new models that were included as part of our response to reviewer **hPct** and **Presto**, the model mentioned by the reviewer.\n\n    3. Weaknesses\n> 2\\.1 C: There is a lack of intuitive explanations for methodological choices\n\nA: Thank you. We would be grateful if the reviewer could guide us towards the passages where we failed to deliver a convincing argument for a given design choice. In the original submission, we discussed the following design choices:\n1. Tokenization layer.\n2. Encoder and decoders.\n3. Losses.\n4. The choices for the two AEs.\n5. Waveform generation.\n\nWe realized that the reliance on image composites (i.e., median reflectance of a time series) was not entirely clear, instead of using a time series as input. This has been addressed.\n\n> 2\\.2 C: Some figures and tables are unclear and difficult to interpret\n\nA: Please see our responses to your concerns 4\\.3\\. and 4\\.4\\. In particular, we have:\n1. Modified Figure 2.\n2. Modified Tables 1 and 2.\n\n> 3\\. C: Minimal qualitative examples.\n\nA: We agree. In the revised version of our manuscript, we have split Figure 3 into three figures and included additional maps / product. We hope that this change satisfies this concern.\n\n    4. Other Comments Or Suggestions\n> 1\\. C: The methodology section would benefit from more intuitive descriptions\n\nA: Please refer to our response in 3\\.2\\.1\n\n> 2\\. C: Including a small figure to clearly demonstrate the advantages of pixel-sized vs. patch-sized embeddings\n\nA: Thank you. We have included a new figure in the Appendix of the revised version, which shows the difference between the two variants and the loss of detail induced by patch-sized embedding models.\n\n> 3\\. C: Tables 1 and 2 are really hard to read and should be restructured to improve readability\n\nA: We appreciate your suggestion. In the revised version, Tables 1 and 2 are now four tables. Please refer to our response to reviewer **7y1C** (Concern 1.2) on this subject.\n\n> 4\\. C: Figure 2 has a ton of details which makes it hard to follow\n\nA: We have simplified Figure 2, only keeping the main blocks that help to understand the methodology. The original Figure 2 has been moved to the Appendix as a reference.\n\n    5. Questions For Authors\n\n> 1\\. C: Table 1 suggests DUNIA underperforms on PASTIS significantly - why do the authors think that is?\n\nA: Our objective with DUNIA is to ensure accessibility and broad applicability. Thus, several compromises had to be made. One of them was not to rely on time series (TS) data as input, but rather a median composite that still retains some phenological information. As an advantage, this would:\n1. Alleviate the need to store large volumes of data.\n3. Broader applicability, especially over areas with persistent cloud cover, where having a TS over a given area would not be possible.\n2. Make the model more efficient, as it only processes a single image instead of a full TS.\n\nThe trade-off is that a simple aggregation function like the median inevitably captures less temporal information than a dedicated TS-aware module. However, in six downstream tasks, we show that TS input is not always necessary to achieve strong performance. We believe that our approach has a good balance between accuracy, efficiency, and accessibility, even if it results in lower performance on datasets like PASTIS, which would strongly benefit from richer temporal information.\n\n> 2\\. C: Could you provide more intuitive explanations or visual examples that highlight why specific methodological choices were made?\n\nA: In the revised version, we have added an additional figure to illustrate the advantages of pixel-based versus patch-based embeddings. We have also clarified our reasoning regarding the use of composite imagery vs. TS. If the reviewer feels that further justifications are needed, we will be happy to provide additional clarifications.", "claims_and_evidence": "The claims are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense for the task of learning pixel-level EO embeddings. Encoding vertical information in the pixel-level embeddings through the contrastive procedure is an interesting idea and the authors validate it on seven tasks spanning four datasets where strong pixel-level information is required for good performance. However, there may be at least one baseline that should be compared against for proper contextualization in the literature.", "theoretical_claims": "No theoretical claims were made in this work.", "experimental_designs_or_analyses": "The experimental designs and analyses presented in the main text are sound.", "relation_to_broader_scientific_literature": "The contributions of this paper have implications for how to design strong pixel-level embeddings, which have several applications in remote sensing / EO. The idea to contrast LIDAR data with satellite imagery is new as far as I am aware, and it may inspire future research on self-supervised learning for pixel-level (or more broad self-supervised learning) EO data.", "essential_references_not_discussed": "One notable self-supervised method [1] learns a pixel-level encoder for satellite imagery. This should be discussed, and potentially even compared to in the results for proper contextualization in the literature. Additionally, it does not seem like the paper has a “self-supervised learning for EO” related work section, which further takes away from its positioning in the literature and makes it difficult for readers to assess the key differences and contributions from prior work.\n\n[1] Lightweight, Pre-trained Transformers for Remote Sensing Timeseries.  Tseng et al. 2023.", "other_strengths_and_weaknesses": "Strengths:\n1. Introduces a novel technique for generating high-resolution, pixel-sized embeddings of EO data.\n2. The method demonstrates broad applicability across a range of environmental monitoring tasks, sometimes outperforming specialists.\n3. Provides extensive supplementary material that supports and extends the main findings.\n\nWeaknesses:\n1. Little-to-no discussion of self-supervised EO related work.\n2. The presentation needs work overall. For example:\n    1. There is a lack of intuitive explanations for methodological choices. This makes it difficult for readers to understand why the authors made certain decisions, putting them into question.\n    2. Some figures and tables are unclear and difficult to interpret, which could hinder understanding.\n3. Minimal qualitative examples which makes it hard for readers to gain a sense of how their method performs on specific examples compared to other methods.", "other_comments_or_suggestions": "1. The methodology section would benefit from more intuitive descriptions and examples to aid understanding.\n2. Including a small figure to clearly demonstrate the advantages of pixel vs. patch-sized embeddings would be helpful. Perhaps revamping Figure 1 to do this could make sense.\n3. Tables 1 and 2 are really hard to read and should be restructured to improve readability.\n4. Figure 2 has a ton of details which makes it hard to follow. Are all of these details necessary, or can some be removed in favor of focusing on the important aspects and simplifying the whole figure?", "questions_for_authors": "1. Table 1 suggests DUNIA underperforms on PASTIS significantly - why do the authors think that is?\n2. Could you provide more intuitive explanations or visual examples that highlight why specific methodological choices were made?\n\nStrong responses to the weaknesses as well as my other comments and questions could result in an improvement in my evaluation of the paper.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8TOPt8Dpna", "reviewer": "Reviewer_hPct", "summary": "The paper presents DUNIA (Dense Unsupervised Nature Interpretation Algorithm), an approach for learning pixel-sized embeddings for Earth observation applications through cross-modal alignment between satellite imagery and LiDAR data. The main contributions include a framework that learns dense pixel-level embeddings by aligning forest vertical structure information from LiDAR waveforms with satellite imagery using contrastive learning. This enables both vertical and horizontal structure understanding at the pixel level. The proposed method can perform zero-shot predictions for multiple forest/land monitoring tasks", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer **hPcT** for the thorough review, comments, and suggestions.\n\n    1. Claims and Evidence\n\n> 1\\. Concern (C): The paper's claim that it \"often outperforms specialized supervised models\" is partially supported by the evidence presented\n\nAnswer (A): We agree and have included performance results in comparison to: Lang, Fayad, Liu and Tolan (et al.). We have also added an additional dataset for this task (i.e., ALS). The results below clearly show that our model outperforms all current specialist canopy height (CH) models even in the zero-shot (ZS) setting. \n\n|Height Dataset|Metric|DUNIA (ZS)|Tolan|Liu|Lang|Fayad|\n|--|--|--|--|--|--|--|\n|ALS| rmse|3.1|9.2|3.9|6.5|3.9|\n|GEDI | rmse|2.0|8.5|5.2|5.6|2.8|\n\n> 2\\. C: The analysis would benefit from more extensive comparisons across different data regime sizes to better understand the model's behavior\n\nA: We chose the sizes based on empirical findings. We presented the lowest data size below which the products were unusable for a given task. In the revised version, we have added an ablation on lower data regimes. All models showed a degradation in performance. However, the model ranking remained the same as that presented in the original submission. Failure cases include overfitting or convergence towards a worse solution.\n\n> 3\\. C: There is limited discussion or analysis explaining why the model performs well in low-data scenarios\n\nA: This is mainly due to the self-supervised training with a contrastive objective. In the main text (line 69), we have included two references that discuss it, and our results corroborate their findings. \n\n    2. Methods And Evaluation Criteria\n\n> 1\\. C: I strongly recommend that the authors compare the proposed method with well-established remote sensing foundational models\n\nA: We had pre-trained DOFA, SatMAE, and DeCUR. Initially, they were not included due to their performance gaps on vertical structure related tasks. Due to time constraints we couldn't pretrain SatMAE++, and we couldn't find any implementations of SkySense. Below are the non-linear probing (MLP) results for our model (DUNIA), DOFA, SatMAE, and DeCUR. All models were probed concurrently on the same dataset. The results show that these models severely underperform (except for the PF data set) compared to our model.  \n\n|Dataset|Metric|DUNIA|SatMAE|DeCUR|DOFA|\n|--|--|--|--|--|--|\n|Height| rmse|1.34|10.5|11.0|11.0|\n|Cover| rmse|9.8|30.2|28.5|29.2|\n|CLC+| wF1 | 90.3|75.0|75.1|72.0|\n|PF| wF1 | 82.2 | 79.8 | 78.9 | 78.8 |\n\n> 2\\. C: Benchmarking against recognized benchmarks (e.g., BigEarthNet) to assess robustness across different years and in different geographical regions.\n\nA: We thank the reviewer for this suggestion. First, we would like to point out that BigEarthNet is a scene understanding task (i.e., an image embedding (not pixel) is used to predict a multi-label). As such, patch-based models are expected to perform better than pixel-based ones. The results below show that even though DUNIA's encoder was not pre-trained on the BigEarthNet dataset, and scene understanding is not the intended use case for DUNIA, yet it compares favorably with the other FMs. \n\n|Dataset|Metric|DUNIA|CROMA|SatMAE|DeCUR|DOFA|\n|--|--|--|--|--|--|--|\n|BigEarthNet| mAP|84.9|84.3|82.1|82.8|83.5|\n\n    3. Essential References Not Discussed\n\nA: Thank you for the suggestion. We have updated our manuscript accordingly. Please see 1\\.1\\.\n\n    4. Weaknesses\n\n> 1\\. C: Restricted comparison with recent specialized models\n\nA: Please see 1\\.1.\n\n> 2\\. C: Limited geographical coverage (mainly French territory)\n\nA: Please see 2\\.2\\.\n\n> 3\\. C: Insufficient robustness testing across different years\n\nA: We thank the reviewer for this comment. We have performed two test cases:\n1. Use the pre-trained model for year 2020 and fine-tune it (MLP) for 2019 and 2021\n2. Pretrain the model using data from three years: 2019, 2020 and 2021.\n\nFor both cases, performance results on canopy height mapping (below) show no significant differences year-to-year in either scenario.\n\nTest case 1: Below is the fine-tuning performance on height estimation:\n\n|Dataset|Metric|2019|2020|2021|\n|--|--|--|--|--|\n|Height | rmse |1.35|1.34|1.40|\n\nTest case 2: Below is the zero-shot performance on height retrieval:\n\n|Dataset|Metric|2019|2020|2021|\n|--|--|--|--|--|\n|Height | rmse |2.4|2.0|2.1|\n\n> 4\\. C: Limited analysis of low-data regime behavior\n\nA: Please see 1\\.2\\.\n\n> 5\\. C: Cross-validation across different environmental conditions\n\nA: We compared DUNIA in the Zero-Shot setting (ZS) across France's 10 major ecological regions (GRECOs A to J). These regions span 12 of the 18 Köppen-Geiger climate types found across Europe and geographically range from lowland plains and coastal plateaus to mountainous terrains. Standard deviation on the height, cover, and CLC+ metrics across the 10 GRECOS regions was respectively 0.28 m, 0.26%, and 0.68%. Indicating similar performance across different environmental conditions.", "claims_and_evidence": "The paper's claim that it \"often outperforms specialized supervised models\" is partially supported by the evidence presented, though the scope of comparison could be broader. While the results demonstrate superior performance in several cases, the limited set of supervised models used for comparison somewhat weakens the generality of this claim. A more comprehensive benchmarking against recent specialized models would provide stronger support for this assertion. A few examples include Lang et al., 2023, Fayad et al., 2023; Tolan et al., 2024\n\n\n* Lang, Nico, Walter Jetz, Konrad Schindler, and Jan Dirk Wegner. \"A high-resolution canopy height model of the Earth.\" Nature Ecology & Evolution 7, no. 11 (2023): 1778-1789.\n* Fayad, Ibrahim, Philippe Ciais, Martin Schwartz, Jean-Pierre Wigneron, Nicolas Baghdadi, Aurélien de Truchis, Alexandre d'Aspremont et al. \"Hy-TeC: a hybrid vision transformer model for high-resolution and large-scale mapping of canopy height.\" Remote Sensing of Environment 302 (2024): 113945.\n* Tolan, Jamie, Hung-I. Yang, Benjamin Nosarzewski, Guillaume Couairon, Huy V. Vo, John Brandt, Justine Spore et al. \"Very high resolution canopy height maps from RGB imagery using self-supervised vision transformer and convolutional decoder trained on aerial lidar.\" Remote Sensing of Environment 300 (2024): 113888.\n\nSimilarly, while the paper demonstrates promising low-shot learning capabilities by showing good performance with reduced training data, this aspect of the research could be more thoroughly explored. The analysis would benefit from more extensive comparisons across different data regime sizes to better understand the model's behavior with varying amounts of training data. Additionally, there is limited discussion or analysis explaining why the model performs well in low-data scenarios.", "methods_and_evaluation_criteria": "The paper's robustness testing could be enhanced, particularly regarding the model's behavior across different years. While the current evaluation demonstrates effectiveness within a specific temporal window, there is limited discussion about how the model performs when applied to data from different years, which is crucial for understanding its long-term applicability in Earth observation tasks. \n\nAdditionally, concerning the datasets used for land cover classification, while they serve their purpose for the current evaluation, expanding the validation to include more diverse land cover datasets would strengthen the model's claimed generalization capabilities. In addition, I strongly recommend that the authors compare the proposed method with well-established remote sensing foundational models, such as SkySense (CVPR 2024), SatMAE++ (CVPR 2024), and DeCUR (ECCV 2024), using recognized benchmarks such as BigEarthNet, fMoW, and DIOR.", "theoretical_claims": "Not Applicable", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "The paper's contributions advance Earth observation research by building on developments in the field. On the foundation model front, it expands upon work like Scale-MAE (Reed et al., 2023) and DOFA (Xiong et al., 2024) in handling multi-resolution data, while also advancing the multi-source data integration approaches seen in OmniSat (Astruc et al., 2025) and AnySat (Astruc et al., 2024).", "essential_references_not_discussed": "The paper overlooks several contributions in the field of tree canopy height mapping. Notably absent is a discussion of recent deep learning-based methods developed by Liu et al. (2023) and Lang et al. (2023), as well as important advances in vision transformer applications by Fayad et al. (2023) and Tolan et al. (2024). \n\n* Liu, Siyu, Martin Brandt, Thomas Nord-Larsen, Jerome Chave, Florian Reiner, Nico Lang, Xiaoye Tong et al. \"The overlooked contribution of trees outside forests to tree cover and woody biomass across Europe.\" Science Advances 9, no. 37 (2023): eadh4097.\n* Lang, Nico, Walter Jetz, Konrad Schindler, and Jan Dirk Wegner. \"A high-resolution canopy height model of the Earth.\" Nature Ecology & Evolution 7, no. 11 (2023): 1778-1789.\n* Fayad, Ibrahim, Philippe Ciais, Martin Schwartz, Jean-Pierre Wigneron, Nicolas Baghdadi, Aurélien de Truchis, Alexandre d'Aspremont et al. \"Hy-TeC: a hybrid vision transformer model for high-resolution and large-scale mapping of canopy height.\" Remote Sensing of Environment 302 (2024): 113945.\n* Tolan, Jamie, Hung-I. Yang, Benjamin Nosarzewski, Guillaume Couairon, Huy V. Vo, John Brandt, Justine Spore et al. \"Very high resolution canopy height maps from RGB imagery using self-supervised vision transformer and convolutional decoder trained on aerial lidar.\" Remote Sensing of Environment 300 (2024): 113888.\n\nFurthermore, while the paper focuses on single-year height estimation, it fails to acknowledge important work on multi-year time series analysis by Dixon et al. (2025), Kacic et al. (2023), and Turubanova et al. (2023). Including these references would provide a more comprehensive understanding of how the proposed approach advances or differs from existing temporal analysis methods in forest structure mapping.\n\n* Dixon, Dan J., Yunzhe Zhu, and Yufang Jin. \"Canopy height estimation from PlanetScope time series with spatio-temporal deep learning.\" Remote Sensing of Environment 318 (2025): 114518.\n* Kacic, Patrick, Frank Thonfeld, Ursula Gessner, and Claudia Kuenzer. \"Forest structure characterization in Germany: novel products and analysis based on GEDI, Sentinel-1 and Sentinel-2 data.\" Remote Sensing 15, no. 8 (2023): 1969.\n* Turubanova, Svetlana, Peter Potapov, Matthew C. Hansen, Xinyuan Li, Alexandra Tyukavina, Amy H. Pickens, Andres Hernandez-Serna et al. \"Tree canopy extent and height change in Europe, 2001–2021, quantified using Landsat data archive.\" Remote Sensing of Environment 298 (2023): 113797.", "other_strengths_and_weaknesses": "STRENGTHS\n\n* Novel integration of horizontal and vertical structure understanding at pixel level\n* Creative combination of different contrastive learning approaches for different alignment tasks\n* Practical Utility: Addresses real-world challenges in Earth observation\n\nWEAKNESSES:\n\n* Limited evaluation scope: restricted comparison with recent specialized models, limited geographical coverage (mainly French territory), and insufficient robustness testing across different years\n* Methodological gaps: limited analysis of low-data regime behavior, insufficient discussion of failure cases, and lack of cross-validation across different environmental conditions\n* Insufficient support for generalization claims: needs stronger evidence for outperforming specialized models, limited validation across diverse land cover datasets, and insufficient analysis of performance in different geographical regions", "other_comments_or_suggestions": "No", "questions_for_authors": "Please refer to the above responses", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5iBHIaaj86", "reviewer": "Reviewer_zT7y", "summary": "The paper proposes DUNIA (Dense Unsupervised Nature Interpretation Algorithm), a method that generates pixel-level embeddings by aligning forest vertical structure information (obtained from space-born full waveform LiDAR) with satellite imagery, by using contrastive learning. The pixel-level nature is what makes DUNIA different from related works, that more typically produce patch-sized embedding. Due to the contrastive learning approach, the resulting embedding can be directly used for various EO tasks in a zero-shot fashion. Experiments are conducted, comparing e.g. with the recent AnySat approach, which show that DUNIA (in fine-tuning setting) yields performance on par with or better than state-of-the-arts on five out of six explored tasks. The embeddings of DUNIA can -- for the first time, to the best of mine and the authors' knowledge -- be used to directly generate waveforms representing the forest's vertical structure from pixel inputs.\n\n## updates after rebuttal\nI thank the other reviewers and the authors for all their efforts. I have read the other reviews + associated rebuttals + the rebuttal to my review. I think overall that the authors have done a thorough job in addressing concerns (but I of course must leave it to the other reviewers to assess what they think of the responses to their respective reviews), including a significant amount of additional relevant experiments.\n\nThe original reviews were quite diverse: 1 strong accept, 2 weak reject, 1 weak accept, with my weak accept being kind of in the middle. I note that after the rebuttal, one of the weak reject reviewers (WDGb) has updated to weakly accepting the paper (instead of weakly rejecting), so it seems the majority of us believe the paper should now be accepted. I therefore score the work as \"accept\" now (before, \"weak accept\").", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First, we wish to thank reviewer **zT7y** for their complete & thorough review of our submission.\n\n    1.Unsupported claims\n\n> 1\\. Concern (C): The struggle to directly estimate the full vertical structure\n\nResponse (A): Reconstructing the full vertical structure (W) requires modeling a complex distribution P(W|Pix). A pixel embedding (Pix) aggregates spectral information over a 2D footprint but lacks the explicit depth-wise cues needed to resolve W directly (e.g. using a decoder head). In contrast, LDMs learn to iteratively denoise samples toward a plausible W, conditioned on aligned pixel embeds. By this process, LDMs implicitly capture the conditional distribution P(W|Pix) in a more expressive manner. We have revised accordingly.\n\n> 2\\. C: Pixel-level alignment necessary for dense predictions\n\nA: We agree with the reviewer that in the fine-tuning case, pixel-level alignment is not necessary for dense prediction tasks, as usually a decoder is trained on the generated patch-sized embeddings. We revised accordingly.\n\n    2. Methods And Evaluation Criteria\n\n> 1\\. C: The use of the weighted F1 score\n\nA: We relied on this metric to remain consistent with the performance scores reported in the literature. The authors of the PF dataset used the weighted F1 score due to high class imbalance. For the PASTIS data set and the CLC+ datasets, OA accuracy has been reported. For these reasons, we opted for the weighted F1 score. Due to lack of time, we only managed to run partial fine-tuning tests. The results indicate that the micro- and weighted- F1 scores have the same order of magnitude. For the macro F1 score, it is several percentage points lower (e.g., \\~7% lower for the PF dataset). However, model ranking remains the same regardless of the score.\n\n> 2\\. C: Inference runtime speeds of DUNIA and AnySat\n\nA: Due to design choices and data requirements, AnySat is orders of magnitude slower than DUNIA zero-shot. Below are the wall clock times (in seconds) to generate a \\~20x20 km area (\\~4.19M pixels), assuming a retrieval database containing 256K keys/values for DUNIA and 100 NNs. The test excludes data loading times.\n\n|Model|forward pass|retrieval|KNN|Total|\n|--|--|--|--|--|\n|DUNIA|2.52        |0.36     |1.34|4.22|\n|AnySat|177.37     | -       | -  |177.37|\n\nFor a database with 512K K/V, retrieval increased to 0.4s. For NN = 200, KNN increased to 1.88s\n\n    3. Experimental Designs Or Analyses\n\n> 1\\. C: Effect of importance of using both S-1 and S-2 data\n\nA: In the limited time that we had for the rebuttal, we only managed to test on heights and land cover classification (CLC+). Below are the results:\n\n|Dataset|Metric|S-1 only|S-2 only|S-1 & S-2|\n|--|--|--|--|--|\n|Heights|rmse|3.8|2.8|1.34|\n|CLC+|wF1|90.2|90.0|90.3|\n\nThe results show that DUNIA leverages both modalities for heights and, in general, using S-2 yields more accurate results than S-1 for heights. For land cover classification, using either modality leads to similar performances. In the revised version, we will include the remaining datasets.\n\n> 2\\. C: Reliance on timeseries data could anyway have been explored a bit\n\nA: We have revised Sections 3 (Approach) and 4.1.1 (first paragraph) to better reflect this point. More specifically, while image composites such as a median composite may be richer than single-date images, they are still less informative than a full time series, as they only provide a median reflectance value over a given period. On the other hand, even this simple form of aggregation preserves significantly more information than a single-date image. Below are the fine-tuning results for two products (Heights, and PASTIS) using randomly acquired-in-time S-1 & S-2 imagery.\n\n|Dataset|Metric|Single-date image|Median composite|\n|--|--|--|--|\n|Heights|rmse|1.9|1.34|\n|PASTIS|wF1|42.3|77.0|\n\n> 3\\. C: Issue raised regarding the effect of image resolution\n\nA: We agree with the raised concern. Our tests show that image size has no effect on the quality of the resulting product. Below are performance results for different resolution images. The tests were performed in the zero-shot setting:\n\n|Dataset|metric|128x128|256x256|512x512|\n|--|--|--|--|--|\n|Heights | rmse | 2.2   | 2.0   | 2.1   |\n|Cover  | rmse | 11.6   | 11.7  | 11.7  | \n|CLC+   | wF1  | 80.2   | 80.1  | 80.2  |\n\n    4. Weaknesses\n\n> 1\\. C: How the model is supposed to perform reasonably well in cases where it does not have composite images\n\nA: Please refer to 3\\.2.\n\n> 2\\. C: The statement: outperform specialized supervised models, even in low-labeled data regimes is wrong.\n\nA: Yes we agree with the raised concern. We have revised accordingly.\n\n    5. Questions For Authors\n\n> 1\\. C: statement(s) about future code and/or model availability\n\nA: We thank the reviewer for raising this point. We are currently preparing the code for publication. We will link to the repository in the finalized version.\n\n> Other questions\n\nA: For Qs 2,3, and 4, please refer to 2\\.1, 2\\.2, and 3\\.3 respectively.", "claims_and_evidence": "Yes, I would say that most of the claims to the best of my understanding are well-backed-up; an example of such a claim that is supported is:\n\n* **Claim #1:** \"In the fine-tuning setting, we show strong low-shot capabilities with performance near or better than state-of-the-art on five out of six tasks.\"\n    - **Quality of evidence for claim:**\n        + I think this is backed up well in the experiments section, see in particular Table 2. There we see that DUNIA obtains best results in 4 tasks, is more or less on par (82.2% vs 82.3%) for one task, and obtains quite a bit worse results (than AnySat) on one task.\n\nOne claim that is not quite as well backed up is:\n\n* **Claim #2:** At 2nd column of first page, it says that related work approaches \"struggle with more complex output like the full vertical structure of vegetation\". A similar statement in Sec. 2.3, where it is claimed that pixel-level alignment is necessary for dense predictions in EO applications.\n    - **Quality of evidence for claim:**\n        + I didn't quite feel that the statement was backed up, e.g. by citing some works / results that show that these related works struggle with this. This could perhaps be alleviated by in the text referring the reader to the experiment section, where e.g. the comparisons to AnySat show some of this claim (?).\n        + Also, I feel like that part about pixel-level alignment being necessary for dense predictions is wrong. Many approaches (including AnySat) is used for dense predictions, despite not having pixel-level alignment.", "methods_and_evaluation_criteria": "Yes, I would say so, e.g. for these reasons (see further positives also under \"Experimental Designs or Analyses\"):\n\n* Good that many tasks (7) are explored, and that the proposed DUNIA obtains strong zero-shot improvements on most tasks, including when comparing with specialized supervised models (and really strong improvements on some -- see Table 1).\n\n* Continuing on the previous note, also great fine-tuning results (Table 2).\n\nSome comments on the negative side of things:\n\n* It was not quite clear to me why the standard F1 score was not (also) used as an evaluation criterion, and only the weighted wF1 score. \n\n* I could not see any reporting of inference runtime speeds (as in actual \"wall-clock\" time), this should have been good to have e.g. in the supplement (D.2.2). I assume it's roughly in the same order of compute-efficiency as e.g. AnySat, but would be good to compare them. Especially interesting to see if the number of neighbors in the kNN part affects this a lot.", "theoretical_claims": "No. Did not see proofs nor theoretical claims in the paper.", "experimental_designs_or_analyses": "Yes, I had a look at all experiment designs / analyses in the paper (and where needed checked if some things that appeared missing in the main were done in the supplement).\n\nPositives:\n\n* The main results in Table and Table 2 seem well-thought-through. A lot of tasks (7) explored, and many relevant methods compared against. DUNIA is best in most cases, and where it is not best, relevant commentary is added (see below).\n\n* I found the analysis of why performance is worse on \"PASTIS\" to be insightful and relevant (see beginning of left column, p7, where the reasoning is that the variability of the phonological cycles of crops cannot be well-captured by a single median composite).\n\n* I liked that in Supp. D.3.5 an analysis of the embedding sensitivity to horizontal and vertical structures was included, as it is such a core methodological contribution of the paper (and as shown in the corresponding Table 7, the proposed design of cross-modal alignment with vertical structure data is important).\n\nNegatives: \n\n* I was missing some ablation or similar into the effect of importance of using both S-1 and S-2 data. In the supplement, for example, there could have been room for trying DUNIA while omitting one of the two.\n\n* While it's good that limitations are made transparent (see latter part of Sec. 5), I find that the comment on the reliance on timeseries data could anyway have been explored a bit, empirically speaking. Because to the best of my understanding, image timeseries were anyway processed in such a way that it in practice could be replaced with a single image (albeit at an expecation of worse results). Thus it would have been interesting to see what happens if one uses a single image instead (and it would still be OK if results get much worse).\n\n* Around L291 in main paper, it is mentioned that fixed-resolution imagery (256x256 pix) is used during inference. I think that it would have been good in the supplement to provide some insight into how results may be affected if using another resolution, as comparison.", "relation_to_broader_scientific_literature": "To the best of my knowledge, the related work mentioned looks good and covers the necessary literature. In particular, I think lots of relevant related work is covered in the beginning of page 2 (and it's particularly good to mention how those works are various ways to remedy the challenges listed towards the end of the previous page (p.1)), followed also in Sec. 2.1 and 2.2. In addition, an extended set of related work is provided in the supplement.\n\nAs for the key contributions of this paper, I think the section 2.3 covers it well. In particular, this work builds on earlier works that develop cross-modal (e.g. AnySat) ML-EO methods, but makes a strong contribution relative to prior works in that DUNIA (for the first time, at least to the best of my knowledge, and based on the authors' claims) due to its pixel-level embeddings can directly approximate forest vertical structure from pixel inputs.", "essential_references_not_discussed": "To the best of my knowledge, no essential references were missed; in particular, no references that made it hard for me to understand this paper were missed.", "other_strengths_and_weaknesses": "Strengths:\n\n* I re-iterate this as one of the main strengths of this work: The embeddings of DUNIA can -- for the first time, to the best of mine and the authors' knowledge -- be used to directly generate waveforms representing the forest's vertical structure from pixel inputs.\n\n* I think in general that the design choices etc of the method have been carefully thought through. An example of many that illustrate this is in Sec. 3.4, where the reasoning about when and why a certain alignment loss is used makes sense (also, such things are well-ablated in the supplement as well).\n\n* It's good that limitations are lifted in the discussion section (e.g. admitting the reliance of cases where timeseries are available).\n\n* The impact statement on p.9 is important and good. It really shows the importance of this line of work.\n\nWeaknesses:\n\n* Not all parts were quite clear to me, e.g.\n    - In Sec. 3.3, I got most of the explanation about that composite image I from S1 and S2 timeseries. What I did not get was how the model is supposed to perform reasonably well in cases where it does not have composite images but just \"simple / plain\" images. I.e., if it's trained assuming access to such \"privileged information\" as composite images, how can it be expected to not get a \"performance drop\" when that input changes to something less \"rich\"? **EDIT:** I keep this weakness, even though it was later made clear (in the listing of limitations in Sec. 5) that this is one of the weaknesses of the approach. But I think this should be more clearly stated earlier as well, as is apparent from the fact that I got confused about it.\n    - In the abstract it reads (L33) \"... outperform specialized supervised models, even in low-labeled data regimes\". This formulation was a bit surprising to me, given that it is my understanding that this is the most expected case, and is thus not surprising at all (since supervised models often require high-labeled data regimes to work).. (?)", "other_comments_or_suggestions": "Some minor things such as typos:\n\n* Be consistent in \"Earth observation\" vs \"Earth Observation\". Pick one. Both work.\n\n* Be consistent in the way Sentinel-1 and -2 are abbreviated, e.g. in Fig. 2 caption it is written as \"S-1 & S2\" <-- write in one of the two ways all the time.\n\n* I suggest \"words\" in math env to not be italic, e.g. in eq. (2) mse could be written in non-italics (and perhaps capital letters for that is mor common).\n\n* When referring to figures in the SM (e.g. Fig 5 to 8 right before Sec. 4.2.2), please state that they are in the SM.", "questions_for_authors": "* I could not see statement(s) about future code and/or model availability. Will these be made publicly available? When?\n\n* Why was weighted F1 used, instead of the \"equal-weighted\" F1 score? (Or why not both?) Finally, can you provide some equal-weighted F1 scores to compare with too?\n\n* What is the runtime for DUNIA vs AnySat? Does this depend a lot on the number of neighbors in the kNN?\n\n* Would the method provide roughly as good results if doing inference on smaller-res images (e.g. 128x128) or higher (512x512), instead of the current default of 256x256?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper received divergent scores after the rebuttal. Reviewers 7y1C, zT7y, and WDGb rated positive scores, while reviewer hPct remained at Weak Reject.

All reviewers praised the paper for its significance of the framework that generates pixel-level embeddings by effectively integrating Sentinel-1/2 imagery with LiDAR waveform data. They liked that this work not only advances cross-modal alignment methods but also enables strong zero-shot and low-shot performance on multiple Earth observation tasks.  In addition, the authors successfully resolved part of the issues regarding methodological clarity. However, there are still concerns regarding presentation issues,  particularly the readability of figures and tables, and the overall theoretical motivation behind some architectural decisions. 

After carefully reviewing the paper, the reviews, and the rebuttal, the AC acknowledges that the concerns raised by reviewers WDGb and hPct are valid. Nonetheless, the paper shows sufficient technical novelty and strong performance in diverse environmental monitoring applications. Based on these considerations, the AC recommends a Weak Accept, with the expectation that the authors will carefully address the comments in the final version of the paper.
Decision: Accept (poster)