UrbanFusion: Stochastic Multimodal Fusion for Contrastive Learning of Robust Spatial Representations

## Abstract

Abstract Forecasting urban phenomena such as housing prices and public health indicators requires the effective integration of various geospatial data. Current methods primarily utilize task-specific models, while recent foundation models for spatial representations often support only limited modalities and lack multimodal fusion capabilities. To overcome these challenges, we present UrbanFusion, a Geo-Foundation Model (GeoFM) that features Stochastic Multimodal Fusion (SMF). The framework employs modality-specific encoders to process different types of inputs, including street view imagery, remote sensing data, cartographic maps, and points of interest (POIs) data. These multimodal inputs are integrated via a Transformer-based fusion module that learns unified representations. An extensive evaluation across 41 tasks in 56 cities worldwide demonstrates UrbanFusion’s strong generalization and predictive performance compared to state-of-the-art GeoAI models. Specifically, it 1) outperforms prior foundation models on location-encoding, 2) allows multimodal input during inference, and 3) generalizes well to regions unseen during training. UrbanFusion can flexibly utilize any subset of available modalities for a given location during both pretraining and inference, enabling broad applicability across diverse data availability scenarios. All source code is available at https://github.com/DominikM198/UrbanFusion .

## 1 Introduction

Urban areas currently accommodate the majority of the world’s population and are expected to absorb billions more in the upcoming decades (United Nations, 2019). This rapid growth has increased the need for accurate forecasting tools to support urban planning and inform sustainable policy decisions through urban analytics (United Nations, 2024; Daniel & Pettit, 2025). Efficient urban operations increasingly depend on precise, location-specific predictions, such as housing price estimation (Yao et al., 2018), mobility prediction (Wiedemann, 2025), and land use classification (Che et al., 2025). These challenges have traditionally been tackled using task-specific models designed for a single domain and geographic region. A widely adopted approach to improve predictive performance is to augment coordinates with geospatial context data (Hong et al., 2023) such as census statistics, business directories, street view imagery, or remote sensing inputs (Mühlematter et al., 2024b; Wang et al., 2023).

However, these models often face limitations due to the availability and quality of context data, which can vary widely across regions (Klemmer et al., 2025). This variability restricts their scalability and applicability, while also making their development costly and dependent on domain expertise (Koldasbayeva et al., 2024). Recently, the rise of foundation models in language (Brown et al., 2020), vision (Assran et al., 2023), and multimodal domains (Radford et al., 2021) has inspired efforts to build general-purpose geospatial models, commonly referred to as Geo-Foundation Models (GeoFMs) (Mai et al., 2025; Jakubik et al., 2023).

Figure: Figure 1: UrbanFusion. Pretrained modality-specific encoders extract features projected into tokens. After random token masking, a Transformer fuses the tokens. The output feeds into two heads: one for Contrastive Location Alignment (CL), the other for Latent Modality Reconstruction (Rec.). For downstream tasks, coordinates or available modalities are input into the frozen encoder (green arrows) to obtain feature vectors for training downstream models.
Refer to caption: Figures/architecture_new.png

Inspired by the success of language-image models like CLIP (Radford et al., 2021), recent research often uses self-supervised learning to align spatial coordinates (longitude and latitude) with other data types, such as satellite or street view imagery (Vivanco et al., 2023; Klemmer et al., 2025; Liu et al., 2025). These works offer general, task-agnostic representations of geographic locations that were shown to improve performance across a wide range of downstream tasks and domains. Nevertheless, they are limited by the modalities they can support. The requirement for paired samples of all modalities at each location, coupled with the use of pairwise contrastive loss between modalities, makes the inclusion of additional modalities challenging. Moreover, each modality is treated
independently, missing the opportunity to learn richer representations by modeling their interactions (Dufumier et al., 2025). Many geospatial tasks, however, rely on the combined signals of diverse spatial data. For instance, housing prices may depend not only on visual features from imagery but also on nearby infrastructure and services, such as road networks or points of interest (POIs) (Yao et al., 2018). To effectively support such tasks, models must go beyond isolated modality processing and instead learn representations that reflect the complex, layered nature of urban environments (Mai et al., 2025).

To bridge these gaps, we present UrbanFusion, a novel GeoFM tailored for urban environments. UrbanFusion integrates multiple spatial modalities across various scales, including remote sensing imagery, street view images, cartographic basemaps, and POIs, into compact, multi-scale, task-agnostic embeddings. To better capture the multimodal context, we propose a training framework called Stochastic Multimodal Fusion (SMF), which combines contrastive learning with self-supervised reconstruction. At each training step, the model samples two distinct subsets of available modalities, aligns their embeddings, and reconstructs the modalities.

UrbanFusion not only outperforms state-of-the-art methods across diverse tasks, but also offers significant advantages: (1) it efficiently supports multiple modalities with minimal training overhead through modality masking, (2) enables joint training and inference on heterogeneous datasets containing arbitrary subsets of modalities, (3) adeptly learns to represent modality-specific features while integrating shared and synergistic information via SMF, and (4) generalizes well to regions unseen during training.

In summary, our contributions are as follows:

- 1.
We propose Stochastic Multimodal Fusion (SMF), a model-agnostic contrastive learning framework that jointly captures modality-specific, shared, and synergistic information. While developed for spatial data, SMF is broadly applicable to general multimodal learning.
- 2.
We introduce UrbanFusion, the first foundation model to flexibly integrate street view imagery, remote sensing data, cartographic basemaps, and POIs into unified spatial representations.
- 3.
We conduct a large-scale evaluation of UrbanFusion on 41 downstream tasks, including housing price prediction, healthcare, environmental variable estimation, land use and land cover classification, demographic inference, urban perception prediction, and energy consumption forecasting. Our results show that UrbanFusion consistently outperforms state-of-the-art models across these domains.

## 2 Related Work

Table: Table 1: Summary of existing and proposed methods for spatial representation learning. Modalities and open access (✓ = available, ✗ = unavailable) are shown as binary indicators. Pretraining sources are listed with the number of training locations in parentheses. Training objectives include contrastive loss (CL) and reconstruction loss (Rec.). Interaction modeling (Inter.) is analyzed via partial information decomposition (PID), which quantifies modality interactions in terms of redundancy (R), uniqueness (U), and synergy (S).

Spatial Representation Learning. A core challenge in geospatial machine learning is learning transferable representations of locations from spatially distributed, heterogeneous data. A substantial body of work has focused on local spatial representation learning, where models are trained for specific cities or regions, often using self-supervised objectives (Wang et al., 2020; 2025). For instance, GPS2Vec created local encoders for 120 UTM zones based on Flickr image tags (Yin et al., 2019). Subsequent studies expanded this approach by incorporating visual features into the representations (Yin et al., 2021). The Population Dynamics Foundation Model (PDFM) used graph neural networks to learn multimodal embeddings for county and postal code regions in the US, integrating signals from Internet search trends, Google Maps data, activity levels, and environmental factors to model population dynamics (Agarwal et al., 2024). However, this research did not address geographic generalization beyond those areas represented in the training data.

This local trend evolved into global coordinate encoders inspired by CLIP-style contrastive training (Radford et al., 2021). For example, the CSP model aligned encoded coordinates with images from the iNaturalist and FMoW datasets (Mai et al., 2023). Meanwhile, GeoCLIP employed a similar approach, using Random Fourier Features to encode coordinates and aligning them with street view imagery (Vivanco et al., 2023; Tancik et al., 2020). To tackle the issue of unevenly distributed image data in previous methods, SatCLIP introduced a global model that utilized Spherical Harmonics and SIREN networks for coordinate encoding, learning representations through contrastive alignment with globally distributed Sentinel-2 satellite imagery (Klemmer et al., 2025; Rußwurm et al., 2024). Most recently, GAIR pioneered a multimodal setting by training encoders for coordinates, street view imagery, and satellite image modalities, employing a pairwise contrastive loss framework (Liu et al., 2025).

A detailed overview of all baseline models is provided in Table [1](https://arxiv.org/html/2510.13774v1#S2.T1) and Appendix [F.3](https://arxiv.org/html/2510.13774v1#A6.SS3).
Our work builds upon these approaches by integrating new modalities essential for urban prediction tasks and proposing a cohesive, unified multimodal encoder trained using a novel technique that combines contrastive loss with masking.

On the Limitations of Multimodal Contrastive Alignment. While contrastive foundation models demonstrated strong performance by aligning paired samples across different modalities, the information preserved in their embeddings is inherently limited. Contrastive losses, such as InfoNCE (van den Oord et al., 2018), primarily capture redundant information between modalities while neglecting unique modality-specific content and failing to capture synergistic interactions between them (Dufumier et al., 2025). This decomposition of information is formalized by the partial information decomposition (PID) framework (Williams & Beer, 2010; Dufumier et al., 2025), which provides a principled way to disentangle the different types of information shared between input modalities and a target variable. Prior work has explored intra-modal alignment through data augmentations, aiming to capture the uniqueness of individual modalities (Liang et al., 2023; Yuan et al., 2021) or even their synergistic relationships (Dufumier et al., 2025). However, such augmentations are often handcrafted and not well defined across all modalities. Another approach involves retrieval-augmented generation (RAG) (Lewis et al., 2020), in which the model’s representations can be used to query a database. This strategy has also been explored in the context of location representation learning (Dhakal et al., 2025). See Appendix [D.1](https://arxiv.org/html/2510.13774v1#A4.SS1) for additional discussion. We propose a novel integration of reconstruction loss and random modality masking to mitigate the shortcomings of using contrastive loss alone.

## 3 Methods

### 3.1 Architecture and Training with Stochastic Multimodal Fusion (SMF)

Our overall model architecture is illustrated in Figure [1](https://arxiv.org/html/2510.13774v1#S1.F1). Let $\mathcal{A}_{i}=\{m_{1},m_{2},\dots,m_{K}\}$ be a set of available input modalities at a location $i$. Each modality $m\in\mathcal{A}_{i}$ is processed via a fixed, pretrained encoder $f_{m}$ to extract a latent feature vector $\mathbf{h}_{m}$, which is then projected into a token $\mathbf{t}_{m}\in\mathbb{R}^{d}$. The tokens are fused by a Transformer encoder $\mathcal{T}_{\theta}$, parameterized by $\theta$, followed by average pooling to obtain the final representation $\mathbf{z}_{i}=\mathcal{T}_{\theta}(\{\mathbf{t}_{m}:m\in\mathcal{A}_{i}\})\in\mathbb{R}^{d}$ used for downstream tasks.

During training, we randomly mask a subset of modalities $\mathcal{M}_{i}\subset\mathcal{A}$,
and denote the inverse non-masked complement subset as $\overline{\mathcal{M}}_{i}=\mathcal{A}\setminus\mathcal{M}_{i}$. Both $\mathcal{M}_{i}$ and $\overline{\mathcal{M}}_{i}$ are passed through the modality-specific encoders $f_{m}$ and the multimodal Transformer encoder $\mathcal{T}_{\theta}$, yielding embeddings $\mathbf{z_{i}^{\mathcal{M}}}$ and $\mathbf{z_{i}^{\overline{\mathcal{M}}}}$, respectively. Finally, two decoder heads are applied on top of the fused representation, each implemented as a lightweight projection network. The first head performs Contrastive Location Alignment: the fused embeddings $\mathbf{z}_{i}$ and $\mathbf{z_{i}^{\overline{\mathcal{M}}}}$, derived from the masked subset $\mathcal{M}_{i}$ and its complement $\overline{\mathcal{M}}_{i}$, respectively, are passed through a shared decoder $\mathcal{E}$ and aligned via a symmetric InfoNCE objective (van den Oord et al., 2018), formalized as:

$$ $\displaystyle\mathcal{L}_{\text{contr}}$ $\displaystyle=-\frac{1}{N}\sum_{i=1}^{N}\Biggl[\log\frac{f(\mathbf{z}_{i}^{\mathcal{M}},\mathbf{z}_{i}^{\overline{\mathcal{M}}})}{\sum_{j=1}^{N}f(\mathbf{z}_{i}^{\mathcal{M}},\mathbf{z}_{j}^{\overline{\mathcal{M}}})}+\log\frac{f(\mathbf{z}_{i}^{\overline{\mathcal{M}}},\mathbf{z}_{i}^{\mathcal{M}})}{\sum_{j=1}^{N}f(\mathbf{z}_{i}^{\overline{\mathcal{M}}},\mathbf{z}_{j}^{\mathcal{M}})}\Biggr],$ $\displaystyle f(\mathbf{a},\mathbf{b})$ $\displaystyle\coloneqq\exp\!\left(\frac{\operatorname{sim}(\mathcal{E}(\mathbf{a}),\,\mathcal{E}(\mathbf{b}))}{\tau}\right),$ $$

where $N$ denotes the number of training examples in the mini-batch. Here, $\text{sim}(u,v)=\frac{u^{\top}v}{\|u\|\|v\|}$ denotes cosine similarity and $\tau$ is a learnable temperature. Views derived from the same geographic location form positive pairs, while views from different locations act as negatives. It is important to note that at both pretraining and inference time, the model can flexibly operate on an arbitrary subset of modalities, depending on data availability and task-specific requirements. For example, consider the case where six modalities exist in total, but at location $i$ data is available for only three: $a$, $b$, and $c$. In this case, the training loss can be computed using $\mathcal{M}_{i}={a}$ and $\overline{\mathcal{M}_{i}}={b,c}$. This flexibility enables the combination of large-scale datasets with differing modality compositions for pretraining.

The second head performs Latent Modality Reconstruction: a modality-specific projection network $g_{m}$ is trained to reconstruct the latent vector $\mathbf{h}_{m}$ for all modalities $m\in\mathcal{A}$, based on the fused representation $\mathbf{z}_{i}$. The loss is computed as the average mean squared error over all modalities:

$$ $\mathcal{L}_{\text{recon}}=\frac{1}{2|\mathcal{A}|}\sum_{m\in\mathcal{A}}\Bigl(\|g_{m}(\mathbf{z}_{i}^{\mathcal{M}})-\mathbf{h}_{m}\|_{2}^{2}+\|g_{m}(\mathbf{z}_{i}^{\overline{\mathcal{M}}})-\mathbf{h}_{m}\|_{2}^{2}\Bigr),$ $$

where all latent features are z-score normalized (per dimension, mean = 0, variance = 1) to ensure equal contribution across modalities. By operating in the latent space, the reconstruction reduces the
computational cost compared to full input reconstruction and encourages the model to focus on abstract representations rather than reconstructing input noise (Assran et al., 2023). The total training loss is a weighted sum of the two objectives: $\mathcal{L}_{\text{total}}=(1-\lambda)\cdot\mathcal{L}_{\text{contr}}+\lambda\cdot\mathcal{L}_{\text{recon}}$, where $\lambda$ controls the balance between alignment and reconstruction.

This training method, termed Stochastic Multimodal Fusion, enables the model to capture modality-interactions beyond redundant information, alleviating the shortcomings of contrastive loss outlined above. Intuitively, the reconstruction loss together with random masking of modalities encourages the model to retain unique information from individual modalities as well as synergistic information from sets of modalities that could help to reconstruct another modality, retaining similar information as required to solving downstream tasks. This is formalized as follows:
Lemma 1: Assume that for each downstream task $Y$, there exists at least a proxy modality or subset $S_{Y}\subseteq\mathcal{A}$ such that predicting $S_{Y}$ is at least as demanding as predicting $Y$: $I(\mathcal{A}\setminus S_{Y};\,Y)\;\leq\;I(\mathcal{A}\setminus S_{Y};\,S_{Y})$. Under this assumption, the SMF loss ($\mathcal{L}_{\text{total}}$) encourages $\mathcal{T}_{\theta}$ to retain redundant, synergistic, and unique information, thereby maximizes a lower bound on $I(m_{1},\dots,m_{K};Y)=R+S+\sum_{i}^{K}U_{i}$.
A formal proof is provided in Appendix [D.2](https://arxiv.org/html/2510.13774v1#A4.SS2).

### 3.2 Encoders and Data

To ensure fair comparisons and consistency with prior work, we adopt modalities and encoders closely aligned with those used in previous studies. Wherever applicable, we use the same architectures and freeze the encoder weights during training, following standard practice in GeoCLIP, and SatCLIP. Although end-to-end fine-tuning may yield additional performance gains, particularly when combined with parameter-efficient strategies (Hu et al., 2022; Mühlematter et al., 2024), we leave this direction for future work.

For pretraining, we build upon the Place Pulse 2.0 (PP 2.0) dataset (Dubey et al., 2016), which contains 110’988 locations with corresponding geographic coordinates and street-view images across 56 cities spanning all continents except Antarctica. From the PP 2.0 dataset, we hold out six cities for testing Cross-Regional Generalization on unseen regions. The remaining data is split into 80 percent for training and 20 percent for validation. While PP2.0 provides core geographic signals in the form of coordinates and street-view imagery, we further enrich each location with additional geospatial modalities to provide a more comprehensive representation of place. We refer to the resulting enriched multimodal dataset as PP2-M, which we release publicly for reproducibility and future research. Below, we describe each modality and the corresponding encoder used in our model.

Coordinates. We represent geographic coordinates (longitude and latitude) using the Equal Earth projection (Šavrič et al., 2019), followed by Random Fourier Features computed at multiple spatial scales (Tancik et al., 2020). These features are passed through a multi-layer perceptron (MLP). This approach, also adopted in GeoCLIP and GAIR, helps the model capture location information across resolutions (Vivanco et al., 2023; Liu et al., 2025).
Street-view imagery. We encode the PP2.0 images using the CLIP ViT-L/14 model, pretrained on large-scale image-text datasets (Radford et al., 2021). Thanks to its strong generalization capabilities and prior success in GeoCLIP (Vivanco et al., 2023), this model provides a reliable and consistent basis for evaluation.
Remote sensing imagery. We enrich the dataset with 12-channel multispectral Sentinel-2 imagery for each location (Drusch et al., 2012), following prior work that incorporates large-scale contextual signals (Klemmer et al., 2025). For encoding, we use the ViT-S/16 model pretrained on Sentinel-2 data (Wang et al., 2022), adopting the same configuration as in SatCLIP (Klemmer et al., 2025).
Cartographic basemaps. We extract cartographic basemaps from OpenStreetMap (OpenStreetMap contributors, 2017), generating patches at 300 m, 600 m, and 1200 m resolutions. These maps provide a multi-scale, human-interpretable source of geospatial information, including buildings, land cover, and transportation infrastructure (Mühlematter et al., 2024a). To encode the maps, we pretrain a ViT-B/16 backbone using the Masked Autoencoder (MAE) framework (He et al., 2022). The model is initialized with ImageNet-pretrained weights.
POIs. We augment each location with POI data from OpenStreetMap (OpenStreetMap contributors, 2017), collected within a 200 m radius. For each location, we extract the 15 nearest POIs and generate textual prompts describing their names, categories, and distances. These prompts are encoded using the BAAI/bge-small-en-v1.5 language model (Xiao et al., 2023), similar to prior work (Wang et al., 2025).
Multimodal Fusion Encoder. We use a single-block transformer with an embedding dimension of 768, eight attention heads, and learned positional encodings (Vaswani et al., 2017).
Further information about the dataset and encoders can be found in Appendix [E.1](https://arxiv.org/html/2510.13774v1#A5.SS1) and [F.1](https://arxiv.org/html/2510.13774v1#A6.SS1).

## 4 Experiments

### 4.1 Experimental Setting

We report results on *downstream prediction tasks* in urban environments. A small downstream model (linear regression or small MLP) is trained to predict the labels based on the location embedding generated with the pretrained and frozen UrbanFusion model (see [Figure 1](https://arxiv.org/html/2510.13774v1#S1.F1)). We first test Coordinate-Only Spatial Encoding, where the foundation models receives only geographical coordinates as input, and secondly investigate Multimodal Spatial Encoding, which uses additional modalities also at inference time. Both approaches are evaluated on out-of-sample locations within the same geographic region as the training data, representing an interpolation setting. To assess extrapolation, we test Cross-Regional Generalization by applying models to cities entirely outside the spatial extent of the training data. UrbanFusion and other methods trained on the PP2-M dataset are pretrained for 400 epochs. Pretraining details, training accuracy, and loss curves, are provided in Appendix [F.2](https://arxiv.org/html/2510.13774v1#A6.SS2).

We use a large suite of urban prediction datasets. Our primary source is the PP2-M dataset (Dubey et al., 2016), from which we select out-of-sample locations and assign target variables for multimodal tasks. Alternatively, for some Coordinate-Only downstream tasks, we use the locations provided directly by the corresponding task-specific datasets. We evaluate regression tasks involving Housing Prices, Energy Consumption, Urban Perception, Crime Incidence, and postal code-level Health, Socioeconomic, and Environmental Indicators. Classification tasks include Land Cover Prediction and Coarse-to-Fine Land Use Classification in Europe. Ridge regression and small MLPs are used for regression, while logistic regression and small MLPs are used for classification.

We compare UrbanFusion against the most relevant existing methods: SatCLIP (Klemmer et al., 2025), GeoCLIP (Vivanco et al., 2023), and GAIR (Liu et al., 2025). To ensure a fair evaluation, we include both the original versions of these models as well as variants trained on the PP2-M dataset for 400 epochs each. Additionally, we evaluate against other location representation approaches, including CSP (Mai et al., 2023), as well as local models such as both versions of GPS2Vec (Yin et al., 2019; 2021) and PDFM (Agarwal et al., 2024). As a simple baseline, we also include an Identity model that uses raw geographical coordinates directly as input, without any transformation. In the following result tables, we indicate below each method the dataset on which the model was trained.

Following prior work (Klemmer et al., 2025), raw geographical coordinates are concatenated to the model embeddings for evaluation. For each task, the dataset is split into 60% for training, 20% for validation and hyperparameter tuning, and 20% for testing. We report linear probing performance on the held-out test set, with hyperparameters optimized using the Optuna framework (Akiba et al., 2019).
Complete results, including MLP performance and additional baselines, are provided in Appendix [B](https://arxiv.org/html/2510.13774v1#A2), with further details on the datasets, baselines, evaluation protocols available in Appendix [E.2](https://arxiv.org/html/2510.13774v1#A5.SS2), [F.3](https://arxiv.org/html/2510.13774v1#A6.SS3), and [G](https://arxiv.org/html/2510.13774v1#A7), respectively.

### 4.2 Coordinate-Only Spatial Encoding

Table [2](https://arxiv.org/html/2510.13774v1#S4.T2) presents linear probing results for a widely studied use case of location representations: generating embeddings from raw geographical coordinates. UrbanFusion outperforms all other methods on 5 out of 8 datasets. Notably, UrbanFusion consistently outperforms the most closely related methods when all models are trained on the PP2-M dataset. The local model GPS2Vec achieves superior performance on the Housing Prices and Energy Consumption prediction tasks compared to UrbanFusion. This is at least partially due to the epistemic uncertainty in UrbanFusion, stemming from the limited training set of only 2’146 locations in the covered regions, whereas GPS2Vec may benefit from a denser coverage, as suggested by the significantly larger training dataset (see Table [1](https://arxiv.org/html/2510.13774v1#S2.T1)). Notably, UrbanFusion surpasses even Google’s PDFM, a domain-specific model specifically designed for ZIP code prediction, on this task, which we find particularly surprising. PDFM performs best on ZIP Code-level health indicator tasks, potentially due to the inclusion of internet search trends. Notably, it is the only model in this comparison where evaluated locations are not out-of-sample, as the published representations correspond to in-sample training locations.

Table: Table 2: Evaluation of coordinate-only spatial encoding. Best results are shown in bold, second-best are underlined, and top scores across all models trained on PP2-M are indicated in [brackets].

In contrast to prior work by Klemmer et al. (2025), but consistent with findings from Agarwal et al. (2024), GeoCLIP consistently outperforms SatCLIP, even when evaluated against the proposed fine-grained model, L40. This performance gap can be attributed to two key factors: (1) satellite imagery captures broader spatial context compared to street-view images, providing less fine-grained information; and (2) Random Fourier Features offer a more effective representation of high-frequency spatial variations than Spherical Harmonics (see Tancik et al. (2020) and Ji et al. (2024)). A visual analysis (Appendix Figure [4](https://arxiv.org/html/2510.13774v1#A2.F4)) further supports this finding, showing the embeddings reduced to 3D by PCA and mapped to RGB color codes. While UrbanFusion produces smooth and fine-grained representations, GeoCLIP exhibits less spatial granularity, likely due to the absence of explicitly multimodal and multiscale inputs. In contrast, SatCLIP’s location encoder fails to adequately model high-frequency intra-city variation, limiting its performance on urban prediction tasks.

### 4.3 Multimodal Spatial Encoding

Table [3](https://arxiv.org/html/2510.13774v1#S4.T3) presents results using linear probing for an additional use case of GeoFMs: incorporating not only coordinates but also auxiliary location information such as satellite or street view imagery. Since the benefit of a specific modality often depends on the downstream task, we select the subset of modalities for each model based on validation performance. While UrbanFusion is the only model that natively supports multimodal inputs by fusing them into a single embedding vector, we compare to the baseline models by concatenating the representations from the individual modality encoders. For example, for GAIR, we concatenate representations derived separately from coordinates, satellite, and street-view inputs.

UrbanFusion outperforms all other methods in 4 out of 6 downstream tasks when they are trained on the PP2-M dataset. Compared to the original versions of each model, UrbanFusion still achieves the highest performance on 3 datasets. The closest competitor overall is GeoCLIP, trained on the MP-16 dataset, which achieves the best performance on 3 out of 6 datasets. It is worth emphasizing that GeoCLIP is trained on approximately 65 times more locations than UrbanFusion (see Table [1](https://arxiv.org/html/2510.13774v1#S2.T1)).

Table: Table 3: Evaluation of multi-modal spatial encoding. Best results are shown in bold, second-best are underlined, and top scores across all models trained on PP2-M are indicated in [brackets].

### 4.4 Cross-Regional Generalization

As training a global, fine-grained GeoFM for location representations is often infeasible due to compute and data limitations, a third use case of such models is zero-shot generalization to unseen regions. Since coordinate encodings do not generalize well to unseen regions, the model receives only non-coordinate modalities at inference time, using the same multimodal evaluation setup as in the previous section. Table [4](https://arxiv.org/html/2510.13774v1#S4.T4) presents results using linear models on cities that were held out during training. To ensure that no model has seen samples from the unseen regions during training, we evaluate only models trained on the PP2-M dataset, excluding the selected cities during training.

UrbanFusion outperforms other methods, ranking first on 5 out of 6 tasks. The closest competitor is GAIR, which performs best on one tasks. Both models clearly outperform unimodal baselines such as SatCLIP and GeoCLIP, highlighting the benefits of multimodal representation learning for geographic generalization.

Table: Table 4: Evaluation of cross-region spatial encoding. Best results are shown in bold, second-best are underlined.

Additional insights into the superior performance of multimodal models over single-modal models on various tasks are provided by the k-means clusters shown in Figure [2](https://arxiv.org/html/2510.13774v1#S4.F2). While UrbanFusion produces spatially smooth clusters that still preserve high-frequency variations, the street-view representations of GeoCLIP lack spatial smoothness despite the obvious conceptual similarities of nearby city districts. In contrast, the satellite-view representations of SatCLIP fail to capture high-frequency changes.

Figure: (a) UrbanFusion
Refer to caption: Figures/NewYork_Out-of-region_Coordinates_masked_epoch_400_UrbanFusionV2_1_0_clusters_V3.png

### 4.5 Empirical Analysis of Information Preservation in GeoFM Models for Location Representation

As discussed in Section [2](https://arxiv.org/html/2510.13774v1#S2) and [3.1](https://arxiv.org/html/2510.13774v1#S3.SS1), UrbanFusion captures synergistic and unique information, unlike other methods that rely on contrastive losses. To validate this property empirically, we construct synthetic data with unique random values per modality, location-specific values as redundant information, and predicting the sum of the unique values as synergistic information (see Appendix [C.2](https://arxiv.org/html/2510.13774v1#A3.SS2) for details). As shown in Figure [3(a)](https://arxiv.org/html/2510.13774v1#S4.F3.sf1), contrastive-only methods fail to capture the full information spectrum. GeoCLIP cannot recover synergistic content due to its unimodal design. GAIR shows some unique feature recovery, likely due to convergence artifacts rather than objective design. This is supported by average unique signal contributions in the first encoder layer (Figure [3(b)](https://arxiv.org/html/2510.13774v1#S4.F3.sf2)). UrbanFusion reliably captures all mutual information components. Finally, our results are in line with real-world findings in Dhakal et al. (2025), that contrastive methods often neglect task-relevant unique signals.

Figure: (a) Predictive performance across different information types in Multi-Modal Spatial Encoding.
Refer to caption: Figures/synthetic_in_region.png

### 4.6 Training with Incomplete Multimodal Data

All modalities used for training UrbanFusion are open-source and globally available, except street-view imagery, which limits performance in regions without coverage (Klemmer et al., 2025). However, existing multimodal methods typically require that each location has all modalities present. Ideally, a GeoFM shall reuse a collection of existing datasets for pretraining, even if not geographically aligned. We therefore conduct an ablation study where each location has only coordinates and one modality (Bimodal). Despite using only $\sim$25% of the data per modality, this reduced-modality setting retains 99.35% of the performance of the model trained with complete modality pairs, matching or outperforming it in 40% of the evaluated domains. These results show that UrbanFusion is both flexible and data-efficient, as even incomplete modality pairs can be used for effective pretraining. Aligning a single modality with coordinates already yields strong performance. This result is consistent with Girdhar et al. (2023) and addresses a key limitation of prior work, which required fully paired modalities during training. More details are in Appendix [C.1](https://arxiv.org/html/2510.13774v1#A3.SS1).

## 5 Discussion and conclusion

We introduced UrbanFusion, a geofoundation model that learns fused, multimodal representations of urban locations via Stochastic Multimodal Fusion (SMF). The model flexibly supports varying modality combinations, generalizes across diverse urban settings, and achieves state-of-the-art performance on 41 tasks despite being trained on a smaller dataset compared to other methods. Its ability to train on modality subsets further enables scaling with large, heterogeneous global datasets. Limitations include imperfect temporal alignment across modalities and our focus on urban environments, which may limit applicability to rural or global-scale settings. Future directions include incorporating temporal data (e.g., satellite image sequences) for dynamic tasks such as land-use change detection or urban growth forecasting, as well as expanding to new modalities like mobility traces, social media, or location descriptions.
Beyond geospatial data, SMF offers a general-purpose framework for multimodal learning, capturing cross-modal interactions without handcrafted augmentations. We hope this work advances scalable, transferable GeoAI representations and inspires broader innovation in multimodal learning.

## Reproducibility statement

Our results are fully reproducible with the code repository(^1^11[https://github.com/DominikM198/UrbanFusion](https://github.com/DominikM198/UrbanFusion)), which contains scripts for preprocessing, training, and evaluation to facilitate accurate reproduction of our experiments. Additionally, to enhance accessibility for a broader audience, we provide tutorial notebooks that guide users step-by-step through the process. We have also published our modified version of the Place Pulse 2.0 dataset(^2^22https://huggingface.co/datasets/DominikM198/PP2-M). This dataset includes original SVI, extracted POIs, OSM BaseMaps, and Remote Sensing data. Finally, we provide trained model weights(^3^33https://huggingface.co/DominikM198/UrbanFusion) to further support reproducibility and facilitate downstream applications.

## References

- Agarwal et al. (2024)
Mohit Agarwal, Mimi Sun, Chaitanya Kamath, Arbaaz Muslim, Prithul Sarker, Joydeep Paul, Hector Yee, Marcin Sieniek, Kim Jablonski, Yael Mayer, David Fork, Sheila de Guia, Jamie McPike, Adam Boulanger, Tomer Shekel, David Schottlander, Yao Xiao, Manjit Chakravarthy Manukonda, Yun Liu, Neslihan Bulut, Sami Abu-El-Haija, Arno Eigenwillig, Parth Kothari, Bryan Perozzi, Monica Bharel, Von Nguyen, Luke Barrington, Niv Efron, Yossi Matias, Greg Corrado, Krish Eswaran, Shruthi Prabhakara, Shravya Shetty, and Gautam Prasad.
General geospatial inference with a population dynamics foundation model.
*arXiv preprint arXiv:2411.07207*, 2024.
- Akiba et al. (2019)
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama.
Optuna: A next-generation hyperparameter optimization framework.
In *Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining*, 2019.
- Ashby (2017)
Matthew P J Ashby.
Crime open database (code).
[https://osf.io/zyaqn](https://osf.io/zyaqn), 2017.
Accessed 8 July 2025.
- Assran et al. (2023)
Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas.
Self-supervised learning from images with a joint-embedding predictive architecture.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2023.
- Ba et al. (2016)
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton.
Layer normalization.
*arXiv preprint arXiv:1607.06450*, 2016.
- Barber & Agakov (2004)
David Barber and Felix Agakov.
The im algorithm: a variational approach to information maximization.
*Advances in neural information processing systems*, 16(320):201, 2004.
- Bishop & Nasrabadi (2006)
Christopher M Bishop and Nasser M Nasrabadi.
*Pattern recognition and machine learning*, volume 4.
Springer, 2006.
- Brown et al. (2020)
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.
Language models are few-shot learners.
In *Advances in Neural Information Processing Systems (NeurIPS)*, 2020.
- Che et al. (2025)
Lin Che, Yizi Chen, Tanhua Jin, Martin Raubal, Konrad Schindler, and Peter Kiefer.
Unsupervised urban land use mapping with street view contrastive clustering and a geographical prior.
*arXiv preprint arXiv:2504.17551*, 2025.
- Chen et al. (2020)
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton.
A simple framework for contrastive learning of visual representations.
In *International Conference on Machine Learning (ICML)*, 2020.
- Christie et al. (2018)
Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee.
Functional map of the world.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2018.
- Copernicus Land Monitoring Service & European Environment Agency (2021)
Copernicus Land Monitoring Service & European Environment Agency.
Urban atlas land cover land use 2018 (vector), europe.
[https://land.copernicus.eu/en/products/urban-atlas/urban-atlas-2018](https://land.copernicus.eu/en/products/urban-atlas/urban-atlas-2018), 2021.
Accessed 8 July 2025.
- Daniel & Pettit (2025)
Claire Daniel and Chris Pettit.
The use of urban analytics in strategic planning – a case study of the greater sydney region plan.
*Computers, Environment and Urban Systems*, 117:102249, 2025.
- Data Commons (2024)
Data Commons.
Data commons 2024, cdc places, electronic dataset.
[https://datacommons.org](https://datacommons.org), 2024.
Accessed 29 May 2024.
- Deng et al. (2009)
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.
Imagenet: A large-scale hierarchical image database.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2009.
- Department for Energy Security and Net Zero (2024)
Department for Energy Security and Net Zero.
Postcode level all domestic meters electricity 2023.
[https://www.gov.uk/government/statistics/postcode-level-electricity-statistics-2023](https://www.gov.uk/government/statistics/postcode-level-electricity-statistics-2023), 2024.
Accessed 8 July 2025.
- Dhakal et al. (2025)
Aayush Dhakal, Srikumar Sastry, Subash Khanal, Adeel Ahmad, Eric Xing, and Nathan Jacobs.
Range: Retrieval augmented neural fields for multi-resolution geo-embeddings.
*arXiv preprint arXiv:2502.19781*, 2025.
- Dollinger et al. (2025)
Johannes Dollinger, Damien Robert, Elena Plekhanova, Lukas Drees, and Jan Dirk Wegner.
Climplicit: Climatic implicit embeddings for global ecological tasks.
In *Proceedings of the International Conference on Learning Representations (ICLR)*, 2025.
- Dosovitskiy et al. (2021)
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.
An image is worth 16x16 words: Transformers for image recognition at scale.
In *Proceedings of the International Conference on Learning Representations (ICLR)*, 2021.
- Drusch et al. (2012)
M. Drusch, U. Del Bello, S. Carlier, O. Colin, V. Fernandez, F. Gascon, B. Hoersch, C. Isola, P. Laberinti, P. Martimort, A. Meygret, F. Spoto, O. Sy, F. Marchese, and P. Bargellini.
Sentinel-2: Esa’s optical high-resolution mission for gmes operational services.
*Remote Sensing of Environment*, 120:25–36, 2012.
- Dubey et al. (2016)
Abhimanyu Dubey, Nikhil Naik, Devi Parikh, Ramesh Raskar, and César A. Hidalgo.
Deep learning the city: Quantifying urban perception at a global scale.
In *European Conference on Computer Vision (ECCV)*, pp. 196–212, 2016.
- Dufumier et al. (2025)
Benoit Dufumier, Javiera Castillo Navarro, Devis Tuia, and Jean-Philippe Thiran.
What to align in multimodal contrastive learning?
In *Proceedings of the International Conference on Learning Representations (ICLR)*, 2025.
- Free Map Tools (2024)
Free Map Tools.
Uk postcodes with latitude and longitude.
[https://www.freemaptools.com/download-uk-postcode-lat-lng.htm](https://www.freemaptools.com/download-uk-postcode-lat-lng.htm), 2024.
- Gao et al. (2021)
Tianyu Gao, Xingcheng Yao, and Danqi Chen.
SimCSE: Simple contrastive learning of sentence embeddings.
In *Empirical Methods in Natural Language Processing (EMNLP)*, 2021.
- Girdhar et al. (2023)
Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra.
Imagebind: One embedding space to bind them all.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2023.
- Google LLC (2007)
Google LLC.
Street view static api.
[https://developers.google.com/maps/documentation/streetview](https://developers.google.com/maps/documentation/streetview), 2007.
Accessed 14 August 2025.
- Gorelick et al. (2017)
N. Gorelick, M. Hancher, M. Dixon, S. Ilyushchenko, D. Thau, and R. Moore.
Google earth engine: Planetary-scale geospatial analysis for everyone.
*Remote Sensing of Environment*, 202:18–27, 2017.
- He et al. (2020)
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.
Momentum contrast for unsupervised visual representation learning.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2020.
- He et al. (2022)
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.
Masked autoencoders are scalable vision learners.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2022.
- Hendrycks & Gimpel (2016)
Dan Hendrycks and Kevin Gimpel.
Gaussian error linear units (GELUs).
*arXiv preprint arXiv:1606.08415*, 2016.
- Herbrich et al. (2006)
Ralf Herbrich, Tom Minka, and Thore Graepel.
Trueskill: a bayesian skill rating system.
In *Advances in Neural Information Processing Systems (NeurIPS)*, 2006.
- Hong et al. (2023)
Ye Hong, Emanuel Stüdeli, and Martin Raubal.
Evaluating geospatial context information for travel mode detection.
*Journal of Transport Geography*, 113:103736, 2023.
- Hu et al. (2022)
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.
LoRA: Low-rank adaptation of large language models.
In *Proceedings of the International Conference on Learning Representations (ICLR)*, 2022.
- Jakubik et al. (2023)
Johannes Jakubik, Sujit Roy, Christopher Phillips, Paolo Fraccaro, Denys Godwin, Bianca Zadrozny, Daniel Szwarcman, Carlos Gomes, Gabby Nyirjesy, Blair Edwards, Daiki Kimura, Naomi Simumba, Linsong Chu, S. Karthik Mukkavilli, Devyani Lambhate, Kamal Das, Ranjini Bangalore, Dario Oliveira, Michal Muszynski, Kumar Ankur, Muthukumaran Ramasubramanian, Iksha Gurung, Sam Khallaghi, Hanxi Li, Michael Cecil, Maryam Ahmadi, Fatemeh Kordi, Hamed Alemohammad, Manil Maskey, Raghu Kiran Ganti, Kommy Weldemariam, and Rahul Ramachandran.
Foundation models for generalist geospatial artificial intelligence.
*arXiv preprint arXiv:2310.18660*, 2023.
- Ji et al. (2024)
Xinlong Ji, Fangneng Zhan, Shijian Lu, Shi-Sheng Huang, and Hua Huang.
Mixlight: Borrowing the best of both spherical harmonics and gaussian models.
*arXiv preprint arXiv:2404.12768*, 2024.
- Kingma & Ba (2015)
Diederik P. Kingma and Jimmy Ba.
Adam: A method for stochastic optimization.
In *Proceedings of the International Conference on Learning Representations (ICLR)*, 2015.
- Klemmer et al. (2025)
Konstantin Klemmer, Esther Rolf, Caleb Robinson, Lester Mackey, and Marc Rußwurm.
Satclip: Global, general-purpose location embeddings with satellite imagery.
*Proceedings of the AAAI Conference on Artificial Intelligence*, 39(4):4347–4355, 2025.
- Koldasbayeva et al. (2024)
Diana Koldasbayeva, Polina Tregubova, Mikhail Gasanov, Alexey Zaytsev, Anna Petrovskaia, and Evgeny Burnaev.
Challenges in data-driven geospatial modeling for environmental research and practice.
*Nature Communications*, 15(1):10700, 2024.
- Lewis et al. (2020)
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela.
Retrieval-augmented generation for knowledge-intensive nlp tasks.
In *Advances in Neural Information Processing Systems (NeurIPS)*, volume 33, pp. 9459–9474, 2020.
- Liang et al. (2023)
Paul Pu Liang, Zihao Deng, Martin Ma, James Zou, Louis-Philippe Morency, and Ruslan Salakhutdinov.
Factorized contrastive learning: Going beyond multi-view redundancy.
In *Advances in Neural Information Processing Systems (NeurIPS)*, 2023.
- Liu et al. (2025)
Zeping Liu, Fan Zhang, Junfeng Jiao, Ni Lao, and Gengchen Mai.
Gair: Improving multimodal geo-foundation model with geo-aligned implicit representations.
*arXiv preprint arXiv:2503.16683*, 2025.
- Loshchilov & Hutter (2019)
Ilya Loshchilov and Frank Hutter.
Decoupled weight decay regularization.
In *Proceedings of the International Conference on Learning Representations (ICLR)*, 2019.
- Mai et al. (2023)
Gengchen Mai, Ni Lao, Yutong He, Jiaming Song, and Stefano Ermon.
Csp: Self-supervised contrastive spatial pre-training for geospatial-visual representations.
In *International Conference on Machine Learning (ICML)*, 2023.
- Mai et al. (2025)
Gengchen Mai, Yiqun Xie, Xiaowei Jia, Ni Lao, Jinmeng Rao, Qing Zhu, Zeping Liu, Yao-Yi Chiang, and Junfeng Jiao.
Towards the next generation of geospatial artificial intelligence.
*International Journal of Applied Earth Observation and Geoinformation*, 136:104368, 2025.
- Mühlematter et al. (2024)
Dominik J. Mühlematter, Michelle Halbheer, Alexander Becker, Dominik Narnhofer, Helge Aasen, Konrad Schindler, and Mehmet Ozgur Turkoglu.
LoRA-ensemble: Efficient uncertainty modelling for self-attention networks.
*arXiv preprint arXiv:2405.14438*, 2024.
- Mühlematter et al. (2024a)
Dominik J. Mühlematter, Sebastian Schweizer, Chenjing Jiao, Xue Xia, Magnus Heitzler, and Lorenz Hurni.
Probabilistic road classification in historical maps using synthetic data and deep learning.
*arXiv preprint arXiv:2410.02250*, 2024a.
- Mühlematter et al. (2024b)
Dominik J. Mühlematter, Nina Wiedemann, Yanan Xin, and Martin Raubal.
Spatially-aware station based car-sharing demand prediction.
*Journal of Transport Geography*, 114:103765, 2024b.
- Oord et al. (2018)
Aaron van den Oord, Yazhe Li, and Oriol Vinyals.
Representation learning with contrastive predictive coding.
*arXiv preprint arXiv:1807.03748*, 2018.
- OpenStreetMap contributors (2017)
OpenStreetMap contributors.
Planet dump retrieved from https://planet.osm.org.
[https://www.openstreetmap.org](https://www.openstreetmap.org), 2017.
- Oshan et al. (2019)
Taylor M. Oshan, Ziqi Li, Wei Kang, Levi J. Wolf, and A. Stewart Fotheringham.
mgwr: A python implementation of multiscale geographically weighted regression for investigating process spatial heterogeneity and scale.
*ISPRS International Journal of Geo-Information*, 8(6), 2019.
- Paszke et al. (2019)
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.
PyTorch: an imperative style, high-performance deep learning library.
In *Advances in Neural Information Processing Systems (NeurIPS)*, 2019.
- Pedregosa et al. (2011)
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay.
Scikit-learn: Machine learning in Python.
*Journal of Machine Learning Research*, 12:2825–2830, 2011.
- Radford et al. (2021)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.
Learning transferable visual models from natural language supervision.
In *International Conference on Machine Learning (ICML)*, 2021.
- Robbins & Monro (1951)
Herbert Robbins and Sutton Monro.
A stochastic approximation method.
*The Annals of Mathematical Statistics*, 22(3):400–407, 1951.
- Rolf et al. (2021)
E. Rolf, J. Proctor, T. Carleton, I. Bolliger, V. Shankar, M. Ishihara, B. Recht, and S. Hsiang.
A generalizable and accessible approach to machine learning with global satellite imagery.
*Nature Communications*, 12(1):4392, 2021.
- Rußwurm et al. (2024)
Marc Rußwurm, Konstantin Klemmer, Esther Rolf, Robin Zbinden, and Devis Tuia.
Geographic location encoding with spherical harmonics and sinusoidal representation networks.
In *Proceedings of the International Conference on Learning Representations (ICLR)*, 2024.
- Sun et al. (2024)
M. Sun, C. Kamath, M. Agarwal, A. Muslim, H. Yee, D. Schottlander, S. Bavadekar, N. Efron, S. Shetty, and G. Prasad.
Community search signatures as foundation features for human-centered geospatial modeling.
*arXiv preprint arXiv:2410.22721*, 2024.
- Tancik et al. (2020)
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng.
Fourier features let networks learn high frequency functions in low dimensional domains.
In *Advances in Neural Information Processing Systems (NeurIPS)*, 2020.
- United Nations (2019)
United Nations.
*World Urbanization Prospects: The 2018 Revision*.
United Nations, New York, 2019.
- United Nations (2024)
United Nations.
*World Cities Report 2024: Cities and Climate Action*.
United Nations, Nairobi, 2024.
- U.S. Geological Survey, Earth Resources Observation and Science Center (2024)
U.S. Geological Survey, Earth Resources Observation and Science Center.
Annual national land cover database (nlcd) collection 1.1, 1985–2023.
[https://www.usgs.gov/centers/eros/science/annual-national-land-cover-database](https://www.usgs.gov/centers/eros/science/annual-national-land-cover-database), 2024.
Accessed 8 July 2025.
- van den Oord et al. (2018)
Aaron van den Oord, Yazhe Li, and Oriol Vinyals.
Representation learning with contrastive predictive coding.
*arXiv preprint arXiv:1807.03748*, 2018.
- Van Horn et al. (2018)
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie.
The iNaturalist species classification and detection dataset.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2018.
- Vaswani et al. (2017)
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.
Attention is all you need.
In *Advances in Neural Information Processing Systems (NeurIPS)*, 2017.
- Vivanco et al. (2023)
Vicente Vivanco, Gaurav Kumar Nayak, and Mubarak Shah.
Geoclip: Clip-inspired alignment between locations and images for effective worldwide geo-localization.
In *Advances in Neural Information Processing Systems (NeurIPS)*, 2023.
- Wang et al. (2023)
Sherrie Wang, Jordi Laguarta, and Thomas Friedel.
Combining deep learning and street view imagery to map smallholder crop types.
In *AGU Fall Meeting Abstracts*, 2023.
- Wang et al. (2025)
Xinglei Wang, Tao Cheng, Stephen Law, Zichao Zeng, Lu Yin, and Junyuan Liu.
Multi-modal contrastive learning of urban space representations from poi data.
*Computers, Environment and Urban Systems*, 120:102299, 2025.
- Wang et al. (2022)
Yi Wang, Nassim Ait Ali Braham, Zhitong Xiong, Chenying Liu, Conrad M Albrecht, and Xiao Xiang Zhu.
Ssl4eo-s12: A large-scale multi-modal, multi-temporal dataset for self-supervised learning in earth observation.
*arXiv preprint arXiv:2211.07044*, 2022.
- Wang et al. (2020)
Zhecheng Wang, Haoyuan Li, and Ram Rajagopal.
Urban2vec: Incorporating street view imagery and pois for multi-modal urban neighborhood embedding.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, 2020.
- Wiedemann (2025)
Nina Wiedemann.
*Spatio–temporal effects in GeoAI: From predictability to evaluation*.
PhD dissertation, ETH Zurich, Zürich, Switzerland, 2025.
URL [https://doi.org/10.3929/ethz-b-000738075](https://doi.org/10.3929/ethz-b-000738075).
- Williams & Beer (2010)
Paul L. Williams and Randall D. Beer.
Nonnegative decomposition of multivariate information.
*arXiv preprint arXiv:1004.2515*, 2010.
- Wright (2025)
Jake Wright.
London house price data.
[https://www.kaggle.com/datasets/jakewright/house-price-data](https://www.kaggle.com/datasets/jakewright/house-price-data), 2025.
Accessed 8 July 2025.
- Xiao et al. (2023)
Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff.
C-pack: Packaged resources to advance general chinese embedding.
*arXiv preprint arXiv:2309.07597*, 2023.
- Yao et al. (2018)
Yao Yao, Jinbao Zhang, Ye Hong, Haolin Liang, and Jialv He.
Mapping fine scale urban housing prices by fusing remotely sensed imagery and social media data.
*Transactions in GIS*, 22(2):561–581, 2018.
- Yin et al. (2019)
Yifang Yin, Zhenguang Liu, Ying Zhang, Sheng Wang, Rajiv Ratn Shah, and Roger Zimmermann.
Gps2vec: Towards generating worldwide gps embeddings.
In *Proceedings of the 27th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems*, pp. 416–419, 2019.
- Yin et al. (2021)
Yifang Yin, Ying Zhang, Zhenguang Liu, Yuxuan Liang, Sheng Wang, Rajiv Ratn Shah, and Roger Zimmermann.
Learning multi-context aware location representations from large-scale geotagged images.
In *Proceedings of the 29th ACM International Conference on Multimedia*, pp. 899–907, 2021.
- Yuan et al. (2021)
Xin Yuan, Zhe Lin, Jason Kuen, Jianming Zhang, Yilin Wang, Michael Maire, Ajinkya Kale, and Baldo Faieta.
Multimodal contrastive training for visual representation learning.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2021.
- Šavrič et al. (2019)
Bojan Šavrič, Tom Patterson, and Bernhard Jenny.
The equal earth map projection.
*International Journal of Geographical Information Science*, 33(3):454–465, 2019.

## Appendix

## Appendix A Abbreviations

Encoded coordinates

Street view image

Remote sensing imagery

Cartographic basemap from OpenStreetMap

Point of interest

Multilayer Perceptron

Vision Transformer

Graphics Processing Unit

Geo-Foundation Model

## Appendix B Detailed Results Overview

In this section, we provide and discuss further results, including additional baselines, downstream learners, and ablations.

### B.1 Coordinate-Only Spatial Encoding

Table [10](https://arxiv.org/html/2510.13774v1#A9.T10) presents linear probing and MLP results for a widely studied use case of location representations: generating embeddings from raw geographical coordinates. UrbanFusion outperforms all other methods on 5 out of 8 datasets for linear probing and consistently outperforms the most closely related baselines when all models are trained on the PP2-M dataset, as extensively discussed in Section [4.2](https://arxiv.org/html/2510.13774v1#S4.SS2). These findings are generally consistent with the MLP-based results, where UrbanFusion achieves superior performance on the majority of tasks. Notably, we observe that MLP results tend to be less stable across the board, often performing worse than simple linear probes for certain tasks and methods, especially on smaller datasets. This instability suggests that the added model capacity of MLPs can lead to overfitting when training data is limited or noisy, despite extensive hyperparameter tuning. An exception is the Identity baseline, which directly uses raw geographical coordinates as input. This method benefits significantly from the added capacity of MLPs, which aligns with expectations: since raw coordinates are not embedded in a higher-dimensional space, MLPs are better suited to model the nonlinear decision boundaries required to extract meaningful patterns directly from the coordinate space.

Table [11](https://arxiv.org/html/2510.13774v1#A9.T11) provides detailed results for the Place Pulse 2.0 Urban Perception task, where all methods exhibit low $R^{2}$ values. This can be attributed to the inherent noise and bias in the dataset, which is based on human perceptions of street view imagery. The subjective nature of the annotations, influenced by factors such as weather conditions or traffic, likely reduces the spatial correlation of the target labels. Even more flexible models such as MLPs tend to perform poorly on this task, potentially due to overfitting to these noisy and weakly spatially structured labels.

Table [12](https://arxiv.org/html/2510.13774v1#A9.T12) and Table [13](https://arxiv.org/html/2510.13774v1#A9.T13) present detailed results for ZIP Code-level prediction tasks using linear probing and MLPs, respectively. On average across all categories, UrbanFusion achieves the highest performance among all evaluated methods, even outperforming Google’s PDFM model, which was explicitly designed for this task. While PDFM performs strongly on health-related tasks, likely due to its use of web search trend data, it underperforms on environmental tasks, possibly due to the absence of visual inputs such as street view images, satellite imagery, or cartographic basemaps. In general, we find that MLPs perform worse than linear models on ZIP Code-level tasks, often resulting in catastrophic overfitting for some baselines and different random seeds, despite extensive hyperparameter tuning. This can be explained by the relatively small number of ZIP codes within the evaluated urban areas, which limits the amount of training data and increases the risk of overfitting when using higher-capacity models.

Figure [5](https://arxiv.org/html/2510.13774v1#A2.F5) presents representations produced by UrbanFusion and several baseline models, reduced to three dimensions via principal component analysis (PCA) and mapped to RGB color space. A visual inspection reveals that UrbanFusion produces the most detailed and spatially coherent representations, likely due to its multimodal fusion strategy. GAIR and GeoCLIP follow closely, although with slightly less granularity. In contrast, SatCLIP yields coarser and less structured spatial patterns. This aligns with its lower quantitative performance and can be largely attributed to its use of Spherical Harmonics for coordinate encoding (for a more detailed discussion, see Section [C.4](https://arxiv.org/html/2510.13774v1#A3.SS4)).

GPS2Vec (tag) yields smooth and detailed representations, whereas GPS2Vec (visual) exhibits more high-frequency variation, resulting in lower spatial smoothness. CSP displays limited spatial variation, suggesting lower sensitivity to local features. Finally, PDFM exhibits limited spatial smoothness—an expected outcome, given that its design limits spatial resolution to the ZIP code level, assigning identical representations to all locations within the same ZIP code during inference.

Figure: (a) UrbanFusion (PP2-M)
Refer to caption: Figures/Portland_In-region_Coordinates_epoch_400_UrbanFusionV2_1_1_2_3_4_V3.png

Figure: (a) SatCLIPL40 (PP2-M)
Refer to caption: Figures/Portland_In-region_Coordinates_epoch_400_SatCLIPL40trained_1_0_V3.png

### B.2 Multimodal Spatial Encoding

Table [14](https://arxiv.org/html/2510.13774v1#A9.T14) presents results using linear probing and MLP for an additional use case of GeoFMs: incorporating not only coordinates but also auxiliary location information such as satellite imagery or street view imagery as input to foundation models. Since the benefit of a specific modality often depends on the downstream task, we select the subset of modalities for each model based on validation performance. While UrbanFusion is the only model that natively supports multimodal inputs, we also evaluate baseline models by concatenating representations obtained from individual modality encoders. For example, for GAIR, we concatenate representations derived separately from coordinates, satellite, and street view inputs. Results for linear probing are extensively discussed in Section [4.3](https://arxiv.org/html/2510.13774v1#S4.SS3) of the main paper. For MLPs, UrbanFusion outperforms all other methods on 3 out of 6 datasets, and on 5 out of 6 datasets when comparing only to baselines trained on the PP2-M dataset. In general, performance improves compared to Coordinate-Only encoding for most methods that support additional modalities, highlighting the limitations of purely coordinate-based encoders. Moreover, when analyzing the selected modalities for each downstream task, it becomes evident that modality selection is highly task-dependent. Notably, previously underexplored modalities such as cartographic basemaps and point-of-interest data also serve as valuable predictors.

Table [15](https://arxiv.org/html/2510.13774v1#A9.T15) presents results for the Urban Perception task. All baselines that support encoding street view imagery perform significantly better than those relying solely on coordinate inputs. This highlights the potentially low spatial autocorrelation and biases introduced during the data collection process, as discussed in Section [B.1](https://arxiv.org/html/2510.13774v1#A2.SS1).

Tables [16](https://arxiv.org/html/2510.13774v1#A9.T16) and [17](https://arxiv.org/html/2510.13774v1#A9.T17) provide detailed results for ZIP Code-level tasks using linear probing and MLPs, respectively, while Table [18](https://arxiv.org/html/2510.13774v1#A9.T18) lists the selected modalities for each task. Again, it is evident that models capable of encoding additional modalities beyond coordinates consistently achieve higher performance.

### B.3 Cross-Regional Generalization

Training a global, fine-grained GeoFM for location representations is often infeasible due to computational and data limitations. A third important use case for such models is zero-shot generalization to entirely unseen regions. Since coordinate encodings do not generalize well to locations outside the training distribution, the model is provided only with non-coordinate modalities at inference time, using the same multimodal evaluation setup as described in Section [4.3](https://arxiv.org/html/2510.13774v1#S4.SS3). Table [19](https://arxiv.org/html/2510.13774v1#A9.T19) presents results for linear probing on cities that were completely held out during training. To ensure that no model has been exposed to these regions, we evaluate only models trained on the PP2-M dataset while explicitly excluding the selected cities from the training set.

UrbanFusion outperforms all other methods, ranking first on 5 out of 6 tasks for linear probing. The closest competitor is GAIR, which achieves the best performance on one task. Both models clearly outperform unimodal baselines such as SatCLIP and GeoCLIP, demonstrating the advantages of multimodal representation learning for geographic generalization. A similar pattern is observed for MLP results, where UrbanFusion achieves the highest performance on most tasks.

Table [20](https://arxiv.org/html/2510.13774v1#A9.T20) provides detailed results for the Place Pulse 2.0 Urban Perception task, where UrbanFusion achieves the strongest performance. Tables [21](https://arxiv.org/html/2510.13774v1#A9.T21), [22](https://arxiv.org/html/2510.13774v1#A9.T22), and [23](https://arxiv.org/html/2510.13774v1#A9.T23) present detailed results for ZIP Code-level tasks, in which GAIR outperforms all other methods for linear probing. Interestingly, for MLP models, raw coordinates yield the best results. This may be explained by the small number of input dimensions, which can reduce the risk of overfitting in downstream learners, despite extensive hyperparameter tuning. In contrast, some higher-capacity models exhibit extremely poor fits for specific tasks and random seeds, as shown in Table [22](https://arxiv.org/html/2510.13774v1#A9.T22).

Additional insights into the superior performance of multimodal models compared to single-modal models are provided by the k-means clustering visualizations shown in Figure [6](https://arxiv.org/html/2510.13774v1#A2.F6). UrbanFusion produces spatially smooth clusters that still preserve high-frequency variations, indicating an effective balance between global coherence and local detail. In contrast, the street view representations produced by GeoCLIP lack spatial smoothness, while the satellite-view representations of SatCLIP fail to capture high-frequency spatial changes. The multimodal GAIR model produces clusters that are smoother than those of GeoCLIP, but less smooth than those of UrbanFusion and SatCLIP.

Figure: (a) UrbanFusion (PP2-M), SV+RS+OSM+POI
Refer to caption: Figures/NewYork_Out-of-region_Coordinates_masked_epoch_400_UrbanFusionV2_1_0_clusters_V3.png

## Appendix C Ablation Studies

This chapter presents additional ablation studies aimed at gaining deeper insights into both UrbanFusion and relevant baselines. Specifically, we investigate whether UrbanFusion can be trained on incomplete modality sets; analyze how different methods capture modality interactions through the lens of the partial information decomposition framework; evaluate the effect of various loss function choices for UrbanFusion; and examine the limitations of the Spherical Harmonics-based location encoding used in the state-of-the-art model SatCLIP.

### C.1 Training with Incomplete Multimodal Data

UrbanFusion utilizes open-source and globally accessible data sources, including Sentinel-2 imagery and OpenStreetMap (OSM) data, both of which are freely available worldwide. In contrast, street view imagery presents a significant limitation due to its restricted spatial availability. This constraint has previously been shown to degrade the performance of GeoFM’s in regions where such imagery is absent (Klemmer et al., 2025).

Although it is technically feasible to collect data on a global scale, training a GeoFM model typically requires that all modalities be available for each location. To improve data efficiency and enable better reuse of existing datasets, which may be incomplete or not geographically aligned, we explore training strategies that tolerate partially missing modalities.

To this end, we conduct an ablation study using incomplete multimodal datasets, which we refer as Partial. Specifically, we randomly drop modalities across the training data as follows:

- •
25% of the locations contain coordinates along with all four modalities
- •
25% contain coordinates and three modalities
- •
25% contain coordinates and two modalities
- •
25% contain coordinates and only one modality

We also examine a Bimodal training setup in which each location includes only coordinates and a single modality. In this case, each of the four possible coordinate-modality combinations constitutes 25% of the training set. To accelerate training, we sample batches such that all examples within a batch share the same set of available modalities.

As shown in Table [4.6](https://arxiv.org/html/2510.13774v1#S4.SS6), the model maintains competitive performance even under these constrained conditions. Remarkably, the Bimodal setup leverages only approximately 25% of the data per modality, yet still achieves strong results. This finding indicates that aligning a single modality with coordinates can be effective for pretraining, consistent with observations from (Girdhar et al., 2023). The approach broadens the scope of pretraining by enabling the use of arbitrary, modality-specific datasets, without requiring full geospatial alignment across all modalities. These results highlight the flexibility and data efficiency of our method, making it well-suited for training in regions where comprehensive modality coverage is lacking.

Table: Table 5: UrbanFusion performance with varying modality input. All: full modalities; Partial: coords + 2–5 modalities; Bimodal: coords + 1 modality. Results are Coordinates-Only / Multimodal / Cross-Regional. Best scores are in bold, the second-best underlined.

### C.2 Experiments on Synthetic Data Demonstrating Empirical Information Decomposition

This section presents a complementary ablation study aimed at deepening our understanding of how multimodal contrastive alignment and Stochastic Multimodal Fusion (SMF) preserve or discard different types of information. As outlined in Chapter [2](https://arxiv.org/html/2510.13774v1#S2), contrastive alignment of multiple modalities using loss functions such as InfoNCE is theoretically prone to preserving redundant information, while unique (modality-specific) and synergistic information tends to be neglected (Dufumier et al., 2025; Dhakal et al., 2025). In UrbanFusion, we address this limitation by augmenting Contrastive Location Alignment with Latent Modality Reconstruction.

In real-world scenarios, we often deal with high-dimensional input data such as street view imagery or multispectral remote sensing data. In these cases, unambiguously assigning feature dimensions to either redundant or unique modality-specific information is inherently challenging. To enable systematic analysis of the preserved information, we design carefully controlled low-dimensional feature representations and auxiliary tasks, an approach inspired by similar analyses conducted in non-spatial multimodal models (see (Dufumier et al., 2025)).

#### C.2.1 Synthetic Data Generation

Contrastive spatial representation learning models differ fundamentally from other multimodal models such as CLIP (Radford et al., 2021) or CoMM (Dufumier et al., 2025), primarily due to the use of high-capacity location encoders that directly process geographical coordinates as one of the modalities. Consequently, the notion of redundant information in these models includes any signal that is useful for geolocalization.

In our experiments, we focus on aligning geographical coordinates with two synthetic modalities. To simulate geolocatable signals, we assign to each modality and each location two feature dimensions within the range $[0,1]$, which uniquely identify the location. These dimensions constitute the redundant information shared across both modalities and the coordinate representation, as illustrated in Figure [7](https://arxiv.org/html/2510.13774v1#A3.F7). Designing feature dimensions that capture solely unique (modality-specific) information is more challenging. Even randomly sampled noise per location can unintentionally assist geolocalization, similar to how postal codes can implicitly encode spatial structure. To address this, we introduce a third feature dimension to each modality. During training, this dimension is batch-augmented: a single random value is sampled from the range $[0,1]$ and assigned to all locations within the batch. During inference, however, this dimension contains independently sampled values per location. This strategy ensures that the third feature dimension has zero mutual information with the geolocation task, thereby functioning as a truly unique, non-redundant modality-specific signal.

We sample 40’000 equally spaced locations covering a rectangular area defined by latitudes 36.9 and 37.1, and longitudes -122.1 and -121.9. From this dataset, we use all locations in the third quadrant (36.9 - 37.0 latitude and from -122.1 to -122.0 longitude) for Cross-Regional Generalization experiments. This helps us evaluate the extrapolation capabilities of our models to unseen regions. The remaining locations are randomly sampled, with 80% used for pretraining the spatial models and 20% reserved for validation, as shown in Figure [7](https://arxiv.org/html/2510.13774v1#A3.F7).

Figure: Figure 7: Synthetic data generation. Each location includes two modalities, each represented by a three-dimensional vector. The first two dimensions contain localization-relevant information. The third dimension consists of random values during inference, and during training, is batch-augmented to remain constant across samples. This enforces zero mutual information with location, effectively decoupling unique information from redundant information.
Refer to caption: Figures/synthetic_data.png

#### C.2.2 Architecture and Pretraining

For all compared methods, we use the same location encoder: Random Fourier Features (Tancik et al., 2020) configured identically to those used in our other experiments (Vivanco et al., 2023), followed by a multilayer perceptron (MLP) with two hidden layers of 128 dimensions each. The modality encoders take the three-dimensional input vectors and process them through a single hidden layer with a dimensionality of four. The resulting final representation has a dimensionality of 9, corresponding to three dimensions per modality. This setup is designed to strike a balance. On one hand, it avoids the need for aggressive compression, since tasks can be solved by simply copying the input. On the other hand, it prevents excessive overparameterization, which would make the experiments less representative. In practice, input modalities often have much higher dimensionality than the learned spatial representations.

We train all models using stochastic gradient descent (SGD) with a learning rate of 0.0003, momentum, and a cosine decay schedule (Robbins & Monro, 1951). This choice is motivated by the well-understood convergence behavior of this optimizer. We apply no weight decay in order to isolate the optimization dynamics induced purely by the loss functions. All models are trained for 250 epochs, ensuring approximate convergence.

We compare UrbanFusion with and without Latent Modality Reconstruction against GAIR, as well as against GeoCLIP, which uses only modality 1 as input.

#### C.2.3 Tasks

To evaluate the types of information captured by the learned location representations, we formulate a set of targeted tasks designed to probe for redundant, unique, and synergistic information. The experimental setup is as follows:

Redundant information in our setup corresponds to geolocation (i.e., the shared signal between the modalities and the coordinates). To quantify this, we regress the learned location embeddings onto the original geographical coordinates.

To measure unique (modality-specific) information, we focus on the third feature dimension of each modality, which contains random values that are uninformative for geolocation. The task is to reconstruct this feature dimension from the shared location representation. Since this dimension is intentionally made independent of the geolocation, successful reconstruction implies preservation of modality-unique information.

Synergistic information refers to signals that emerge only when multiple modalities are combined. To assess this, we define a task in which the target is the sum of the unique feature dimensions (i.e., the third dimension) from both modalities. A regression model is trained to predict this sum from the location embeddings. This task cannot be solved using information from a single modality alone, requiring cross-modal synergy in the learned representation.

#### C.2.4 Evaluation Protocol

We evaluate all tasks on out-of-sample locations. For the Multimodal Spatial Encoding, we compute a fused UrbanFusion representation; for comparison methods, we concatenate the outputs of the individual modality-specific encoders. For each task, we train a ridge regression model using five-fold cross-validation. The regularization parameter $\alpha$ is selected from 100 logarithmically spaced values in the range $[10^{-4},10^{4}]$. Parameter tuning is performed using closed-form leave-one-out cross-validation on the training folds. For the uniqueness task, we report the average $R^{2}$ score across both modalities’ reconstruction regressors.

### C.3 Effectiveness of Combining Contrastive Learning with Reconstruction Loss

As discussed in Section [3.1](https://arxiv.org/html/2510.13774v1#S3.SS1), there is theoretical motivation for combining contrastive learning with a reconstruction loss. While contrastive learning has proven highly effective for spatial representation learning, it primarily emphasizes features relevant for geolocalization, often overlooking modality-specific information that, although less critical for localization, may be valuable for downstream tasks.

Incorporating a reconstruction loss addresses this limitation by encouraging the model to capture richer, modality-specific details in addition to localization cues. To evaluate this empirically, we trained models using contrastive loss only (CL), reconstruction loss only (Rec.), and a combination of both (CL+Rec.). The results, presented in Table [6](https://arxiv.org/html/2510.13774v1#A3.T6), show that assigning a higher weight to the reconstruction objective improves performance in Multimodal and Cross-Regional Generalization settings, whereas contrastive learning alone is more effective for Coordinate-Only encoding. Although certain tasks benefit more from one objective than the other, the combined approach consistently produces the strongest location encoder overall.

Table: Table 6: UrbanFusion performance with varying loss functions. Results are Coordinates-Only / Multimodal / Cross-Regional. Best scores are in bold, the second best are underlined.

### C.4 Limitations of Spherical Harmonics for Modeling Urban Areas with High-Frequency Variations

Spherical Harmonics are a widely used approach for global location representation learning due to their ability to model arbitrary functions on the sphere without introducing artifacts (Rußwurm et al., 2024). This method has demonstrated high effectiveness for spatial location encoding (Klemmer et al., 2025), particularly also in ecological applications (Dollinger et al., 2025). However, in our evaluation, consistent with the results in Agarwal et al. (2024), the performance of Spherical Harmonics is limited in urban tasks. This is because they model smooth functions on the sphere and therefore struggle to capture high-frequency variations (Tancik et al., 2020; Ji et al., 2024), which are essential for distinguishing fine-grained spatial patterns in urban environments, as shown in Figure [4](https://arxiv.org/html/2510.13774v1#A2.F4).

The expressive power of Spherical Harmonics can be increased by raising the order of the Legendre polynomials, allowing for the modeling of higher-frequency variations on the Earth’s surface. However, the number of basis functions grows on the order of $\mathcal{O}(n^{2})$ with the polynomial order, resulting in very high-dimensional feature vectors. For example, the authors of SatCLIP recommend setting the $L$ hyperparameter to 40 for local tasks such as housing price prediction. In their implementation, this corresponds to using Legendre polynomials up to degree 39 ($l=0\dots 39$), as degrees are included up to $L-1$.

Given the limited performance observed, we conducted an ablation study in which we increased the $L$ hyperparameter to 100 for even more local expressiveness, the highest value supported for analytical calculation in the codebase of Klemmer et al. (2025). The results of this study are shown in Table [7](https://arxiv.org/html/2510.13774v1#A3.T7). Even with the increased order resulting in 10’000 basis functions, the capacity to capture high-frequency variations remains limited, while computing these functions incurs substantial additional costs and leads to unstable training.

Table: Table 7: SatCLIP performance with varying hyperparameter $L$. Results are Coordinates-Only / Multimodal / Cross-Regional. Best scores are in bold, the second best are underlined. Time forward pass is measured as a single forward pass through the module using analytic spherical harmonic expressions, implemented as in Klemmer et al. (2025), with a batch size of 256 on an NVIDIA RTX 3090 GPU.

## Appendix D Learning Redundant, Unique, and Synergistic Information

### D.1 On the Limitations of Multimodal Contrastive Alignment

While contrastive foundation models demonstrated strong performance by aligning paired samples across different modalities, the information preserved in their embeddings is inherently limited. Contrastive losses, such as InfoNCE (van den Oord et al., 2018), primarily capture redundant information between modalities while neglecting unique modality-specific content and failing to capture synergistic interactions between them (Dufumier et al., 2025). This decomposition of information is formalized by the partial information decomposition (PID) framework (Williams & Beer, 2010; Dufumier et al., 2025), which provides a principled way to disentangle the different types of information shared between input modalities and a target variable.

More concretely, let $Y$ represent a downstream task variable (e.g., land use), and let $m_{1}$ and $m_{2}$ denote two input modalities (e.g., satellite and street view imagery). The mutual information $I$ between ${m_{1},m_{2}}$, and $Y$ can be decomposed as:

$$ $I(m_{1},m_{2};Y)=R+U_{m_{1}}+U_{m_{2}}+S,$ (1) $$

where:

- •
$R$ is the redundant information, present in both $m_{1}$ and $m_{2}$.
- •
$U_{m_{1}}$ and $U_{m_{2}}$ are the unique, modality-specific contributions from $m_{1}$ and $m_{2}$ respectively.
- •
$S$ is the synergistic information that is only accessible when combining $m_{1}$ and $m_{2}$ jointly.

This decomposition ensures:

$$ $\displaystyle I(m_{1};Y)$ $\displaystyle=R+U_{m_{1}}$ (2) $\displaystyle I(m_{2};Y)$ $\displaystyle=R+U_{m_{2}}$ (3) $$

Standard contrastive objectives such as InfoNCE typically use modality-specific encoders $f_{m}$, with representations $z_{m_{1}}=f_{m_{1}}(m_{1})$ and $z_{m_{2}}=f_{m_{2}}(m_{2})$. The loss is

$$ $\mathcal{L}_{\text{InfoNCE}}=-\log\frac{\exp\left(\mathrm{sim}(z_{m_{1}},z_{m_{2}})/\tau\right)}{\sum_{j}\exp\left(\mathrm{sim}(z_{m_{1}},z_{m_{2}}^{(j)})/\tau\right)},$ (4) $$

where $\mathrm{sim}(\cdot,\cdot)$ is a similarity metric (e.g., cosine similarity), and $\tau$ is a temperature hyperparameter. This objective encourages $z_{m_{1}}$ and $z_{m_{2}}$ to become maximally similar for matching pairs and dissimilar otherwise. Importantly, van den Oord et al. (2018) proved that optimizing this objective maximizes a lower bound on the mutual information between the two representations:

$$ $I(z_{m_{1}};z_{m_{2}})\geq\log(N)-\mathcal{L}_{\text{InfoNCE}},$ (5) $$

where $N$ is the total number of samples in a mini-batch. Since $z_{m_{1}}$ and $z_{m_{2}}$ are encodings of $m_{1}$ and $m_{2}$, respectively, this bound effectively encourages preservation of the information shared between the two modalities, by treating $I(z_{m_{1}};z_{m_{2}})$ as a proxy for $I(m_{1};m_{2})$.

This connection suggests that contrastive objectives implicitly assume that the mutual information between paired modalities, $I(m_{1};m_{2})$, serves as a good approximation for task-relevant information $I(m_{1};Y)$. Under this assumption, InfoNCE maximizes the redundant information shared across modalities. However, as shown by Dufumier et al. (2025), this training objective is blind to information that is unique to a single modality ($U_{m_{1}}$, $U_{m_{2}}$) or synergistic ($S$), that is information that only emerges from combining modalities. Such content is not aligned and is therefore suppressed in the learned representations, limiting downstream performance when tasks depend on modality-specific or joint signals (see Dufumier et al. (2025) for a rigorous treatment).

To address these limitations, prior work has explored intra-modal alignment through data augmentations, aiming to capture the uniqueness of individual modalities (Liang et al., 2023; Yuan et al., 2021) or even their synergistic relationships (Dufumier et al., 2025). However, such augmentations are often handcrafted and not well defined across all modalities. Another approach involves retrieval-augmented generation (RAG) (Lewis et al., 2020), in which the model’s representations can be used to query a database. This strategy has also been explored in the context of location representation learning (Dhakal et al., 2025).

Optimally, a foundation model should be trained to retain all relevant information, including redundant, unique, and synergistic components, in its embeddings while enabling flexible inference with any available subset of input modalities. We address these goals with UrbanFusion, which leverages the underlying Stochastic Multimodal Fusion (SMF) framework to capture the full spectrum of modality interactions through a unified fusion strategy, without relying on handcrafted data augmentations.

### D.2 Learning Multimodal Information with SMF

In the following, we present a proof that SMF retains all types of interactions. Proving that the embedding $z_{i}$ maximizes the retained mutual information is challenging as the downstream task $Y$ is unknown. Dufumier et al. (2025) circumvented this difficulty by posing a strong assumption, claiming that the mutual information between an input modality $m$ and its augmentation $m^{\prime}$, $I(m;m^{\prime})$, is the same as $I(m;Y)$. Here, we propose a significantly weaker assumption on $Y$: We assume that the mutual information between the inputs and $Y$ is lower equal than the information between the inputs and another modality or a set of modalities:

Assumption 1:
For each downstream task $Y$, there exists at least a proxy modality or subset $S_{Y}\subseteq\mathcal{A}$ such that predicting $S_{Y}$ is at least as demanding as predicting $Y$:

$$ $I(\mathcal{A}\setminus S_{Y};\,Y)\;\leq\;I(\mathcal{A}\setminus S_{Y};\,S_{Y}).$ (A) $$

Lemma 1: The SMF loss ($\mathcal{L}_{\text{total}}$) encourages $\mathcal{T}_{\theta}$ to retain redundant, synergistic, and unique information, thereby maximizes a lower bound on $I(m_{1},\dots,m_{K};Y)=R+S+\sum_{i}^{K}U_{i}$.

Proof:
For two random (masked/complement) views of modalities from the same location, $\mathcal{L}_{contr}$ maximizes a variational lower bound on the mutual information between the corresponding representations:

$$ $I(\mathbf{z}^{\mathcal{M}};\mathbf{z}^{\overline{\mathcal{M}}})\;\geq\;\log N-\mathcal{L}_{\mathrm{contr}},$ $$

where N is the batch size. Thus $\mathcal{L}_{contr}$ increases a computable lower bound on cross-modal shared information, i.e., redundant information R as shown by Oord et al. (2018); Dufumier et al. (2025).
(For clarity we omit expectation notation; all bounds are understood in expectation over the data, the masking distribution, and negative sampling.)

The reconstruction head reconstructs the latent $\mathbf{h}_{m}$ from the fused representation $\mathbf{z}$ with $\mathcal{L}_{recon}$. Minimizing this MSE loss is equivalent to maximizing the average log-likelihood under a Gaussian variational decoder $q_{\phi}^{(m)}$ with fixed covariance Bishop & Nasrabadi (2006):

$$ $\displaystyle q_{\phi}^{(m)}(\mathbf{h}_{m}\mid\mathbf{z})\;$ $\displaystyle=\;\mathcal{N}\!\big(\mathbf{h}_{m};\ g_{\phi}^{(m)}(\mathbf{z}),\,\sigma_{m}^{2}I\big),$ (6) $\displaystyle\log q_{\phi}^{(m)}(\mathbf{h}_{m}\mid\mathbf{z})$ $\displaystyle=-\tfrac{1}{2\sigma^{2}}\,\mathbb{\|}g_{\phi}^{(m)}(\mathbf{z})-\mathbf{h}_{m}\|^{2}-\tfrac{d_{m}}{2}\log\!\big(2\pi\sigma_{m}^{2}\big)$ (7) $$

where $d_{m}$ is the dimensionality of $\mathbf{h}_{m}$. Using

$$ $\displaystyle I(\mathbf{z};\mathbf{h}_{m})$ $\displaystyle=H(\mathbf{h}_{m})-H(\mathbf{h}_{m}\mid\mathbf{z}),\;H(\mathbf{h}_{m}\mid\mathbf{z})=-\log p(\mathbf{h}_{m}\mid\mathbf{z}),$ (8) $\displaystyle I(\mathbf{z};\mathbf{h}_{m})$ $\displaystyle=H(\mathbf{h}_{m})+\log p(\mathbf{h}_{m}\mid\mathbf{z})$ (9) $$

and replacing the intractable true conditional $p(\mathbf{h}_{m}\mid\mathbf{z})$ with the $q_{\phi}^{(m)}$ , we obtain the Barber–Agakov lower bound Barber & Agakov (2004)

$$ $\displaystyle I(\mathbf{z};\mathbf{h}_{m})\;\geq\;H(\mathbf{h}_{m})+\log q_{\phi}^{(m)}(\mathbf{h}_{m}\mid\mathbf{z}).$ (10) $$

Since $H(\mathbf{h}_{m})$ is constant, minimizing $\mathcal{L}_{recon}$ directly maximizes a computable lower bound on $I(\mathbf{z};\mathbf{h}_{m})$. Since random subsets of modalities $M$ are masked during SMF training (see Section [3.1](https://arxiv.org/html/2510.13774v1#S3.SS1), any set of modalities will be used at some point to create the embedding $\mathbf{z}$. When $|M|=1$, $M$ contains a single input modality, the objective preserves that modality’s *unique* information $U$; when $|M|\geq 2$, $M$ contains multiple modalities, simultaneously reconstructing all $\{\mathbf{h}_{m}\}$ forces $\mathbf{z}$ to integrate complementary cues, thereby promoting *synergistic* $S$ information.

Finally, under Assumption 1, for each task $Y$ there exists at least one proxy subset $S_{Y}$ such that predicting $S_{Y}$ is at least as demanding as predicting $Y$. Hence minimizing the latent reconstruction loss maximizes Barber–Agakov–style computable lower bounds $\{I(\mathbf{z};\mathbf{h}_{m})\}_{m\in S_{Y}}$, prevents $z$ from discarding information required for $Y$. Together with the symmetric InfoNCE loss which increases the bound on $I(\mathbf{z}^{\mathcal{M}};\mathbf{z}^{\overline{\mathcal{M}}})$, the total loss $\mathcal{L}_{\mathrm{total}}$ maximizes two tractable mutual information surrogates and encourages retention of the PID components (redundant, unique, and synergistic) that matter for downstream tasks.

## Appendix E Data

### E.1 Pretraining

Figure: Figure 8: Urban areas covered by the PP2-M dataset and the corresponding data splits.
Refer to caption: Figures/distribution_PP2.png

For pretraining, we leverage the Place Pulse 2.0 (PP 2.0) dataset (Dubey et al., 2016), which comprises 110’988 locations, each with associated geographic coordinates and street view images. We enrich this dataset with additional modalities, referring to the resulting extended version as PP2-M. The dataset spans 56 cities across 28 countries on all continents except Antarctica. We divide it into three parts. First, we select seven cities from six distinct continents to construct a Cross-Regional Generalization split, enabling evaluation of the model’s ability to generalize beyond regions seen during training. From the remaining locations, we perform an 80/20 split into training and validation sets. The spatial distribution of the included urban regions is illustrated in Figure [8](https://arxiv.org/html/2510.13774v1#A5.F8), while the number of samples per city is detailed in Table [9](https://arxiv.org/html/2510.13774v1#A5.T9). The street view images were obtained from Google Street View (Google LLC, 2007) and have a resolution of $400\times 300$ pixels. To ensure compatibility with the CLIP backbone and other popular models, we apply standard preprocessing following established practices from GeoCLIP (Vivanco et al., 2023): each image is resized to $256\times 256$ pixels, center-cropped to $224\times 224$, converted to a float tensor, and normalized using mean values (0.485, 0.456, 0.406) and standard deviation values (0.229, 0.224, 0.225).

We enrich the dataset with remote sensing imagery by downloading Sentinel-2 Level-2A images (Drusch et al., 2012) acquired between January 1, 2024 and December 31, 2024, selecting only scenes with minimal cloud coverage. Each image patch includes the spectral bands: B01, B02, B03, B04, B05, B06, B07, B08, B08A, B09, B11, and B12, and has a resolution of $256\times 256$ pixels. Since the dataset contains 12 multispectral bands and our pretrained encoder accepts only 13 input channels (Wang et al., 2022), we append the B10 band as a zero-valued mask to match the expected input dimensionality. The raw reflectance values are normalized by dividing by 10’000, and the patches are center-cropped to $224\times 224$ pixels to align with the input resolution of the ViT-S/16 model (Dosovitskiy et al., 2021). The entire preprocessing pipeline for remote sensing imagery follows the same procedure as used in SatCLIP (Klemmer et al., 2025).

Cartographic basemaps offer a globally available and human-interpretable source of geographic information, capturing features such as buildings, land cover, and transportation networks (Mühlematter et al., 2024a). Despite their richness, this modality has been largely overlooked in prior work. To address this, we further enrich the PP 2.0 dataset with basemaps from OpenStreetMap tile server at zoom levels 15, 16, and 17 (OpenStreetMap contributors, 2017), corresponding to spatial resolutions of 1200 m, 600 m, and 300 m, respectively. The tiles were downloaded in May 2025. Each map tile is rendered at a resolution of $256\times 256$ pixels. For compatibility with our Masked Autoencoder (MAE) model for feature extraction from the basemaps (He et al., 2022), we resize each image to $224\times 224$ pixels using bilinear interpolation and apply channel-wise normalization using mean values $(0.485,0.456,0.406)$ and standard deviation values $(0.229,0.224,0.225)$.

To further enrich the dataset with semantic information about the built environment, we extract points of interest (POIs) from OpenStreetMap for each location in the dataset (OpenStreetMap contributors, 2017). For every location, we identify the 15 nearest POIs within an adaptive radius of up to 200 meters. This adaptive search radius captures highly local context in dense urban areas while ensuring sufficient coverage in sparser regions, following principles commonly used in geostatistical analysis (Oshan et al., 2019). We retain POIs with tags under the following key-value pairs: amenity, shop, leisure, tourism, healthcare, theatre, cinema, building=religious, building=transportation, and
public_transport=station. Entries tagged as parking, parking_space, bench, bicycle_parking, motorcycle_parking, post_box, and toilets are excluded. Each retained POI is assigned a single representative category, determined by prioritizing tags in the following order: amenity, leisure, religion, public_transport, shop, and tourism. If no relevant tag is present, the POI is labeled as healthcare if any tag contains the substring healthcare, or as museum if the name includes the word museum. Only POIs with both a defined type and a name are retained for further use. The final set of POIs for each location is used to construct a textual prompt that describes each POI’s name, category, and distance. An example of such a prompt is shown in Figure [9](https://arxiv.org/html/2510.13774v1#A5.F9).

Figure: Figure 9: Example points of interest (POIs) text prompt provided as input to a language model for a single location. The example corresponds to coordinates $40.025,-75.223$ in Philadelphia.

### E.2 Downstream Tasks

This study addresses prediction tasks in urban environments. We evaluate methods for Coordinate-Only Spatial Encoding, where models take only raw geographic coordinates as input, and explore Multimodal Spatial Encoding, which enhances spatial representations with additional contextual information. Both approaches are assessed on out-of-sample locations within the same geographic area as the training set, representing an interpolation scenario. To examine extrapolation, we evaluate Cross-Regional Generalization, where models are tested on cities entirely outside the spatial extent of the training data. Our experiments draw on a diverse collection of urban prediction datasets. The primary source is our PP2-M dataset, from which we select held-out locations and assign target variables for multimodal prediction tasks. Two additional datasets, covering the same region as PP2-M, are used to support large-scale experiments in the Coordinate-Only setting. The remainder of this section provides detailed descriptions of these datasets.

Housing Prices (Wright, 2025). This dataset includes valuable information on residential properties in London, including both historical and current market data. It includes property-specific attributes such as geographic coordinates, sale prices, and structural features like floor area. We restrict our analysis to properties located within the convex hull of the PP2-M training region in London, and further filter for transactions that occurred in the year 2023. The sale prices, which serve as the target variable for regression models, are log-transformed to reduce skewness in their distribution. After preprocessing, the dataset consists of 38’208 residential locations. In addition to using encoded geographic coordinates as inputs to downstream learners, we include several continuous features: num_bathrooms, num_bedrooms, num_living_rooms, and log_floor_area_sqm. We also incorporate categorical variables using one-hot encoding: tenure_type, property_category, and energy_rating. These features collectively support a more realistic evaluation in downstream modeling tasks.

Energy Consumption (Department for Energy Security and Net Zero, 2024). This dataset contains postcode-level electricity usage data for all domestic meters in the United Kingdom during the year 2023. It includes the number of electricity meters and the total energy consumption per postcode, measured in kilowatt-hours (kWh). We focus on the London region by filtering the convex hull of the training area defined in PP2-M for evaluation purposes. Next, we download the geographical centroids of all UK postcodes (Free Map Tools, 2024). For each centroid, we compute the mean energy consumption by aggregating the total energy consumption and the total number of meters, then dividing the total energy by the number of meters. This mean value serves as the target variable for regression models. The final dataset contains 60’326 distinct geographical locations.

Crime Incidence (Ashby, 2017). The Crime Open Database (CODE) provides detailed crime records for various United States cities for the year 2021. Each record includes the type of crime, the date of occurrence, and the geographical coordinates of the incident. We use the out-of-sample locations from the PP2-M dataset and construct a buffer of 500 meters around each location. We then count the number of crimes falling within each buffer. Due to the skewness of the resulting distribution, the counts are transformed using the natural logarithm of one plus the count, since there are also locations with zero crimes. This transformed value serves as the target variable for regression models. Additionally, we removed 17 locations to facilitate the evaluation of the PDFM model, which does not cover all locations. In total, there are 2’454 locations in the cities of Boston, Chicago, Houston, Los Angeles, Minneapolis, San Francisco, and Seattle for the Coordinate-Only and Multimodal settings, and 3’398 locations in New York City for Cross-Regional Generalization.

Urban Perception (Dubey et al., 2016). While the PP2-M dataset serves as the basis for pretraining our models, we use the out-of-sample locations and the six included downstream tasks to evaluate human perception of urban environments based on street view imagery across 56 cities. Specifically, 1’170’000 pairwise comparisons between street view images were collected from 81’630 online volunteers, who assessed six perceptual attributes: safe, lively, boring, wealthy, depressing, and beautiful. The Microsoft TrueSkill algorithm is applied to generate ranking scores for each image across all six attributes, allowing for quantitative comparison (Herbrich et al., 2006). These TrueSkill scores serve as the target variable for the regression models. We recognize the potential limitations of this method, as noted in previous studies (Dubey et al., 2016), including the reliance on virtual representations instead of in-person experiences. Nevertheless, we believe it is a valuable task for modeling human perception of urban environments. In total, we have 18’233 locations for Coordinate-Only and Multimodal settings, and 19’727 locations for Cross-Regional Generalization.

ZIP Code Tasks (Agarwal et al., 2024). We evaluate our method on postal code level prediction tasks in the United States, using datasets previously introduced and released by PDFM (Agarwal et al., 2024). Below, we describe the data acquisition process. The dataset includes the general geospatial benchmark introduced by Sun et al. (2024), accessed via Data Commons (2024) and the Earth Engine Catalog (Gorelick et al., 2017). All health-related tasks are based on the 2022 CDC PLACES metrics, which are available at the postal code resolution through Data Commons. Socioeconomic and environmental variables are selected following Rolf et al. (2021), including income, home value, night lights, tree cover, and elevation, with all data retrieved from either Data Commons or the Earth Engine Catalog. Additionally, we include postal code level poverty data from 2022, accessed through Data Commons. We augment the PP2-M dataset by incorporating out-of-sample locations and attaching the corresponding target variables for regression tasks. In total, we construct 28 downstream tasks across major urban areas, including Philadelphia, Denver, Atlanta, Portland, Houston, Minneapolis, Chicago, Seattle, Washington, D.C., Boston, San Francisco, Los Angeles for Coordinate-Only and Multimodal settings, and New York for Cross-Regional Generalization.

Land Cover (U.S. Geological Survey, Earth Resources Observation and Science Center, 2024). The Annual National Land Cover Database provides land cover data across the continental United States at a spatial resolution of 30 meters, comprising 16 land cover classes. To support comprehensive evaluation scenarios, we augment the out-of-sample locations of the PP2-M dataset with corresponding land use labels. Since our focus is on urban areas, some classes are underrepresented. To address this, we merge all forest-related categories into a single Forest class and exclude the category Emergent Herbaceous Wetlands. Further, we removed 44 locations to evaluate the PDFM model, which does not cover all locations. For both Coordinate-Only and Multimodal Encoding tasks, we focus on five land cover classes: Developed Open Space, Developed Low Intensity, Developed Medium Intensity, Developed High Intensity, and Forest. The resulting dataset contains 4’826 labeled observations spanning the urban areas of Philadelphia, Denver, Atlanta, Portland, Houston, Minneapolis, Chicago, Seattle, Washington, D.C., Boston, San Francisco, and Los Angeles. For the Cross-Regional Generalization setting, which focuses exclusively on the city of New York, we exclude the Open Water and Forest classes due to insufficient sample sizes, leaving four classes: Developed Open Space, Developed Low Intensity, Developed Medium Intensity, and Developed High Intensity, with a total of 3’394 labeled locations.

Coarse to Fine Land Use Classification (Copernicus Land Monitoring Service & European Environment Agency, 2021). The Urban Atlas Land Cover and Land Use 2018 dataset contains 27 categories at a spatial resolution of 10m, covering 785 Functional Urban Areas across Europe with populations exceeding 50’000. We augment the out-of-sample locations from PP2-M with corresponding land use labels. Due to a highly imbalanced distribution of classes, we evaluate this dataset through two distinct tasks. For all tasks, we exclude the category Construction sites due to their temporary nature and potential misalignment with modalities such as remote sensing or street view imagery, and retain only categories with more than 10 samples for each task. In the first task, we group the 27 categories (25 of which are present in the cities we analyze) into 7 supercategories, as illustrated in Table [8](https://arxiv.org/html/2510.13774v1#A5.T8). This results in 6’094 labeled locations for the Coordinate-Only and Multimodal settings across 7 classes, and 4’178 locations across 6 classes in the Cross-Regional Generalization setup, which includes the cities of Milan and Paris. The second task focuses on fine-grained land use classification using the original Urban Atlas categories. Despite the strong class imbalance, this setup allows us to evaluate model performance in highly challenging scenarios, including few-shot cases. This yields 6’109 locations across 18 classes for the Coordinate-Only and Multimodal settings, and 4’178 locations across 13 classes for Cross-Regional Generalization.

**Table 8: Mapping of Urban Atlas land use classes to supercategories used in coarse-grained classification.**
| Urban Atlas Class (2018) | Supercategory |
| --- | --- |
| Continuous urban fabric (S.L. : ¿ 80%) | Urban fabric |
| Discontinuous dense urban fabric (S.L. : 50% - 80%) |  |
| Discontinuous medium density urban fabric (S.L. : 30% - 50%) |  |
| Discontinuous low density urban fabric (S.L. : 10% - 30%) |  |
| Discontinuous very low density urban fabric (S.L. : ¡ 10%) |  |
| Other roads and associated land | Transportation |
| Fast transit roads and associated land |  |
| Railways and associated land |  |
| Port areas |  |
| Airports |  |
| Industrial, commercial, public, military and private units | Industrial & built-up |
| Isolated structures |  |
| Mineral extraction and dump sites |  |
| Green urban areas | Green & recreation |
| Sports and leisure facilities |  |
| Arable land (annual crops) | Cropland & pasture |
| Pastures |  |
| Permanent crops (vineyards, fruit trees, olive groves) |  |
| Complex and mixed cultivation patterns |  |
| Forests | Natural vegetation |
| Herbaceous vegetation associations (natural grassland, moors…) |  |
| Wetlands |  |
| Water | Water & unused land |
| Land without current use |  |
| Construction sites | Excluded from task |

**Table 9: List of urban areas in the PP2-M dataset with corresponding sample counts for training, validation, and cross-regional generalization splits.**
| City | Training | Validation | Cross-Regional Generalization | Total Samples |
| --- | --- | --- | --- | --- |
| Atlanta | 3228 | 806 | - | 4034 |
| Berlin | 3188 | 796 | - | 3984 |
| Tokyo | 3029 | 757 | - | 3786 |
| Rio De Janeiro | - | - | 3659 | 3659 |
| Santiago | 2799 | 699 | - | 3498 |
| New York | - | - | 3398 | 3398 |
| Sydney | - | - | 3359 | 3359 |
| Toronto | 2630 | 657 | - | 3287 |
| Chicago | 2574 | 643 | - | 3217 |
| Houston | 2467 | 616 | - | 3083 |
| Warsaw | 2396 | 599 | - | 2995 |
| Sao Paulo | 2380 | 594 | - | 2974 |
| Moscow | 2304 | 576 | - | 2880 |
| Philadelphia | 2226 | 556 | - | 2782 |
| Melbourne | 2179 | 544 | - | 2723 |
| London | 2146 | 536 | - | 2682 |
| Montreal | 2096 | 524 | - | 2620 |
| Singapore | - | - | 2600 | 2600 |
| Cape Town | - | - | 2513 | 2513 |
| Paris | - | - | 2478 | 2478 |
| Denver | 1920 | 480 | - | 2400 |
| Munich | 1781 | 445 | - | 2226 |
| Rome | 1742 | 435 | - | 2177 |
| Bucharest | 1732 | 433 | - | 2165 |
| Madrid | 1725 | 431 | - | 2156 |
| Mexico City | 1677 | 419 | - | 2096 |
| Belo Horizonte | 1572 | 393 | - | 1965 |
| Portland | 1544 | 385 | - | 1929 |
| Lisbon | 1497 | 374 | - | 1871 |
| Johannesburg | 1494 | 373 | - | 1867 |
| Prague | 1384 | 346 | - | 1730 |
| Milan | - | - | 1720 | 1720 |
| Bangkok | 1272 | 318 | - | 1590 |
| Dublin | 1259 | 314 | - | 1573 |
| Guadalajara | 1239 | 309 | - | 1548 |
| Seattle | 1207 | 301 | - | 1508 |
| Barcelona | 1153 | 288 | - | 1441 |
| Taipei | 1109 | 277 | - | 1386 |
| Boston | 1062 | 265 | - | 1327 |
| Los Angeles | 1036 | 258 | - | 1294 |
| Stockholm | 940 | 234 | - | 1174 |
| Zagreb | 865 | 216 | - | 1081 |
| San Francisco | 815 | 203 | - | 1018 |
| Washington DC | 762 | 190 | - | 952 |
| Glasgow | 761 | 190 | - | 951 |
| Kiev | 711 | 177 | - | 888 |
| Minneapolis | 674 | 168 | - | 842 |
| Kyoto | 577 | 144 | - | 721 |
| Gaborone | 552 | 137 | - | 689 |
| Helsinki | 550 | 137 | - | 687 |
| Tel Aviv | 512 | 128 | - | 640 |
| Bratislava | 512 | 127 | - | 639 |
| Amsterdam | 510 | 127 | - | 637 |
| Hong Kong | 496 | 123 | - | 619 |
| Copenhagen | 401 | 100 | - | 501 |
| Valparaiso | 343 | 85 | - | 428 |
| Total | 73028 | 18233 | 19727 | 110988 |

## Appendix F Implementation Details

### F.1 Detailed Implementation of UrbanFusion’s Encoders and Fusion Modules

To ensure consistency with previous research and allow fair comparisons, we use modality-specific encoders that reflect those typically used in past studies. Whenever possible, we use the same pretrained network architectures and freeze their weights during training, following established practices in studies like GeoCLIP (Vivanco et al., 2023) and SatCLIP (Klemmer et al., 2025). While full fine-tuning, particularly with parameter-efficient techniques like Low-Rank Adaptation (LoRA), has shown promise across various modalities (Hu et al., 2022; Mühlematter et al., 2024), we will leave the exploration of this avenue for future research.

Our approach supports flexible integration of arbitrary encoders and modalities, each producing a dense representation of variable size. To enable multimodal fusion, these representations are linearly projected to a shared dimensionality. This projection can map each modality to a single or multiple tokens; in practice, we find that a single token per modality is sufficient. Each linear layer is followed by a GELU activation (Hendrycks & Gimpel, 2016), which introduces non-linearity, before a learned positional embedding is added to each token. In the following, we describe the implementation of all modality-specific encoders, the multimodal fusion module, and the associated decoders, as also illustrated in Figure [1](https://arxiv.org/html/2510.13774v1#S1.F1).

Location Encoder. To encode geographic coordinates (latitude and longitude), we first apply the Equal Earth projection (Šavrič et al., 2019), yielding a globally consistent 2D spatial representation. We then apply Random Fourier Features (RFF) with multi-scale encoding to capture spatial patterns at different resolutions (Tancik et al., 2020). Specifically, the projected coordinates are mapped into a 256-dimensional frequency space using sinusoidal functions (sine and cosine), resulting in a 512-dimensional RFF output. This process is performed independently for three spatial scales, using $\sigma$ values of $\sigma\in\{2^{0},2^{4},2^{8}\}$. Each scale-specific RFF embedding is then passed through a multi-layer perceptron (MLP) with three hidden layers of 1024 units each. The outputs of the MLPs across the three scales are subsequently summed to produce the final location representation. Our approach closely follows prior work such as GeoCLIP and GAIR (Vivanco et al., 2023; Liu et al., 2025), using the same RFF configuration and MLP hyperparameters to ensure consistency and comparability.

Street View Imagery Encoder. We adopt the CLIP ViT-L/14 architecture, pretrained on large-scale vision-language datasets (Radford et al., 2021; Dosovitskiy et al., 2021), to process street view images. Its demonstrated robustness across a wide range of tasks and prior use in GeoCLIP (Vivanco et al., 2023) supports consistent and fair evaluation.

Remote Sensing Imagery Encoder. To encode satellite imagery, we utilize the ViT-S/16 model trained on Sentinel-2 data using the momentum contrast (MoCo) approach (Dosovitskiy et al., 2021; Wang et al., 2022; He et al., 2020). This setup mirrors the configuration adopted in SatCLIP (Klemmer et al., 2025), ensuring methodological consistency.

Cartographic Basemap Encoder. Despite the rich semantic information contained in cartographic basemaps and the global availability of such data through sources like OpenStreetMap (OpenStreetMap contributors, 2017), their usage in visual representation learning remains limited. A key missing component is the lack of pretrained encoders tailored for feature extraction from this modality. To extract features, we fine-tune a Masked Autoencoder (MAE) (He et al., 2022) with a ViT-B/16 backbone (Dosovitskiy et al., 2021) that was initially pretrained on ImageNet (Deng et al., 2009). Our training setup employs a batch size of 256 over 100 epochs, with early stopping based on validation loss evaluated every 5 epochs. We use the AdamW optimizer with a base learning rate of $3\times 10^{-4}$, weight decay of $0.05$, and betas set to $(0.9,0.95)$ (Loshchilov & Hutter, 2019). A cosine learning rate decay schedule is applied with 2140 warmup steps. We adopt a masking ratio of 0.6 during training. The training data consists of the training split of our PP2-M dataset. Qualitative results of basemap reconstruction are shown in Figure [10](https://arxiv.org/html/2510.13774v1#A6.F10) for the validation set, and in Figure [11](https://arxiv.org/html/2510.13774v1#A6.F11) for inputs from regions not covered in the training data. The model successfully reconstructs the semantic context of the images, including features such as land cover, buildings, and streets. The blurry patches in the reconstruction come from the fact that the decoder receives the non-masked patches as input and reconstructs them as well, but since the loss is only applied to masked tokens, fine details in the non-masked patches are not preserved. During application of the model, we only use the encoder and conduct average pooling over all image tokens. We repeat this process for all three spatial scales and combine their embeddings with a learned projection.

Points of Interest (POI) Encoder. Each generated prompt is embedded using the BAAI/bge-small-en-v1.5 language model (Xiao et al., 2023), following prior work in multimodal location modeling (Wang et al., 2025).

Multimodal Fusion Encoder. We employ a single Transformer block to integrate information across all modalities (Vaswani et al., 2017). This design enables joint processing of heterogeneous input representations in an efficient and unified manner. The Transformer uses an embedding dimension of 768 and the GELU activation function (Hendrycks & Gimpel, 2016). [MASK] tokens are represented as zero vectors. Instead of relying on a [CLS] token, we apply average pooling over all output token embeddings to construct the final fused representation, which is then used for downstream tasks.

Contrastive Learning Head. Instead of directly using the downstream task representation for contrastive learning, as done in prior work (Vivanco et al., 2023; Klemmer et al., 2025; Liu et al., 2025; Mai et al., 2023), we observe that incorporating an additional decoding step improves performance, particularly when combined with Latent Modality Reconstruction, as noted in previous studies (e.g., Chen et al. (2020)). We introduce a lightweight decoder head composed of a LayerNorm layer (Ba et al., 2016), a GELU activation function (Hendrycks & Gimpel, 2016), and a linear projection to 512 dimensions, following established designs in prior work (Vivanco et al., 2023; Liu et al., 2025).

Reconstruction Head. For the Latent Modality Reconstruction objective, we employ a lightweight decoder head composed of a LayerNorm layer (Ba et al., 2016), a GELU activation function (Hendrycks & Gimpel, 2016), and a linear projection to 3842 dimensions. This output dimensionality corresponds to the concatenated length of all modality-specific latent representations produced by the pretrained encoders.

Figure: Figure 10: Cartographic basemap reconstruction using MAE on the validation set. The masked view is used as input to the encoder, while the reconstructed view is the output of the decoder. Since the decoder is trained to reconstruct only the masked tokens (not the visible ones), we additionally present a combined view that merges the input tokens with the reconstructed tokens for better visualization.
Refer to caption: Figures/epoch_99_MAE2_val_in_region_report.png

Figure: Figure 11: Cartographic basemap reconstruction using MAE for generalization to regions unseen during training. The masked view is input to the encoder, and the decoder outputs the reconstruction of the masked tokens. For improved visualization, we also show a combined view that merges the original input tokens with the reconstructed ones.
Refer to caption: Figures/epoch_99_MAE2_val_out_region_report.png

### F.2 Training UrbanFusion

We train UrbanFusion for 400 epochs using the AdamW optimizer (Loshchilov & Hutter, 2019) with a base learning rate of $1\times 10^{-4}$ and a weight decay of $1\times 10^{-5}$. The learning rate follows a cosine decay schedule with linear warm-up during the first 5% of training steps. The batch size is set to 2’560, and early stopping is applied based on the validation loss. At each training step, random masking schemes are sampled uniformly. The validation loss is computed as the mean over all possible masking schemes and is evaluated every ten epochs. For the loss function, we assign a weight $\lambda=0.0625$ to the Latent Modality Reconstruction term, chosen empirically to ensure that the magnitudes of both loss terms are similar during early training. The contrastive term uses a learned temperature parameter in the InfoNCE loss, initialized to 0.07. The implementation is based on torchvision 0.21.0 (Paszke et al., 2019).

Figure: (a) Training and validation loss per epoch.
Refer to caption: Figures/UrbanFusion_loss.png

To reduce memory usage and improve training speed, modality features are precomputed using the frozen modality-specific encoder networks. This reduces GPU memory consumption and speeds up training significantly. The final model is trained on a single NVIDIA RTX 3090 (24 GB) for approximately 8 hours. Figure [12(a)](https://arxiv.org/html/2510.13774v1#A6.F12.sf1) shows the training and validation loss curves. Training loss is averaged over an entire epoch with randomly sampled masks per batch, whereas validation loss is averaged over all possible masking combinations. The higher variability in the training loss compared to the validation loss is due to the random masking. Figure [12(b)](https://arxiv.org/html/2510.13774v1#A6.F12.sf2) reports the Top-5 retrieval accuracy within a batch, which converges to approximately 70% for a batch size of 2’560.

### F.3 Baselines

To benchmark UrbanFusion, we focus on recent Geo-Foundation Models (GeoFMs) that satisfy two key criteria. First, they must support coordinate encoding, that is, the ability to generate representations directly from raw geographic coordinates, with optional integration of additional modalities. Second, the models should be applicable across multiple urban areas, rather than being tailored to a specific city.

Based on these criteria, we select GeoCLIP (Vivanco et al., 2023), SatCLIP (Klemmer et al., 2025), and the preprint version of GAIR (Liu et al., 2025) as primary baselines. These models represent recent state-of-the-art approaches and can be trained on the same dataset as UrbanFusion, allowing for a fair comparison of the underlying learning frameworks. Where available, we additionally evaluate the models using their original pretrained weights. Due to variations in dataset sizes and spatial coverage, direct comparisons are difficult. Nevertheless, we believe that such comparisons can provide insight into the effects of pretraining conditions and scaling.

We also include several additional models to ensure a comprehensive comparison. CSP (Mai et al., 2023) is the first CLIP-style framework developed specifically for location encoding. PDFM is Google’s Population Dynamics Foundation Model, designed for ZIP code-level tasks within the United States (Agarwal et al., 2024). Finally, we compare our approach against the set of local models from GPS2Vec, which serve as strong local baselines (Yin et al., 2019; 2021), as well as a simple Identity model that uses raw coordinates without any transformation.

Some of these models have been commonly used as baselines in prior work for location representation learning (Klemmer et al., 2025; Agarwal et al., 2024; Vivanco et al., 2023). However, to the best of our knowledge, this is the first systematic evaluation of GeoFMs on urban prediction tasks.

#### F.3.1 GeoCLIP

GeoCLIP (Vivanco et al., 2023) proposes a CLIP-inspired framework for image-to-GPS retrieval, jointly embedding images and geographic coordinates into a shared latent space. The location encoder transforms GPS coordinates into high-dimensional representations using Random Fourier Features (RFF) (Tancik et al., 2020) combined with a hierarchical multi-resolution design, which mitigates spectral bias in MLPs and outperforms traditional discrete region-based classifiers (Vivanco et al., 2023). GeoCLIP achieves state-of-the-art performance on image geolocalization benchmarks, surpassing region-classification baselines across diverse thresholds and performs robustly even in low-data settings. Its GPS encoder also generalizes well to downstream tasks such as coordinate regression and classification (Vivanco et al., 2023; Klemmer et al., 2025; Agarwal et al., 2024).

The model is trained contrastively on 4,7 million geo-tagged images, with a frozen CLIP image encoder and a trainable two-layer projection head. The location encoder is optimized via contrastive loss, aided by a memory bank of sampled GPS embeddings (Vivanco et al., 2023).

We evaluate UrbanFusion against the released GeoCLIP weights and also retrain the model on our PP2-M dataset. Following original training settings from Vivanco et al. (2023), we use a batch size of 512, Adam optimizer (Kingma & Ba, 2015), a learning rate of 3e-5 with step decay, weight decay of 1e-6, and a memory bank with queue size 4096. Training proceeds for 400 epochs with early stopping based on validation loss.

#### F.3.2 Satellite Contrastive Location Image Pretraining (SatCLIP)

Klemmer et al. (2025) introduces a dual-encoder, CLIP-style framework that embeds satellite imagery and GPS coordinates into a shared latent space via contrastive learning, following a similar paradigm to GeoCLIP Vivanco et al. (2023). The satellite image encoder (based on CNN or ViT architectures) processes multispectral Sentinel‑2 tiles, while the coordinate encoder generates continuous GPS embeddings using Spherical Harmonics and SIREN-based MLPs (Rußwurm et al., 2024). In Klemmer et al. (2025), SatCLIP embeddings have been shown to outperform previous coordinate encoders such as GeoCLIP (Vivanco et al., 2023), CSP (Mai et al., 2023), and GPS2Vec (Yin et al., 2019; 2021) across nine geospatial downstream tasks, including temperature prediction, population density estimation, housing prices, median income, biome classification, and animal species recognition. They also demonstrate strong geographic generalization, particularly in underrepresented regions and continents (Klemmer et al., 2025).

The model is pretrained on the newly introduced S2‑100K dataset, which contains 100’000 globally distributed Sentinel‑2 image tiles with associated GPS coordinates. This uniform global sampling mitigates the geographic bias found in prior datasets and provides comprehensive spatial coverage Klemmer et al. (2025).

We compare UrbanFusion against both the original SatCLIP pretrained weights and versions pretrained on our PP2-M dataset. For training, we follow the original setup from Klemmer et al. (2025), using a batch size of 8192, learning rate of 1e-4, weight decay of 0.01, and the AdamW optimizer over 400 epochs (Loshchilov & Hutter, 2019). We use the backbone ViT-S/16 pretrained on Sentinel-2, which is also used in UrbanFusion (Dosovitskiy et al., 2021; Wang et al., 2022). We evaluate both published variants L10 and L40.

#### F.3.3 GAIR

GAIR (Liu et al., 2025) proposes the first multimodal CLIP-style location representation model, unifying remote sensing imagery, street view imagery, and GPS coordinates into a shared embedding space. Although GAIR had not undergone peer review at the time of writing, it demonstrates state-of-the-art performance across multiple geospatial benchmarks.

Its architecture consists of three modality-specific encoders: one each for Sentinel-2 satellite tiles, street view images, and GPS coordinates. A key innovation is the Implicit Neural Representation (INR) module, which enables continuous spatial embeddings by interpolating within the satellite image representations at the precise location of the corresponding street view image. These modality-specific embeddings are then aligned through pairwise contrastive learning. GAIR is pretrained on a globally sampled dataset of 1 million paired Sentinel-2 and street view images (Liu et al., 2025).

The official code and pretrained weights were not publicly available at the time of writing. We reimplemented the method following the paper’s description, with minor modifications. Specifically, we use the same encoders for street view and satellite imagery as in UrbanFusion, GeoCLIP (Vivanco et al., 2023), and SatCLIP (Klemmer et al., 2025) to facilitate consistent comparison. While this omits one of GAIR’s core contribution, the INR module, we note that UrbanFusion could similarly support INR, making the comparison still informative.

We train the model on PP2-M using the hyperparameters reported in the original paper (Liu et al., 2025): a batch size of 256, a base learning rate of $1.5\times 10^{-6}$ with a warm-up over the first 5% of training epochs. We use the AdamW optimizer with $\beta_{1}=0.9$, $\beta_{2}=0.999$, a weight decay of 0.01, and a memory bank with queue size 4096.

#### F.3.4 GPS2Vec

GPS2Vec (Yin et al., 2019) introduces a two-level, grid-based approach for encoding global GPS coordinates. The Earth is first partitioned into UTM zones, with a lightweight neural network trained per zone. Each model learns to predict semantic tags directly from GPS coordinates using supervision from one million geo-tagged Flickr images, selecting the 2’000 most frequent tags as the target vocabulary. This method achieved state-of-the-art performance in geo-tagged image classification. The approach was later extended to GPS2Vec+ (Yin et al., 2021), which incorporates additionally visual features extracted from RGB images. This enhanced version, trained on six million Flickr images with tags, further improved performance over the original GPS2Vec.

Although GPS2Vec is not a global GeoFM, but rather a collection of local models, we include it in our benchmark as a strong local baseline. Following previous convention in Klemmer et al. (2025), we refer to the original version as GPS2Vec (tag) and the multimodal version as GPS2Vec (visual).

#### F.3.5 Contrastive Spatial Pre-Training (CSP)

CSP (Mai et al., 2023) introduces the first global‑scale CLIP‑style location encoding framework that aligns raw GPS coordinates with ground‑level (iNaturalist) (Van Horn et al., 2018) or satellite images (FMoW) (Christie et al., 2018) through contrastive learning. It pioneers the treatment of “location” as a distinct modality in a multimodal embedding space, learned by matching coordinate encodings via positional encodings and neural networks to corresponding visual features extracted by a CNN or ViT. CSP extends traditional CLIP objectives with spatially aware sampling strategies, such as random location negatives and SimCSE‑inspired views (Gao et al., 2021). CSP achieves strong performance across tasks like image geolocation, geo-aware image classification, and spatial retrieval (Mai et al., 2023).

Due to the two versions of the model trained on iNaturalist and FMoW, we only include results from the coordinate encoder and do not utilize the image encoder. Previous research indicated that CSP performed worse than other baselines (Klemmer et al., 2025), and the modality encoders are not compatible with our set of geolocated modalities. In particular, street view imagery does not typically contain animal photographs as in iNaturalist, and the spectral channels in the FMoW dataset differ from those used by UrbanFusion and SatCLIP, making direct integration with our framework infeasible.

#### F.3.6 Population Dynamics Foundation Model (PDFM)

PDFM (Agarwal et al., 2024) is a novel geospatial foundation model developed by Google Research that learns location embeddings by integrating diverse data modalities through a graph neural network (GNN). It constructs a geo-indexed dataset covering U.S. ZIP codes and counties, aggregating information such as busyness, map features, search trends, weather, and air quality. A GNN captures spatial relationships across these modalities, yielding fixed-size embeddings for each location that can support a wide variety of downstream prediction tasks.

Its performance surpasses that of GeoCLIP and SatCLIP, achieving state-of-the-art results on ZIP code tasks. Additionally, when combined with time series forecasting models, PDFM enhances predictions for variables such as unemployment and poverty rates, outperforming fully supervised baselines (Agarwal et al., 2024).

Neither the code, training data, nor model weights were publicly available at the time of writing. However, we were able to obtain the model’s predicted representations for ZIP codes within the United States. Because the model learns a lower-dimensional embedding for each county and ZIP code, these representations are inherently in-sample, in contrast to other baselines where evaluation locations are consistently out-of-sample.

#### F.3.7 Identity

As a basic sanity check, instead of first constructing a spatial representation to be used as input for a downstream task, we directly use the raw geographical coordinates $\mathbf{c}=[\text{lat},\text{lon}]$ as input to the predictive model $g$. The problem is thus formulated as $y\sim g(\mathbf{c})$, where $y$ is the target variable of interest, and $g$ denotes the predictive model (e.g., a linear model or a MLP).

## Appendix G Evaluation Protocols for Downstream Task Performance

### G.1 Evaluation Metrics

In this section, we outline the metrics used to evaluate model performance on downstream tasks. These metrics allow us to compare the effectiveness of different representations across both classification and regression settings.

#### G.1.1 R 2 R^{2} Score (Coefficient of Determination)

The $R^{2}$ score, or coefficient of determination, is a commonly used metric to evaluate the performance of regression models. It measures the proportion of variance in the dependent variable that is predictable from the independent variables. The score is given by:

$$ $R^{2}=1-\frac{\sum_{i=1}^{n}(y_{i}-\hat{y}_{i})^{2}}{\sum_{i=1}^{n}(y_{i}-\bar{y})^{2}},$ (11) $$

where $y_{i}$ is the true value, $\hat{y}_{i}$ is the predicted value, $\bar{y}$ is the mean of the true values, and $n$ is the number of data points.

An $R^{2}$ score of 1 indicates perfect prediction, while a score of 0 implies that the model does no better than simply predicting the mean of the target values. Negative values indicate that the model performs worse than the mean predictor.

#### G.1.2 Weighted F1-Score

The weighted F1-score is a metric used to evaluate classification performance by averaging the F1-scores of all classes, weighting each class by its support (the number of true instances for that class). This approach accounts for class imbalance by giving more influence to classes with more samples. It is defined as:

$$ $F1_{\text{weighted}}=\frac{\sum_{j=1}^{C}n_{j}\cdot F1_{j}}{\sum_{j=1}^{C}n_{j}},$ (12) $$

where $C$ is the total number of classes, $n_{j}$ is the number of true instances of class $j$, and $F1_{j}$ is the F1-score for class $j$, given by

$$ $F1_{j}=\frac{2p_{j}r_{j}}{p_{j}+r_{j}}.$ $$

Here, $p_{j}$ and $r_{j}$ denote the Precision and Recall for class $j$, respectively:

$$ $r_{j}=\frac{TP_{j}}{TP_{j}+FN_{j}},\quad p_{j}=\frac{TP_{j}}{TP_{j}+FP_{j}},$ $$

with $TP_{j}$, $FP_{j}$, and $FN_{j}$ representing the number of True Positives, False Positives, and False Negatives for class $j$.

### G.2 Loss Functions

We also describe the loss functions used during downstream model training. These losses guide the optimization of classifiers or regressors applied on top of the pretrained representations.

#### G.2.1 Mean Squared Error (MSE)

The mean squared error (MSE) is a standard metric for training regression models. It calculates the average of the squared differences between predicted and true values, penalizing larger errors more heavily. MSE is defined as:

$$ $\text{MSE}=\frac{1}{n}\sum_{i=1}^{n}(y_{i}-\hat{y}_{i})^{2},$ (13) $$

where $y_{i}$ is the true value, $\hat{y}_{i}$ is the predicted value, and $n$ is the number of data points. Lower values indicate better performance.

#### G.2.2 Cross-Entropy

Cross-entropy is a commonly used loss for classification tasks, measuring the dissimilarity between the predicted probability distribution and the true distribution. For a dataset of $n$ samples with one-hot encoded target vectors $y_{i}$ and predicted probabilities $p_{i}$, the mean cross-entropy is defined as:

$$ $\mathcal{L}=-\frac{1}{n}\sum_{i=1}^{n}\sum_{k=1}^{K}y_{i,k}\log(p_{i,k}),$ (14) $$

where $K$ is the number of classes, $y_{i,k}$ is 1 if sample $i$ belongs to class $k$ and 0 otherwise, and $p_{i,k}$ is the predicted probability for class $k$ in sample $i$.

### G.3 Downstream Evaluation Procedure

For evaluation on downstream tasks, we assess exclusively out-of-sample locations. Specifically, for training downstream models we use only locations and input modalities that were not used for pretraining the foundation models, in order to analyze the generalization capability of the methods. The only exception is PDFM, since all published representations are compressed representations at the postal code level and correspond to in-sample data from the training phase.

For each downstream task, we split the available locations into training (60 percent), validation (20 percent), and test (20 percent) sets. For classification tasks, the splits are stratified to preserve the label distribution across sets. Hyperparameter tuning is performed using the Optuna framework (Akiba et al., 2019). For each method, we conduct 20 trials optimizing one of the following:

- •
The alpha regularization parameter of scikit-learn’s ridge regression (Pedregosa et al., 2011), sampled logarithmically from the interval $[10^{-4},10^{4}]$.
- •
The C regularization parameter of scikit-learn’s logistic regression (Pedregosa et al., 2011), also logarithmically within $[10^{-4},10^{4}]$.
- •
For MLPs implemented in PyTorch (Paszke et al., 2019), we tune the learning rate sampled logarithmically between $[10^{-5},10^{-1}]$ and weight decay similarly between $[10^{-6},10^{-1}]$.

The MLP architecture consists of two hidden layers with 512 and 256 units respectively. Models are trained for 40 epochs using the AdamW optimizer with a batch size of 64 (Loshchilov & Hutter, 2019). Early stopping is applied with a patience of 10 epochs, based on validation performance.

All representations are concatenated with raw coordinates, following common practice in prior work, and normalized to have zero mean and unit variance (Klemmer et al., 2025). Hyperparameter tuning, early stopping, and modality selection are all performed based on the validation score. We use mean squared error (Equation [13](https://arxiv.org/html/2510.13774v1#A7.E13)) for regression tasks and cross-entropy (Equation [14](https://arxiv.org/html/2510.13774v1#A7.E14)) for classification tasks. After selecting the best hyperparameters based on validation, we train five MLP models with different random seeds and report both the mean and standard deviation of their performance.

A specialized evaluation procedure is applied to ZIP code level tasks. The PDFM model is specifically designed for postal code scale, whereas other methods produce representations at varying or multi-scale resolutions. In the PP2-M dataset, multiple locations may exist within a single postal code. To prevent overrepresentation of specific ZIP codes and to ensure consistency, we use a grouped evaluation protocol: during evaluation, predictions for all locations within the same ZIP code are averaged before computing metrics.

## Appendix H Statement on the Use of Generative AI and Declaration of Originality

In the preparation of this paper, generative AI (ChatGPT version 4o) was utilized for language corrections, including grammar and style improvements. The use of AI was limited to improving readability; it was not used to generate original content, conduct research, or contribute to the intellectual development of the work.

## Appendix I Additional Tables

This section presents additional results, including the baselines CSP and SatCLIPL10, as well as detailed results for MLP models, and results on the Urban Perception and ZIP Code tasks.

Figure: Table 10: Evaluation of Coordinate-Only Spatial Encoding. Best results are shown in bold, the second-best are underlined, and top scores across all models trained on the same dataset (PP2-M) are indicated in [brackets]. MLP results include standard deviations computed over 5 random seeds.

Figure: Table 11: Evaluation of Coordinate-Only Spatial Encoding for the Place Pulse 2.0 Urban Perception tasks. Best results are shown in bold, the second-best are underlined, and top scores across all models trained on the same dataset (PP2-M) are indicated in [brackets]. MLP results include standard deviations computed over 5 random seeds.

Figure: Table 12: Evaluation of Coordinate-Only Spatial Encoding for various ZIP Code tasks. Best results are shown in bold, the second-best are underlined, and top scores across all models trained on the same dataset (PP2-M) are indicated in [brackets].

Figure: Table 13: Evaluation of Coordinate-Only Spatial Encoding for various ZIP Code tasks. Best results are shown in bold, the second-best are underlined, and top scores across all models trained on the same dataset (PP2-M) are indicated in [brackets]. MLP results include standard deviations computed over 5 random seeds.

Figure: Table 14: Evaluation of Multimodal Spatial Encoding. Best results are shown in bold, the second-best are underlined, and top scores across all models trained on the same dataset (PP2-M) are indicated in [brackets]. MLP results include standard deviations computed over 5 random seeds. The full names of all modality abbreviations are provided in Appendix [A](https://arxiv.org/html/2510.13774v1#A1).

Figure: Table 15: Evaluation of Multimodal Spatial Encoding for the Place Pulse 2.0 Urban Perception tasks. Best results are shown in bold, the second-best are underlined, and top scores across all models trained on the same dataset (PP2-M) are indicated in [brackets]. MLP results include standard deviations computed over 5 random seeds. The full names of all modality abbreviations are provided in Appendix [A](https://arxiv.org/html/2510.13774v1#A1).

Figure: Table 16: Evaluation of Multimodal Spatial Encoding for various ZIP Code tasks. Best results are shown in bold, the second-best are underlined, and top scores across all models trained on the same dataset (PP2-M) are indicated in [brackets].

Figure: Table 17: Evaluation of Multimodal Spatial Encoding for various ZIP Code tasks. Best results are shown in bold, the second-best are underlined, and top scores across all models trained on the same dataset (PP2-M) are indicated in [brackets]. MLP results include standard deviations computed over 5 random seeds.

Figure: Table 18: Selected modalities of Multimodal Spatial Encoding for various ZIP Code tasks. The full names of all modality abbreviations are provided in Appendix [A](https://arxiv.org/html/2510.13774v1#A1).

Table: Table 19: Cross-Regional Generalization using all available modalities as inputs. Best results are in bold, the second-best are underlined. MLP results include standard deviations across 5 random seeds. The full names of all modality abbreviations are provided in Appendix [A](https://arxiv.org/html/2510.13774v1#A1).

Table: Table 20: Cross-Regional Generalization using all available modalities as inputs for the Place Pulse 2.0 Urban Perception tasks. Best results are in bold, the second-best are underlined. MLP results include standard deviations across 5 random seeds. The full names of all modality abbreviations are provided in Appendix [A](https://arxiv.org/html/2510.13774v1#A1).

Table: Table 21: Cross-Regional Generalization using all available modalities as inputs for ZIP Code tasks. Best results are in bold, the second-best are underlined.

Table: Table 22: Cross-Regional Generalization using all available modalities as inputs for ZIP Code tasks. Best results are in bold, the second-best are underlined. MLP results include standard deviations across 5 random seeds.

Table: Table 23: Selected modalities of Cross-Regional Generalization for various ZIP Code tasks. The full names of all modality abbreviations are provided in Appendix [A](https://arxiv.org/html/2510.13774v1#A1).

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: •  Could you report cross-domain results without concatenating raw coordinates, and analyze the model's sensitivity to coordinate normalization or projection?
•  Could you include ablations on the masking ratio/strategy, pooling mechanisms, and number of fusion layers (single vs. multi-layer)?
•  The paper positions multimodal fusion as a core contribution, yet most baselines are single-modal. This makes it difficult to disentangle the effect of the fusion mechanism from the benefit of using multiple modalities. While the "concatenated multimodal embeddings + linear probing" baseline is informative, it is not equivalent to native multimodal fusion. Please consider adding one or two native multimodal fusion baselines for a fairer comparison.
soundness: 2
strengths: •  Innovative training paradigm: The proposed Stochastic Multimodal Fusion (SMF) framework provides a practically meaningful contribution by integrating stochastic modality masking, symmetric InfoNCE alignment, and latent modality reconstruction into a unified training process.
•  Comprehensive evaluation: The paper conducts extensive and well-structured experiments across multiple tasks and scenarios, including coordinate-only, multimodal, and cross-region zero-shot generalization settings.
•  Reproducibility and practicality: The framework is conceptually simple, reproducible, and demonstrates clear potential for deployment in real-world multimodal GeoAI applications.
summary: This paper presents UrbanFusion, a geofoundation model that learns unified spatial embeddings through Stochastic Multimodal Fusion (SMF). The model encodes heterogeneous urban data — including street-view images, remote sensing, maps, and POIs — using modality-specific encoders, and fuses them via a Transformer-based architecture. It is trained with a contrastive alignment objective between complementary modality subsets, together with latent reconstruction to address missing-modality scenarios. Evaluated across 41 tasks in 56 cities, UrbanFusion achieves state-of-the-art performance and strong cross-region generalization, demonstrating a practical and scalable paradigm for multimodal GeoAI.
weaknesses: •  Limited architectural novelty: While the training objective is interesting, the overall fusion design (“multimodal tokens + a single-layer Transformer + average pooling”) closely resembles standard multimodal architectures. Consequently, the primary novelty lies in the learning strategy rather than in architectural design.
•  Insufficient ablation analysis: The paper lacks systematic ablations on key factors such as masking strategy and ratio, the impact of fusion depth (single- vs. multi-layer), and pooling mechanisms. Including these analyses would strengthen the methodological insight and clarify design choices.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: - How do they define the locations? How much engineering is required to get UrbanFusion to work in a new city? What if the spatial resolution of the data in the new city does not match?
- What is the impact of the image encoder? DinoV3 (Siméoni et al. 2025) was pretrained extensively on satellite imagery; perhaps this would yield a meaningful performance increase? But this likely would not be a fair comparison to GeoCLIP. Could be worth studying in the future
- Why does the multimodal fusion encoder use average pooling instead of the CLS token?
soundness: 3
strengths: - UrbanFusion is flexible wrt what modalities are available inference time. Even though it is outperformed by some baselines, this flexibility is a significant benefit to the community
- Strong (though not dominant) empirical results
- UrbanFusion can effectively generalize to new cities not seen in training
- Effectively integrates cross-modal information, while preserving intra-model information
summary: ### Problem:
- The authors aim to build better geospatial foundation models that can understand and predict geospatial dynamics and urban phenomena at scale
- A good foundation model should:
    - Be task-agnostic
    - Be multi-modal
    - Synthesize information across modalities
    - Be robust to missing modalities
    - Have good generalization to unseen geographic regions


### Solution

- Authors propose a novel foundation model called UrbanFusion
    - UrbanFusion processes several different modalities (street-view, satellite imagery, maps, POIs, and coordinates)
    - Each modality is encoded separately, then fused with a Transformer
    - UrbanFusion is trained via a novel objective called Stochastic Multimodal Fusion (SMF), which combines contrastive learning with self-supervised re-construction
        -  Modalities are randomly masked
        - The model is trained to align representations contrastively across different modality subsets
        - Reconstruct missing modality embeddings from the fused representation
weaknesses: - Empirical performance is not dominant. On tasks/datasets, UrbanFusion is outperformed by baselines
- Some overlap with prior work. Likely worth citing some of these:
    - Jenkins et al. CIKM’19. Unsupervised Representation Learning of Spatial Data via Multimodal Embedding
    - Yan et al. 2024. UrbanCLIP: Learning Text-enhanced Urban Region Profiling with Contrastive Language-Image Pretraining from the Web
- No estimates of variability (e.g., mean + std across several training runs) in experiments.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 3
questions: Could the authors address the issues mentioned in the weaknesses section? Specifically, are raw geographical coordinates appended to the embeddings for all baselines, or only for your model? Can the authors provide results on some of the tasks without appending the raw coordinates?
soundness: 2
strengths: [S1] The work addresses a timely problem of learning unified location embeddings for heterogeneous spatial modalities.

[S2] The proposed stochastic multimodal fusion module provides a flexible approach for aligning and reconstructing random subsets of modalities.

[S3] The paper is well written and the methodology is easy to follow.

[S4] The empirical results show improvements over the selected baselines.
summary: The paper proposes UrbanFusion, a multimodal framework that fuses several inputs (street view images, satellite images, POI data and map data) using a transformer based module. One of the main contributions of the paper is the proposed stochastic multimodal fusion training strategy, which applies a combined contrastive and reconstruction loss on random subsets of modalities to learn a unified representation. The approach is evaluated on 41 downstream urban prediction tasks.
weaknesses: [W1] One of my main **concerns** and confusion is the decision to **append the raw coordinates** to the learned embeddings during downstream evaluation: *"Following prior work [1], raw geographical coordinates are concatenated to the model embeddings for evaluation."* Location is highly correlated with many of the downstream labels (e.g. house prices), so it is unclear if the performance gains come from the approach or from leveraging location information. Moreover, the cited reference [1] is misinterpreted: In SATCLIP, location embedding (not raw coordinates) are concatenated with image features to perform image classification.

[W2] The fusion of different urban modalities has been studied extensively, and more sophisticated fusion approaches exist in the literature (e.g., [2, 3]). For the specific modalities handled here, the authors seem to bypass the core challenge of different spatial resolutions by operating at the patch level [4]. Moreover, the handling of the cartographic basemap is fairly naïve, as it relies on rasterization, rather than leveraging spatial and topological structure, which has been addressed more effectively in prior work [5, 6]. Given that, the SMF component seem to be the main technical contribution here. However, its evaluation is weak: it is validated on a toy example, and the real-world ablation shows that the combination of the reconstruction loss and contrastive loss is not the best overall in the majority of the cases (Appendix C.3.).

[W3] The foundation model claim is an overstatement given the small-scale pretraining dataset and the lack of diversity across tasks. Specifically, the evaluated tasks are variations of point-based prediction problems (regression or classification) confined to the urban domain, which does not demonstrate the broad, cross-domain applicability or diverse reasoning capabilities expected of a true foundation model.

[W4] There are instances where the authors categorize SatCLIP and GeoCLIP as unimodal, although both are inherently multimodal: *"Both models clearly outperform unimodal baselines such as SatCLIP and GeoCLIP, demonstrating the advantages of
multimodal representation learning for geographic generalization"*.

[W5] Similar to W1, several "new" features are introduced when testing on housing value prediction (i.e., # of bedrooms, # of bathrooms etc.), which are already strong predictors of the housing prices on their own. Again, it's unclear whether the improvement comes from the learned embeddings.

[1] Satclip: Global, general-purpose location embeddings with satellite imagery. AAAI 2025.

[2] Urbanclip: Learning text-enhanced urban region profiling with contrastive language-image pretraining from the web. WWW 2024.

[3] Refound: Crafting a foundation model for urban region understanding upon language and visual foundations. KDD 2024.

[4] Gair: Improving multimodal geo-foundation model with geo-aligned implicit representations. arxiv 2025.

[5] Poly2Vec: Polymorphic Fourier-Based Encoding of Geospatial Objects for GeoAI Applications. ICML 2025.

[6] Geo2Vec: Shape-and Distance-Aware Neural Representation of Geospatial Entities. arxiv 2025.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 2
questions: See weakness
Further details of section 4.5 would be useful (not easy to follow)
soundness: 2
strengths: A pertinent challenge for spatial data and spatial foundation model, especially the fusion of data of different modalities.
The tasks are comprehensive, including many downstream tasks and 56 cities.
The paper is generally well-written and clearly structured.
summary: This paper presents UrbanFusion which is a multimodal geospatial representation learning framework designed to integrate multiple data sources, e.g., remote sensing imagery, street-view imagery, POI and OSM, and location information (lat – lon). The paper uses a self-supervised learning strategy, termed SMF, which combines contrastive learning for aligning representations from different modality subsets with a modality reconstruction objective for fusion.
weaknesses: Since the features from different modalities are extracted from frozen backbone encoders, the the primary contribution seems to be on fusion, which refers to specific combination strategy SMF involving contrastive learning and reconstruction with modality masking. This combination is applied effectively to spatial data, but the underlying principle of combining these two types of self-supervised objectives is not entirely new in the broader representation learning literature. It is noteworthy that empirical evidence is ample, while I doubt this is the focus of ICLR.
To the ICLR community, the novel insight given specifically by the paper (if we talk about those beyond showing that combining these existing techniques works well for this multimodal geospatial setting) is limited. This means that it seems to be more of empirical finding paper. 
There are some minor points as well. Using image encoder for OSM naturally leads to information loss and there are papers using object-level information for feature extraction. An example is 
Bai, Lubian, et al. "GeoLink: Empowering Remote Sensing Foundation Model with OpenStreetMap Data." arXiv preprint arXiv:2509.26016 (2025).
The rationality of the baseline comparisons is questionable. The pretrained backbone of the compared models differs (e.g., some are based on CLIP, others on GeoCLIP), which undermines the fairness of the comparison. Would it be more appropriate to compare with multimodal remote sensing foundation models that integrate OSM data? It would be also useful to compare a baseline with all of your frozen features concatenated, since the main contribution is on fusion.


Decision: Reject