GeoBS: Information-Theoretic Quantification of Geographic Bias in AI Models

## Abstract

Abstract The widespread adoption of AI models, especially foundation models (FMs), has made a profound impact on numerous domains. However, it also raises significant ethical concerns, including bias issues.
Although numerous efforts have been made to quantify and mitigate social bias in AI models, geographic bias (in short, geo-bias) receives much less attention, which presents unique challenges.
While previous work has explored ways to quantify geo-bias, these measures are model-specific (e.g., mean absolute deviation of LLM ratings) or spatially implicit (e.g., average fairness scores of all spatial partitions).
We lack a model-agnostic, universally applicable, and spatially explicit geo-bias evaluation framework that allows researchers to fairly compare the geo-bias of different AI models and to understand what spatial factors contribute to the geo-bias.
In this paper, we establish an information-theoretic framework for geo-bias evaluation , called GeoBS ( Geo - B ias S cores). We demonstrate the generalizability of the proposed framework by showing how to interpret and analyze existing geo-bias measures under this framework. Then, we propose three novel geo-bias scores that explicitly take intricate spatial factors (multi-scalability, distance decay, and anisotropy) into consideration.
Finally, we conduct extensive experiments on 3 tasks, 8 datasets, and 8 models to demonstrate that both task-specific GeoAI models and general-purpose foundation models may suffer from various types of geo-bias.
This framework will not only advance the technical understanding of geographic bias but will also establish a foundation for integrating spatial fairness into the design, deployment, and evaluation of AI systems.

## 1 Introduction

Recent years have witnessed a major paradigm shift in the Artificial Intelligence (AI) domain from task-specific models to foundation models (FMs) (Bommasani et al., 2021).
However, the widespread adoption of FMs also raises significant ethical concerns. A major challenge is
bias (Gordon & Desjardins, 1995; Gianfrancesco et al., 2018), which refers to systematic disparities or tendencies in AI models that lead to unfair or prejudiced outcomes in terms of gender, race, religion, nationality, etc.
AI model bias is a well-documented issue that can lead to serious ethical consequences, with notable examples such as Google Photo App’s racist blunder of misclassifying African American people as gorillas (News, 2015). Due to the large training datasets and model sizes, FMs are more prone to inheriting and amplifying these biases (Zhang et al., 2022; Touvron et al., 2023), thus having the potential to cause a huge negative impact on society (Kamiran & Calders, 2012; Dhamala et al., 2021; Hardt et al., 2016; Parrish et al., 2021; Gallegos et al., 2024).
Extensive efforts have been made for social biases evaluation and mitigation in terms of gender, religion, race, color, sexual orientation, etc, in AI models, including FMs, by developing bias evaluation frameworks, benchmark datasets (Nangia et al., 2020; Nadeem et al., 2021; Sheng et al., 2019; Dhamala et al., 2021; Gallegos et al., 2024), and debiasing methods (Yang et al., 2023; Xie et al., 2022).
However, few efforts have been devoted to quantifying and addressing geographic bias of AI and FMs, which demonstrate unique characteristics and require novel bias quantification methods and debiasing approaches (Liu et al., 2022; Faisal & Anastasopoulos, 2023; Manvi et al., 2024).

Figure: ((a))
Refer to caption: x1.png

Most previous work interprets it as a subclass of model bias – a phenomenon that an AI model performs systematically differently across geographic regions beyond reasonable random fluctuations (Liu et al., 2022; Xie et al., 2022; Wu et al., 2024).
Despite the recent advancement in geo-bias research, we identify two major research gaps:

- 1.
Existing geo-bias metrics are often developed ad-hoc for specific models/tasks, and lack a systematic framework. For example, Manvi et al. (2024) proposed a Spearman’s $\rho$ bias score, which is only applicable to certain LLM zero-shot prompting tasks.
- 2.
Existing geo-bias metrics are implicit
and lack a direct connection to their spatial implications, i.e., what spatial factors (distance, direction, scale, etc.) stand behind the observed bias. For example, Xie et al. (2022) proposed to randomly partition the space and use the model’s average performance difference over all possible partitioning as the geo-bias metric, which is hard to interpret because it mixes different types of geo-bias.

In this paper, we propose to bridge the aforementioned gaps with the help of
classic spatial statistics and information theory by establishing an information-theoretic framework for geo-bias evaluation, called GeoBS. From a perspective of spatial point pattern analysis, a set of geolocations associated with corresponding model performance metrics forms a distribution on the Earth’s surface, which can be treated as spatial point patterns (SPP) (Illian et al., 2008; Boots & Getis, 2020). The properties (e.g., Gaussian v.s. Poisson) and strengths (e.g., 10% Gaussian v.s. 90% Gaussian) of such patterns can help us categorize existing geo-bias metrics and design novel ones. For instance, if we define a Gaussian distribution as the “homogeneous” reference pattern that exhibits no geo-bias, then a statistical distance such as KL-divergence or Wasserstein distance effectively measures how far the observed SPP deviates from the predefined homogeneity, which may serve as a valid geo-bias metric.

To concretely demonstrate the power and generalizability of our framework, we first prove that two recently proposed and widely adopted geo-bias scores, Unmarked SSI and Marked SSI (Wu et al., 2024), can be clearly interpreted and organically integrated into our geo-bias framework. Then we propose three novel geo-bias scores under the guidance of this framework, which are able to differentiate and quantify geo-bias related to different spatial factors such as multi-scalability, distance decay, and anisotropy. Specifically, the Scale-Grid Spatial Relative-Entropy (SRE) score (Figure [1(c)](https://arxiv.org/html/2509.23482v1#S1.F1.sf3)) considers the multi-scale heterogeneity, i.e., at what spatial scales the low-performance points concentrate. The Distance-Lag SRE (Figure [1(d)](https://arxiv.org/html/2509.23482v1#S1.F1.sf4)) considers the distance-decay effect, i.e., whether the model performance changes as distance increases. The Direction-Sector SRE (Figure [1(e)](https://arxiv.org/html/2509.23482v1#S1.F1.sf5)) considers directional heterogeneity/anisotropy, i.e., whether the model performs differently in different directions. These scores allow us to locate the intricate spatial factors behind the observed geo-bias and better target potential solutions.
All five geo-bias scores are conceptualized in Figure [1](https://arxiv.org/html/2509.23482v1#S1.F1).
In summary, the major contributions of this paper are:

- 1.
We propose a theoretical framework to evaluate geo-bias from the perspective of information theory, which allows us to systematically categorize and interpret geo-bias.
- 2.
We draw connections between spatial point pattern analysis with information theory, which allows us to design model-agnostic and spatially explicit geo-bias scores.
- 3.
We demonstrate that our theoretical framework can successfully interpret existing geo-bias scores (e.g., Unmarked SSI and Marked SSI) and propose three novel geo-bias scores (Scale-Grid SRE, Distance-Lag SRE, Direction-Sector SRE) that can explicitly capture the intricate spatial factors behind observed geo-bias.
- 4.
We extensively evaluate the geo-bias of both task-specific GeoAI models and task-agnostic foundation models. It is shown that both model groups demonstrate substantial geo-bias, and it is important to use the spatially explicit geo-bias scores to interpret their underlying spatial factors behind geo-bias because many models show geo-bias of mixed types.
- 5.
We implement a plug-and-play Python package called GeoBS for efficiently computing the five geo-bias scores. It will facilitate researchers to promptly check and report the geo-bias of their models, promoting spatial fairness in the community.

## 2 Related Work

AI models, including task-specific GeoAI and general-purpose foundation models, often perform differently across various spatial contexts (Xie et al., 2022; Manvi et al., 2024; Faisal & Anastasopoulos, 2023; Wu et al., 2024).
Such bias may lead to or even exacerbate inequities in resource allocation, social disparities, and vulnerabilities in resilience and sustainability
(Xie et al., 2022)
raising ethical concerns (Nelson et al., 2022).
The objective of geo-bias metrics is to quantify geospatial bias, which are inherently linked to geographic locations or spatial distributions of data samples (Hay, 1995; Xie et al., 2022).

There has been extensive research on improving fairness in AI using pre-processing (Jo & Gebru, 2020; Steed & Caliskan, 2021), in-processing (Kamishima et al., 2011; Serna et al., 2020), and post-processing techniques (Binns, 2018; Caton & Haas, 2024). Most fairness quantification methods focus on categorical-attribute-based biases, i.e., ethnicity, and age (Caton & Haas, 2024). However, geographical bias is in continuous 2D or 3D space, and those methods often fail to account for the intrinsic spatial characteristics of data, such as directional dependence and scale effects.

## 3 Problem Setup

We first give some formal definitions and mathematical notations that will be used throughout the paper in Section [3.1](https://arxiv.org/html/2509.23482v1#S3.SS1). Since we will refer to many concepts from classic spatial point pattern analysis,
we provide a brief introduction of these concepts in Section [3.2](https://arxiv.org/html/2509.23482v1#S3.SS2) for the broader AI community.

### 3.1 Notations and Definitions

###### Definition 3.1 (Geospatial Dataset) .

A geospatial dataset $\mathcal{D}:=\{(X_{i},L_{i},y_{i})|L_{i}\in\mathbb{S}^{2}\}_{i=1}^{n}$ is a set of triples: $X_{i}$ is an observation, for example a streetview image; $L_{i}$ is the geographical location of $X_{i}$ on the Earth surface $\mathbb{S}^{2}$, or sometimes approximated by the Euclidean plane $\mathbb{R}^{2}$; $y_{i}$ is the task-specific ground-truth for $X_{i}$, e.g., class labels in classification tasks and real values in regression tasks.

###### Definition 3.2 (Model & Predictions) .

A model $\mathcal{F}$ maps $X_{i}$ and $L_{i}$ to a prediction $\hat{y}_{i}$ of the ground-truth, that is, $\hat{y}_{i}:=\mathcal{F}(X_{i},L_{i})$.

###### Definition 3.3 (Performance Function) .

A performance function $\bm{\pi}$ compares the ground-truth $y_{i}$ and the prediction $\hat{y}_{i}$ to assign an evaluation $\pi_{i}:=\bm{\pi}(y_{i},\hat{y}_{i})$ to a location $L_{i}$.
$\bm{\pi}$ can
be non-numerical values. For example, in classification tasks, $\pi_{i}$ can be binary (correct v.s. incorrect), continuous (logits of predictions), or categorical (human comments).

###### Definition 3.4 (Location Map & Performance Map) .

A location map is a set of locations $\mathcal{L}_{\mathcal{D}}:=\{L_{i}\}_{i=1}^{n}$. A performance map is a set of location-evaluation tuples $\mathcal{M}_{\mathcal{D},\bm{\pi}}:=\{(L_{i},\pi_{i})\}_{i=1}^{n}$.

Spatial point patterns always involve multiple locations (a single point will not form “patterns”), so we define the unit to evaluate geo-bias as.

###### Definition 3.5 (Region Of Interest (ROI)) .

A region of interest (ROI) $N\in\mathcal{P}(\mathcal{M}_{\mathcal{D},\bm{\pi}})$ is a multiple-point subset of the performance map, where $\mathcal{P}(\mathcal{M}_{\mathcal{D},\bm{\pi}})$ denotes the power set of $\mathcal{M}_{\mathcal{D},\bm{\pi}}$.

###### Definition 3.6 (Local Geo-Bias Score) .

A function $\gamma:\mathcal{P}(\mathcal{M}_{\mathcal{D},\bm{\pi}})\rightarrow\mathbb{R}$ measures the strength of geo-bias in an ROI $N$.
We call $\gamma(N)$ a local geo-bias score.

###### Definition 3.7 (Global Geo-Bias Score) .

Let $\mathcal{N}:=\{N_{m}\in\mathcal{P}(\mathcal{M}_{\mathcal{D},\bm{\pi}})\}_{m=1}^{M}$ be the set of all ROIs where we intend to measure geo-bias.
We compute $\gamma(N_{m})$ for each $N_{m}\in\mathcal{N}$ and use their weighted sum as the global geo-bias score.

###### Definition 3.1 (Geospatial Dataset) .

###### Definition 3.2 (Model & Predictions) .

###### Definition 3.3 (Performance Function) .

###### Definition 3.4 (Location Map & Performance Map) .

###### Definition 3.5 (Region Of Interest (ROI)) .

###### Definition 3.6 (Local Geo-Bias Score) .

###### Definition 3.7 (Global Geo-Bias Score) .

### 3.2 Related Concepts for Spatial Point Pattern Analysis

Unmarked & Marked Point Patterns. In spatial point pattern analysis, unmarked patterns only consider the locations of points, while marked patterns consider both the locations and the attribute values of the points. For example, the locations where bat-eared fox is observed is a typical unmarked spatial point pattern, since it only considers the spatial distributions of species occurrences, while no value is attached to each location. On the other hand, a set of geo-tagged soil samples is a marked spatial point pattern in which we consider both the geolocations of these samples and these samples’ soil attribute values (e.g., soil moisture, pH, salinity, etc.). Based on the above definitions, a location map $\mathcal{L}_{\mathcal{D}}:=\{L_{i}\}_{i=1}^{n}$ is an unmarked spatial point pattern while a performance map $\mathcal{M}_{\mathcal{D},\bm{\pi}}:=\{(\pi_{i},L_{i})\}_{i=1}^{n}$ is a marked spatial point pattern.

Summary Statistics of Spatial Point Patterns. In spatial point pattern analysis, various summary statistics are developed to quantify the spatial autocorrelation or spatial heterogeneity of a spatial point pattern.
These statistics can be classified into two categories: first-order and second-order summary statistics.
The first-order summary statistics (O’sullivan, 2003; Ben-Said, 2021) focus on quantifying the variation of the intensity of point patterns (for unmarked point patterns) and the expectation of attribute values (for marked point patterns) across a study area. Examples include nearest neighbor distribution function, spherical contact distribution (Ben-Said, 2021), Moran’s I (Moran, 1950), LISA (Anselin, 1995), Geary’C, and Local Geary’s C (Anselin, 2019).
In contrast, the second-order statistics focus on quantifying the strength of interactions between points according to distance. Examples include Ripley’s K-function (Ripley, 1977) and L-functions (Besag, 1977).

## 4 Methods

In this work, we have two major objectives:
1) Proposing a systematic, theory-supported framework to categorize and interpret geo-bias;
2) Designing spatially explicit geo-bias quantification under this theoretical framework.
For the former objective, we propose a novel framework of geo-bias interpretation and categorization based on spatial point pattern analysis concepts introduced in Section [3.2](https://arxiv.org/html/2509.23482v1#S3.SS2). This framework serves our purposes perfectly because (1) it is compatible with most existing geo-bias metrics, and (2) it clearly points out three key factors we need to consider when designing new geo-bias metrics. For the latter objective, we combine the key factors with information theory to quantify geo-bias. It is because under our theoretical framework, geo-bias is in effect the difference between the observed spatial point patterns in the location/performance maps and predefined spatially homogeneous (i.e., “unbiased”) spatial point patterns, which can be quantified using information-theoretic terms such as self-information and relative entropy.

In the rest of this section, we will firstly introduce our theoretical framework in Section [4.1](https://arxiv.org/html/2509.23482v1#S4.SS1) and use concrete examples to demonstrate how existing geo-bias metrics can be integrated into our framework in Section [4.2](https://arxiv.org/html/2509.23482v1#S4.SS2). Then, we will propose three novel information-theoretic geo-bias metrics based on our framework in Section [4.3](https://arxiv.org/html/2509.23482v1#S4.SS3), also explaining their spatial implications with illustrations. Finally, we will describe the algorithms for computing the aforementioned geo-bias scores in Section [4.4](https://arxiv.org/html/2509.23482v1#S4.SS4).

### 4.1 Theoretical Framework & Categorization of Geo-Bias

As we have discussed in Section [3.2](https://arxiv.org/html/2509.23482v1#S3.SS2), both the location map $\mathcal{L}_{\mathcal{D}}$ and the performance map $\mathcal{M}_{\mathcal{D},\bm{\pi}}$ of a model are spatial point patterns. Intuitively, the more “random” the location/performance map looks, the less geographically biased it is. From the theoretical perspective of spatial point pattern analysis, this intuition can be precisely described as comparing a location/performance map with a predefined, spatially homogeneous reference pattern, and the more different they are
the more geo-biased the model is (against the chosen reference pattern).
The metrics we use to quantify the difference between patterns are naturally valid geo-bias metrics. For example, if we assume that an ideally unbiased model should perform uniformly well across the space with Gaussian fluctuations, we can compute the Kolmogorov–Smirnov (KS) test statistics of the performance map against a Gaussian distribution and use it as a geo-bias metric – that is, the larger the statistics, the less unlikely the model performs uniformly well as we hypothesized, thus more geo-biased under our assumption of homogeneity.

Based on this interpretation, we can summarize three key factors that differentiate one geo-bias metric from another: (1) the map (location map or performance map) we use for comparison, (2) the reference pattern (i.e., the desired unbiased pattern), and (3) the difference measure between the map and the reference pattern. These factors enable a neat categorization of geo-bias: we can categorize a geo-bias metric as (1) “Unmarked” v.s. “Marked” based on which map it uses, (2) “Gaussian”, “Poisson”, “Permutation”, etc., based on which reference pattern it uses, and (3) “Statistical”, “Information-Theoretic”, etc., based on which difference measure it uses.

Another important dimension of geo-bias categorization is “First-Order” v.s. “Second-Order”. As discussed in Section [3.2](https://arxiv.org/html/2509.23482v1#S3.SS2), a geo-bias metric that summarizes the heterogeneity of a spatial point pattern can be either first-order if it is an averaged number over all points, or second-order if it is a function of point interactions such as covariances against distance. Most existing geo-bias metrics (Manvi et al., 2024; Xie et al., 2022; Wu et al., 2024), as well as the novel geo-bias scores we propose in this paper, are first-order. We leave the investigation of second-order geo-bias metrics to the future.

### 4.2 Existing Geo-Bias Metrics: Spatial Self-Information (SSI) Scores

Figure: Figure 2: Workflows of Unmarked SSI and Marked SSI computation.
Refer to caption: figs/ssi_illustration.png

We use two recently proposed but commonly used geo-bias metrics as concrete examples to demonstrate the applicability of our framework discussed in Section [4.1](https://arxiv.org/html/2509.23482v1#S4.SS1).
Unmarked SSI Score. Unmarked SSI Score is proposed in (Wu et al., 2024) as a measure of dataset geo-bias, i.e., whether the data points are uniformly distributed across the space. Figure [2](https://arxiv.org/html/2509.23482v1#S4.F2) illustrates the computation workflow. According to our theoretical framework, this metric is (1) “Unmarked” because the location map is used, (2) “Permutation” because the reference pattern is a random permutation of foreground/background points, and (3) “Information-Theoretic” because the difference measure is the self-information of the location map (implicitly against the reference pattern) Wang et al. (2024).
Marked SSI Score. Marked SSI Score is proposed in (Wu et al., 2024) as a measure of model performance geo-bias, i.e., whether the model performance (accuracy, MSE, etc.) is consistently good across the space. Figure [2](https://arxiv.org/html/2509.23482v1#S4.F2) illustrates the computation workflow. Similarly, this metric is (1) “Marked” because the performance map is used, (2) “Permutation” because the reference pattern is a random permutation of good/bad performance points, and (3) “Information-Theoretic” because the difference measure is the self-information of the performance map (implicitly against the reference pattern).

### 4.3 Novel Geo-Bias Metrics: Spatial Relative-Entropy (SRE) Scores

We have demonstrated the power of our theoretical framework in decomposing a geo-bias metric into three dimensions: the map, the reference pattern, and the difference measure. This decomposition also helps us be more purposeful when designing geo-bias metrics. We notice that for Unmarked SSI and Marked SSI, the difference measure, i.e., self-information (also known as “surprisal” in information theory), effectively accounts for the geo-bias specifically related to spatial proximity.
This is because Wang et al. (2024) proves that the self-information of a spatial point pattern is an alternative quantification of Moran’s I (Moran, 1950), which measures the autocorrelation among spatial neighbors. This finding inspires us: what if we want to measure the geo-bias related to other important spatial factors?

#### 4.3.1 Spatial Motivations of SRE Scores

Figure: Figure 3: Workflows of SRE Scores computation.
Refer to caption: figs/sri_illustration.png

Analogous to the famous Simpson’s Paradox in statistics (Pearl, 2022), spatial heterogeneity makes the conclusions drawn from spatial data sensitive to the way we partition the space (Mai et al., 2025). It is also related to the fundamental Modifiable Unit Area Problem (Openshaw, 1984; Fotheringham & Wong, 1991; Goodchild, 2022; Chen et al., 2022), and recognized as a major source of bias in various domains such as ecology (Jelinski & Wu, 1996; Swift et al., 2008) and urban geography (Deng et al., 2024). For example, if a model is sensitive to directions, it will demonstrate strong directional heterogeneity, i.e., the model performs significantly differently on data points drawn from different directions. Figure [1](https://arxiv.org/html/2509.23482v1#S1.F1) clearly illustrates that the model performance in each disjoint area (e.g. square, ring, sector) differs from the overall performance, indicating that there is geo-bias.

In this paper, we are interested in (but not limited to) three specific partitionings: scale-grid, distance-lag, and direction-sector. We thus design three novel, model-agnostic and spatially explicit geo-bias metrics according to the partitionings called Scale-Grid SRE Score, Distance-Lag SRE Score, and Direction-Sector SRE Score. They correspond to three important spatial factors – multi-scalability, distance decay, and anisotropy, respectively. SRE stands for Spatial Relative-Entropy, because the difference measures we use in these metrics is the Kullback–Leibler (KL) divergence, also known as relative entropy.

#### 4.3.2 Formal Definitions of SRE Scores

###### Definition 4.1 (Partition Function, Partitioning & Patch) .

$\mathbb{PAR}$ is called a partition function if it divides the spatial area $A$ of an ROI $N$ into a set of disjoint sub-areas $\mathcal{A}:=\bigcup A_{k}$ and maps $N$ into disjoint subsets $P_{k}:=\{(L_{i},\pi_{i})|L_{i}\in A_{k}\}$. The set of disjoint subsets $\Pi:=\bigcup P_{k}$ is called a partitioning of $N$, and each subset $P_{k}$ is called a patch in the partitioning.

###### Definition 4.2 (ROI & Patch Performance Distribution) .

Let $h$ be a mapping from a set of location-evaluation tuples to a probability distribution, e.g., the normalized histogram of correct and wrong predictions. $h(N)$ and $h(P_{k})$ are called an ROI performance distribution and a patch performance distribution, respectively.

###### Definition 4.3 (Local SRE Score) .

Let $d$ be a difference measure between distribution $h(P_{k})$ and distribution $h(N)$, e.g. Kullback–Leibler divergence. The Local SRE Score of ROI $N$ is defined as $\gamma_{\text{SRE}}(N):=\#P_{k}/\#N\sum_{k}d(h(P_{k}),h(N))$.

###### Definition 4.4 (Global SRE Score) .

The Global SRE Score is defined as the weighted sum of all Local SRE Scores: $\Gamma_{\text{SRE}}:=\sum_{m}w_{m}\gamma_{\text{SRE}}(N_{m})$. $w_{m}$ is the user-defined weight for ROI $N_{m}$.

The partition functions used in this paper include: (1) Scale-Grid: Partition an ROI into equal-size squares; (2) Distance-Lag: Partition an ROI into equal-width concentric rings. (3) Direction-Sector: Partition an ROI into equal-angle sectors. However, we encourage researchers to design their own partitioning based on their needs and domain knowledge and enlarge the family of SRE Scores.

Figure: Algorithm 1 Local SRE Algorithm

###### Definition 4.1 (Partition Function, Partitioning & Patch) .

###### Definition 4.2 (ROI & Patch Performance Distribution) .

###### Definition 4.3 (Local SRE Score) .

###### Definition 4.4 (Global SRE Score) .

### 4.4 Implementations of SSI and SRE Geo-Bias Scores in GeoBS

Finally, we will discuss the detailed implementation of both SSI Scores and SRE Scores computation in our GeoBSPython package.

For SSI Scores, the formal definitions and theoretical formulas can be found in Wu et al. (2024) and Wang et al. (2024). While the SSI Score algorithm we use (Algorithm [2](https://arxiv.org/html/2509.23482v1#alg2) in Appendix [B.1](https://arxiv.org/html/2509.23482v1#A2.SS1)) remains mostly unaltered, the quality and stability of implementation in our GeoBS package are significantly improved over the original PyGBS package (Wu et al., 2024). The most important changes include: (1) we modularized the SSI Score algorithm so that it shares common data preprocessing and postprocessing procedures with our SRE Scores, which saves up to 20% of computational costs since these procedures only need to run once and work for all five geo-bias scores; (2) we solved the nan issues commonly encountered in the original implementation by introducing a background point generator which automatically adjusts the point density $\rho$ to avoid “divided by zero” errors; (3) we implement the Fibonacci Lattice algorithm to generate background points in place of the original random background point generation, which solved the reproducibility issue of the original implementation (i.e., two runs of SSI Scores may differ due to different random background points).

For SRE Scores, we choose to use the KL divergence $D_{\text{KL}}(h(N),h(P_{k}))$ as $d$ in Definition [4.3](https://arxiv.org/html/2509.23482v1#S4.Thmtheorem3). The choice of KL divergence is based on two considerations: (1) $d$ needs to be a difference measure between probability distributions, among which KL divergence is the most commonly used and most generally applicable (e.g., KL divergence can be computed in $\mathcal{O}(n)$ complexity for any discrete distributions while Wasserstein distance may be as complex as $\mathcal{O}(n^{3})$ (Edmonds & Karp, 1972)); (2) KL divergence has physical meanings in that it measures the information gap between two distributions (i.e., relative-entropy), which can be interpreted as the bits needed to transform a geo-biased map into an unbiased one, potentially useful in transfer learning. We also choose to use the normalized ROI size $\#N_{m}/\sum_{m}\#N_{m}$ as the weight $w_{m}$, but one can always use other factors of interest, such as area, population, and GDP for weighting.

The computation of Local SRE Scores is described in Algorithm [1](https://arxiv.org/html/2509.23482v1#alg1). Intuitively, if the model performance does not have geo-bias under the given partitioning $\mathbb{PAR}$, the probability of encountering good/bad performance points in each patch $P_{k}$ (e.g., square/ring/sector) should be similar. We use the histograms $h(N)\leftarrow\{\#\{b_{i}\leq\pi_{s}<b_{i+1}\}~|~(\pi_{s},L_{s})\in N\}$ and $h(P_{k})\leftarrow\{\#\{b_{i}\leq\pi_{t}<b_{i+1}\}~|~(\pi_{t},L_{t})\in P_{k}\}$ as the empirical distributions of the model performance in the corresponding ROI $N$ and Patch $P_{k}$. Then, the KL-divergence between $h(N)$ and $h(P_{k})$ is used as the measure of information gap between the patch and the entire ROI. Finally, the weighted sum (over the sizes of patches) of all KL-divergences across the ROI $N$ is used as the SRE Score for the local region of interest (see Definition [4.3](https://arxiv.org/html/2509.23482v1#S4.Thmtheorem3)).

## 5 Experiments

The experiments in this research consist of three tasks:

1. Geo-Aware Image Classification:
We conduct image classification on three geo-tagged image datasets: iNat2017, iNat2018, and fMoW.
The models evaluated include both an image-only classifier (No Prior) and image classifiers enhanced with four commonly used location encoders: Radial Basis Function (RBF), Space2Vec-theory (Space2Vec), NeRF, and Sphere2Vec-sphereC (Sphere2Vec). The model performance is measured in binary numbers: 0 for wrong classes and 1 for correct classes.

2. Geo-Aware Image Regression:
We use the same models as in general image classification tasks, except that we predict continuous values which represent population density, forest coverage percentage, nightlights luminosity, and other indices at the given location. The benchmark we use is MOSAIKS (Rolf et al., 2021).
The model performance is measured in binary numbers: 0 for absolute errors smaller than the empirical variance of all prediction errors and 1 otherwise.

**Table 1: Abbreviations used in experiments.**
| Abbreviation | U-SSI | M-SSI | SG-SRE | DL-SRE | DS-SRE | SPAD |
| --- | --- | --- | --- | --- | --- | --- |
| Meaning | Unmarked SSI | Marked SSI | Scale-Grid SRE | Distance-Lag SRE | Direction-Sector SRE | SPace-As-Distribution Score |

3. Remote Sensing (RS) Image Classification:
We experiment with four RS image classification datasets: EuroSat (Helber et al., 2019), fMoW-sentinel (Cong et al., 2022), WorldStrat-IPCC, and WorldStrat-LCCS (Cornebise et al., 2022).
To study the geo-bias of different FMs, we pick 2 remote sensing foundation models, e.g., CROMA (Fuller et al., 2024) and SATMAE (Cong et al., 2022), along with an LLM, GPT-4o, to classify multi-spectral satellite imagery. The model performance is measured in binary numbers: 0 for wrong classes and 1 for correct classes.
We report the accuracy for classification and $R^{2}$ for regression. We also report all five global geo-bias scores (Unmarked SSI, Marked SSI, Scale-Grid SRE, Distance-Lag SRE, and Direction-Sector SRE) together with a baseline SPace-As-Distribution Score proposed by Xie et al. (2022). The global scores are computed over ROIs that are not all 0s or all 1s for the sake of computational stability (in this case, the scores could become infinity). The abbreviations we use throughout the experiment section are listed in Table [1](https://arxiv.org/html/2509.23482v1#S5.T1). For more information about the experiment setup, please see Appendix Table [5](https://arxiv.org/html/2509.23482v1#A2.T5).

**Table 3: $R^{2}$ and Global Geo-Bias Scores of geo-aware neural regression. All experiments use ROI radius 0.2 radian, scale 0.1 radian, lag 0.05 radian, number of splits 8. Bold numbers indicate the best performance or the lowest geo-bias.**
|  | Model | R^2 $\uparrow$ | U-SSI $\downarrow$ | M-SSI $\downarrow$ | SG-SRE $\downarrow$ | DL-SRE $\downarrow$ | DS-SRE $\downarrow$ | SPAD $\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Population<br>Density | No Prior | 0.38 | 13.11 | 4.26 | 5.58 | 28.75 | 22.44 | 21.68 |
| rbf | 0.25 | 14.13 | 3.97 | 0.47 | 22.33 | 18.29 | 21.34 |  |
| Space2Vec | 0.57 | 15.53 | 3.04 | 1.65 | 33.53 | 22.24 | 20.68 |  |
| NeRF | 0.60 | 17.51 | 3.38 | 18.71 | 26.86 | 16.47 | 22.15 |  |
| Sphere2Vec | 0.63 | 15.01 | 2.82 | 4.57 | 25.91 | 14.86 | 21.90 |  |
| Forest<br>Cover | No Prior | 0.52 | 20.32 | 3.51 | 47.78 | 144.54 | 297.32 | 23.43 |
| rbf | 0.54 | 19.60 | 2.63 | 37.28 | 126.78 | 299.58 | 24.80 |  |
| Space2Vec | 0.73 | 19.48 | 3.87 | 50.67 | 164.10 | 343.70 | 25.15 |  |
| NeRF | 0.68 | 18.08 | 2.85 | 37.98 | 149.73 | 305.72 | 25.84 |  |
| Sphere2Vec | 0.73 | 21.53 | 4.40 | 21.23 | 130.51 | 284.26 | 24.95 |  |
| Nightlight<br>Luminosity | No Prior | 0.33 | 20.48 | 2.51 | 7.45 | 47.32 | 19.26 | 23.25 |
| rbf | 0.32 | 21.62 | 3.71 | 25.40 | 21.81 | 51.73 | 22.92 |  |
| Space2Vec | 0.21 | 20.19 | 2.96 | 4.11 | 7.65 | 12.05 | 21.57 |  |
| NeRF | 0.23 | 20.67 | 2.99 | 9.19 | 17.71 | 78.18 | 22.59 |  |
| Sphere2Vec | 0.35 | 20.13 | 2.18 | 10.23 | 9.76 | 40.05 | 21.60 |  |
| Elevation | No Prior | 0.27 | 22.33 | 3.93 | 30.71 | 21.76 | 106.08 | 21.26 |
| rbf | 0.39 | 21.79 | 4.50 | 9.61 | 6.97 | 24.96 | 19.62 |  |
| Space2Vec | 0.78 | 20.22 | 3.51 | 4.29 | 6.92 | 16.60 | 20.74 |  |
| NeRF | 0.76 | 20.82 | 4.25 | 7.43 | 9.44 | 26.37 | 19.88 |  |
| Sphere2Vec | 0.82 | 21.25 | 4.84 | 4.42 | 17.51 | 26.27 | 20.44 |  |

The hyperparameters used in the experiments are: radius of ROI, grid size for SG-SRE, lag width for DL-SRE, the number of splits for DS-SRE. The choice of hyperparameters affects the amount of data points we use in computing geo-bias scores. For example, if the radius of an ROI is 1 km, it is likely that each ROI only contains one data point, and we are unable to compute geo-bias scores. In order to avoid such extreme cases, we select the appropriate hyperparameters for each dataset based on their data spatial distribution. The principle is: (1) each ROI contains at least 100 points, and (2) at least 2 patches in one ROI contain more than 10 points. All hyperparameters are reported in the experiment tables. As to the baseline SPAD Score, it ranges from 0 to 100 and is calculated using a maximum of 100 rows, 100 columns, and a partitioning sample size of 100 (Xie et al., 2022). For ablation studies on the influence of hyperparameters, please see Table [6](https://arxiv.org/html/2509.23482v1#A2.T6) and Table [7](https://arxiv.org/html/2509.23482v1#A2.T7) in the Appendix.

### 5.1 Geo-Bias of Task-Specific GeoAI Models

Table [2](https://arxiv.org/html/2509.23482v1#S5.T2) reports the geo-tagged image classification geo-bias. By comparing the model accuracy with the geo-bias scores, we can see that there is no strong correlation, which means geo-bias scores are a (relatively) independent dimension of evaluation, and it is not sufficient to only report the overall performance. Moreover, we observe that the geo-bias of task-specific GeoAI models tends to be mostly dependent on datasets rather than on models. For example, all models have significantly lower SRE Scores on the iNat2018 dataset (notice that SSI and SRE Scores are log-based), which suggests that iNat2018 might have more spatially balanced data.

In contrast, while the NeRF model shows very low geo-bias in terms of scale, distance, and direction on the iNat2017 and iNat2018 datasets, it performs significantly more biased on the fMoW dataset. Similar observations can be made from Table [3](https://arxiv.org/html/2509.23482v1#S5.T3.fig1) which reports the geo-aware image regression geo-bias, where the Forest Cover dataset shows drastically larger geo-bias in terms of all three SRE Scores. See Figure [4](https://arxiv.org/html/2509.23482v1#A2.F4) in Appendix [B.2](https://arxiv.org/html/2509.23482v1#A2.SS2) for an intuitive visualization.

Our hypothesis is that it is because such GeoAI models explicitly leverage the geographical metadata (e.g., latitudes and longitudes) for predictions, which causes the model to overfit to the spatial distributions of training data (Mai et al., 2020; 2023). If the dataset is geo-biased in data sampling, the performance will also suffer regardless of which model is used. In this case, we should focus on improving the data quality, such as class balance, spatial coverage, etc.

**Table 4: Accuracy and Global Geo-Bias Scores of remote sensing image classification. All geo-bias scores use an ROI radius of 0.01 radian. Bold numbers indicate the best performance or the lowest geo-bias. Bold numbers indicate the best performance or the lowest geo-bias scores.**
|  | Model | Acc $\uparrow$ | U-SSI $\downarrow$ | M-SSI $\downarrow$ | SG-SRE $\downarrow$ | DL-SRE $\downarrow$ | DS-SRE $\downarrow$ | SPAD $\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| fMoW-sentinel | Hyperparam | - | - | - | 0.01 | 0.01 | 8 | - |
| GPT-4o | 5.72 | 516.80 | 63.96 | 3.81 | 1.32 | 0.76 | 18.25 |  |
| CROMA ft | 52.67 | 560.80 | 447.89 | 61.47 | 16.79 | 19.94 | 39.19 |  |
| CROMA lp | 31.46 | 560.11 | 466.31 | 152.72 | 38.69 | 42.17 | 36.37 |  |
| SatMAE ft | 64.77 | 560.96 | 16.29 | 2.16 | 0.57 | 0.74 | 12.84 |  |
|  | SatMAE lp | 62.76 | 561.29 | 14.06 | 2.47 | 0.61 | 0.88 | 12.36 |
| WorldStrat<br>-LCCS | Hyperparam | - | - | - | 0.05 | 0.005 | 8 | - |
| GPT-4o | 41.33 | 399.27 | 276.18 | 7.87 | 54.87 | 62.21 | 66.93 |  |
| CROMA ft | 60.78 | 354.01 | 275.65 | 12.91 | 18.89 | 23.46 | 63.23 |  |
| CROMA lp | 58.73 | 369.52 | 305.35 | 3.98 | 10.59 | 21.51 | 66.56 |  |
| SatMAE ft | 52.37 | 418.63 | 6.00 | 0.06 | 0.11 | 0.14 | 16.51 |  |
|  | SatMAE lp | 44.29 | 416.44 | 6.95 | 0.06 | 0.12 | 0.16 | 15.29 |
| WorldStrat<br>-IPCC | Hyperparam | - | - | - | 0.05 | 0.005 | 8 | - |
| GPT-4o | 51.92 | 404.86 | 200.12 | 3.82 | 12.06 | 12.40 | 56.40 |  |
| CROMA ft | 69.61 | 359.10 | 251.67 | 7.64 | 13.21 | 14.67 | 52.27 |  |
| CROMA lp | 65.79 | 379.79 | 271.37 | 4.64 | 8.28 | 9.09 | 56.12 |  |
| SatMAE ft | 66.56 | 410.33 | 19.23 | 0.07 | 0.18 | 0.15 | 15.81 |  |
|  | SatMAE lp | 45.36 | 416.16 | 7.06 | 0.08 | 0.14 | 0.16 | 16.07 |
| EuroSAT | Hyperparam | - | - | - | 0.01 | 0.005 | 12 | - |
| GPT-4o | 44.89 | 119.43 | 79.59 | 2.62 | 1.29 | 0.64 | 53.52 |  |
| CROMA ft | 97.43 | 115.72 | 96.58 | 0.25 | 0.67 | 0.48 | 8.67 |  |
| CROMA lp | 92.87 | 100.00 | 60.35 | 0.56 | 0.44 | 0.37 | 19.23 |  |
| SatMAE ft | 74.30 | 115.93 | 13.02 | 0.03 | 0.07 | 0.05 | 15.65 |  |
|  | SatMAE lp | 56.54 | 113.19 | 6.43 | 0.02 | 0.07 | 0.06 | 34.91 |

### 5.2 Geo-Bias of Remote Sensing Foundation Models

Foundation models, which are trained on massive data, are believed to suffer less from data bias. This partially matches our observation. Table [4](https://arxiv.org/html/2509.23482v1#S5.T4.fig1) reports the performance of ChatGPT and two remote sensing foundation models with variations (ft stands for “finetuning” and lp stands for “linear probing”). The geo-bias scores are significantly lower than the task-specific counterparts. Besides, unlike the task-specific case, the differences in geo-bias scores of the same model across different datasets are not prominent. Instead, we observe that while the CROMA ft model outperforms SatMAE almost consistently, it also has way stronger geo-bias of all types. See Figure [5](https://arxiv.org/html/2509.23482v1#A2.F5) in Appendix [B.2](https://arxiv.org/html/2509.23482v1#A2.SS2) for an illustration of this phenomenon.

Our hypothesis is that, while foundation models are trained on a massive amount of data and thus less affected by data geo-bias, their powerful learning capability may overfit the implicit geographical information in the data, especially considering that the EuroSAT dataset only covers Europe, and the implicit spatial patterns can be easily acquired without explicit input of geolocations. In other words, whether a foundation model is prone to geo-bias might be more dependent on its own model architecture, i.e., how suitable this model is for learning spatial features.

## 6 Conclusion, Limitation & Future Work

Our work is an example of using the rich domain knowledge (spatial data analysis, point pattern analysis) to guide AI research. We provide a powerful framework of designing geo-bias scores that explicitly describes what spatial factors you care about and gives clear-cut, information-theoretic interpretations of the evaluation results. We see this as a great opportunity to encourage researchers to report geo-bias scores in their work so that we are not only racing for higher model performance, but also keeping in mind the spatial fairness issues behind it. In this paper, we limit our discussion on first-order, relative-entropy-based geo-bias scores, but we will design more geo-bias scores in our future work that deal with other intricate spatial factors, for example, network and time-space. Besides, since the geo-bias scores we propose are based on self-information and relative information, they are differentiable and compatible with most existing training objectives. We see great potential in introducing the geo-bias scores as debiasing loss functions and help train more fair models.

## Reproducibility Statement

We provide detailed descriptions of our methods, algorithm implementation, models, datasets and hyperparameters in Section [3](https://arxiv.org/html/2509.23482v1#S3), [4](https://arxiv.org/html/2509.23482v1#S4), [5](https://arxiv.org/html/2509.23482v1#S5) and Appendix [B](https://arxiv.org/html/2509.23482v1#A2). To support replication, we have uploaded anonymized source code as supplementary materials.

## Ethics Statement

We use only publicly available datasets and established benchmarks for evaluation; experiments operate at regional/task level rather than individual profiling. All datasets are used under their licenses, and results are reported for scientific benchmarking only. We adhere to the ICLR Code of Ethics throughout submission, review, and discussion.

## References

- Anselin (1995)
Luc Anselin.
Local indicators of spatial association—lisa.
*Geographical analysis*, 27(2):93–115, 1995.
- Anselin (2019)
Luc Anselin.
A local indicator of multivariate spatial association: extending geary’s c.
*Geographical Analysis*, 51(2):133–150, 2019.
- Ben-Said (2021)
Mariem Ben-Said.
Spatial point-pattern analysis as a powerful tool in identifying pattern-process relationships in plant ecology: an updated review.
*Ecological Processes*, 10:1–23, 2021.
- Besag (1977)
Julian Besag.
Contribution to the discussion on dr ripley’s paper.
*JR Stat Soc B*, 39:193–195, 1977.
- Binns (2018)
Reuben Binns.
Fairness in machine learning: Lessons from political philosophy.
In *Conference on Fairness, Accountability and Transparency*, volume abs/1712.0, pp. 149–159, 2018.
URL [http://arxiv.org/abs/1712.03586](http://arxiv.org/abs/1712.03586).
- Bommasani et al. (2021)
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al.
On the opportunities and risks of foundation models.
*arXiv preprint arXiv:2108.07258*, 2021.
- Boots & Getis (2020)
Barry N Boots and Arthur Getis.
Point pattern analysis.
2020.
- Caton & Haas (2024)
Simon Caton and Christian Haas.
Fairness in machine learning: A survey.
*ACM Computing Surveys*, 56(7):1–38, 2024.
- Chen et al. (2022)
Xiang Chen, Xinyue Ye, Michael J Widener, Eric Delmelle, Mei-Po Kwan, Jerry Shannon, Elizabeth F Racine, Aaron Adams, Lu Liang, and Peng Jia.
A systematic review of the modifiable areal unit problem (maup) in community food environmental research.
*Urban Informatics*, 1(1):22, 2022.
- Christie et al. (2018)
Gordon Christie et al.
Functional map of the world.
In *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*, 2018.
- Cong et al. (2022)
Yezhen Cong et al.
Satmae: Pre-training transformers for temporal and multi-spectral satellite imagery.
In *Advances in Neural Information Processing Systems*, volume 35, pp. 197–211, 2022.
- Cornebise et al. (2022)
Julien Cornebise, Ivan Oršolić, and Freddie Kalaitzis.
Open high-resolution satellite imagery: The worldstrat dataset–with application to super-resolution.
*Advances in Neural Information Processing Systems*, 35:25979–25991, 2022.
- Deng et al. (2024)
Haojian Deng, Kai Liu, JiaLi Feng, and Yongzhu Xiong.
Tackling the modifiable areal unit problem: Enhancing urban sustainability through improved land surface temperature and its influencing factors analysis.
*Sustainable Cities and Society*, 114:105747, 2024.
- Dhamala et al. (2021)
Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta.
Bold: Dataset and metrics for measuring biases in open-ended language generation.
In *Proceedings of the 2021 ACM conference on fairness, accountability, and transparency*, pp. 862–872, 2021.
- Edmonds & Karp (1972)
Jack Edmonds and Richard M Karp.
Theoretical improvements in algorithmic efficiency for network flow problems.
*Journal of the ACM (JACM)*, 19(2):248–264, 1972.
- Faisal & Anastasopoulos (2023)
Fahim Faisal and Antonios Anastasopoulos.
Geographic and geopolitical biases of language models.
In *Proceedings of the 3rd Workshop on Multi-lingual Representation Learning (MRL)*, pp. 139–163, 2023.
- Fotheringham & Wong (1991)
A Stewart Fotheringham and David WS Wong.
The modifiable areal unit problem in multivariate statistical analysis.
*Environment and planning A*, 23(7):1025–1044, 1991.
- Fuller et al. (2024)
Anthony Fuller, Koreen Millard, and James Green.
Croma: Remote sensing representations with contrastive radar-optical masked autoencoders.
*Advances in Neural Information Processing Systems*, 36, 2024.
- Gallegos et al. (2024)
Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed.
Bias and fairness in large language models: A survey.
*Computational Linguistics*, pp. 1–79, 2024.
- Gianfrancesco et al. (2018)
Milena A Gianfrancesco, Suzanne Tamang, Jinoos Yazdany, and Gabriela Schmajuk.
Potential biases in machine learning algorithms using electronic health record data.
*JAMA internal medicine*, 178(11):1544–1547, 2018.
- Goodchild (2022)
Michael F. Goodchild.
The openshaw effect.
*International Journal of Geographical Information Science*, 36(9):1697–1698, 2022.
doi: 10.1080/13658816.2022.2102637.
URL [https://doi.org/10.1080/13658816.2022.2102637](https://doi.org/10.1080/13658816.2022.2102637).
- Gordon & Desjardins (1995)
Diana F Gordon and Marie Desjardins.
Evaluation and selection of biases in machine learning.
*Machine learning*, 20:5–22, 1995.
- Hardt et al. (2016)
Moritz Hardt, Eric Price, and Nati Srebro.
Equality of opportunity in supervised learning.
*Advances in neural information processing systems*, 29, 2016.
- Hay (1995)
Alan M Hay.
Concepts of equity, fairness and justice in geographical studies.
*Transactions of the Institute of British Geographers*, pp. 500–508, 1995.
- Helber et al. (2019)
Patrick Helber et al.
Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.
*IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing*, 12(7):2217–2226, 2019.
- Illian et al. (2008)
Janine Illian, Antti Penttinen, Helga Stoyan, and Dietrich Stoyan.
*Statistical analysis and modelling of spatial point patterns*.
John Wiley & Sons, 2008.
- (27)
iNaturalist 2018 competition dataset.
iNaturalist 2018 competition dataset.
[https://github.com/visipedia/inat_comp/tree/master/2018](https://github.com/visipedia/inat_comp/tree/master/2018), 2018.
- Jelinski & Wu (1996)
Dennis E Jelinski and Jianguo Wu.
The modifiable areal unit problem and implications for landscape ecology.
*Landscape ecology*, 11:129–140, 1996.
- Jo & Gebru (2020)
Eun Seo Jo and Timnit Gebru.
Lessons from archives: Strategies for collecting sociocultural data in machine learning.
In *Proceedings of the 2020 conference on fairness, accountability, and transparency*, pp. 306–316, 2020.
- Kamiran & Calders (2012)
Faisal Kamiran and Toon Calders.
Data preprocessing techniques for classification without discrimination.
*Knowledge and information systems*, 33(1):1–33, 2012.
- Kamishima et al. (2011)
Toshihiro Kamishima, Shotaro Akaho, and Jun Sakuma.
Fairness-aware learning through regularization approach.
In *2011 IEEE 11th International Conference on Data Mining Workshops*, pp. 643–650. IEEE, 2011.
- Liu et al. (2022)
Zilong Liu, Krzysztof Janowicz, Ling Cai, Rui Zhu, Gengchen Mai, and Meilin Shi.
Geoparsing: Solved or biased? an evaluation of geographic biases in geoparsing.
*AGILE: GIScience Series*, 3:9, 2022.
- Mai et al. (2020)
Gengchen Mai, Krzysztof Janowicz, Bo Yan, Rui Zhu, Ling Cai, and Ni Lao.
Multi-scale representation learning for spatial feature distributions using grid cells.
In *The Eighth International Conference on Learning Representations*. openreview, 2020.
- Mai et al. (2023)
Gengchen Mai, Yao Xuan, Wenyun Zuo, Yutong He, Jiaming Song, Stefano Ermon, Krzysztof Janowicz, and Ni Lao.
Sphere2vec: A general-purpose location representation learning over a spherical surface for large-scale geospatial predictions.
*ISPRS Journal of Photogrammetry and Remote Sensing*, 202:439–462, 2023.
- Mai et al. (2025)
Gengchen Mai, Yiqun Xie, Xiaowei Jia, Ni Lao, Jinmeng Rao, Qing Zhu, Zeping Liu, Yao-Yi Chiang, and Junfeng Jiao.
Towards the next generation of geospatial artificial intelligence.
*International Journal of Applied Earth Observation and Geoinformation*, 136:104368, 2025.
- Manvi et al. (2024)
Rohin Manvi, Samar Khanna, Marshall Burke, David B Lobell, and Stefano Ermon.
Large language models are geographically biased.
In *International Conference on Machine Learning*, pp. 34654–34669. PMLR, 2024.
- Mildenhall et al. (2020)
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.
Nerf: Representing scenes as neural radiance fields for view synthesis.
In *ECCV*, 2020.
- Mildenhall et al. (2021)
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.
Nerf: Representing scenes as neural radiance fields for view synthesis.
*Communications of the ACM*, 65(1):99–106, 2021.
- Moran (1950)
Patrick AP Moran.
Notes on continuous stochastic phenomena.
*Biometrika*, 37(1/2):17–23, 1950.
- Nadeem et al. (2021)
Moin Nadeem, Anna Bethke, and Siva Reddy.
Stereoset: Measuring stereotypical bias in pretrained language models.
In *Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)*, pp. 5356–5371, 2021.
- Nangia et al. (2020)
Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R Bowman.
Crows-pairs: A challenge dataset for measuring social biases in masked language models.
In *2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020*, pp. 1953–1967. Association for Computational Linguistics (ACL), 2020.
- Nelson et al. (2022)
TA Nelson, MF Goodchild, and DJ Wright.
Accelerating ethics, empathy, and equity in geographic information science.
*Proceedings of the National Academy of Sciences*, 119(19):e2119967119, 2022.
- News (2015)
BBC News.
Google apologises for photos app’s racist blunder.
*BBC News*, 2015.
URL [https://bbc.com/news/technology-33347866](https://bbc.com/news/technology-33347866).
- Openshaw (1984)
Stan Openshaw.
The modifiable areal unit problem.
*Concepts and techniques in modern geography*, 1984.
- O’sullivan (2003)
D O’sullivan.
*Geographic information analysis*, volume 436.
Wiley, 2003.
- Parrish et al. (2021)
Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R Bowman.
Bbq: A hand-built bias benchmark for question answering.
*arXiv preprint arXiv:2110.08193*, 2021.
- Pearl (2022)
Judea Pearl.
Comment: understanding simpson’s paradox.
In *Probabilistic and causal inference: The works of judea Pearl*, pp. 399–412. 2022.
- Ripley (1977)
Brian D Ripley.
Modelling spatial patterns.
*Journal of the Royal Statistical Society: Series B (Methodological)*, 39(2):172–192, 1977.
- Rolf et al. (2021)
Esther Rolf, Jonathan Proctor, Tamma Carleton, Ian Bolliger, Vaishaal Shankar, Miyabi Ishihara, Benjamin Recht, and Solomon Hsiang.
A generalizable and accessible approach to machine learning with global satellite imagery.
*Nature Communications*, 12(1), July 2021.
ISSN 2041-1723.
doi: 10.1038/s41467-021-24638-z.
URL [http://dx.doi.org/10.1038/s41467-021-24638-z](http://dx.doi.org/10.1038/s41467-021-24638-z).
- Serna et al. (2020)
Ignacio Serna, Aythami Morales, Julian Fierrez, Manuel Cebrian, Nick Obradovich, and Iyad Rahwan.
Sensitiveloss: Improving accuracy and fairness of face representations with discrimination-aware deep learning.
*arXiv preprint arXiv:2004.11246*, 2020.
- Sheng et al. (2019)
Emily Sheng, Kai-Wei Chang, Prem Natarajan, and Nanyun Peng.
The woman worked as a babysitter: On biases in language generation.
In *Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)*, pp. 3407–3412, 2019.
- Steed & Caliskan (2021)
Ryan Steed and Aylin Caliskan.
Image representations learned with unsupervised pre-training contain human-like biases.
In *Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency*, pp. 701–713, 2021.
- Swift et al. (2008)
Andrew Swift, Lin Liu, and James Uber.
Reducing maup bias of correlation statistics between water quality and gi illness.
*Computers, Environment and Urban Systems*, 32(2):134–148, 2008.
- Touvron et al. (2023)
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.
Llama: Open and efficient foundation language models.
*arXiv preprint arXiv:2302.13971*, 2023.
- Wang et al. (2024)
Zhangyu Wang, Krzysztof Janowicz, Gengchen Mai, and Ivan Majic.
Probing the Information Theoretical Roots of Spatial Dependence Measures.
In Benjamin Adams, Amy L. Griffin, Simon Scheider, and Grant McKenzie (eds.), *16th International Conference on Spatial Information Theory (COSIT 2024)*, volume 315 of *Leibniz International Proceedings in Informatics (LIPIcs)*, pp. 9:1–9:18, Dagstuhl, Germany, 2024. Schloss Dagstuhl – Leibniz-Zentrum für Informatik.
ISBN 978-3-95977-330-0.
doi: 10.4230/LIPIcs.COSIT.2024.9.
URL [https://drops.dagstuhl.de/entities/document/10.4230/LIPIcs.COSIT.2024.9](https://drops.dagstuhl.de/entities/document/10.4230/LIPIcs.COSIT.2024.9).
- Wu et al. (2024)
Nemin Wu, Qian Cao, Zhangyu Wang, Zeping Liu, Yanlin Qi, Jielu Zhang, Joshua Ni, Xiaobai Yao, Hongxu Ma, Lan Mu, et al.
Torchspatial: A location encoding framework and benchmark for spatial representation learning.
*arXiv preprint arXiv:2406.15658*, 2024.
- Xie et al. (2022)
Yiqun Xie, Erhu He, Xiaowei Jia, Weiye Chen, Sergii Skakun, Han Bao, Zhe Jiang, Rahul Ghosh, and Praveen Ravirathinam.
Fairness by “where”: A statistically-robust and model-agnostic bi-level learning framework.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 36, pp. 12208–12216, 2022.
- Yang et al. (2023)
Ke Yang, Charles Yu, Yi R Fung, Manling Li, and Heng Ji.
Adept: A debiasing prompt framework.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 37, pp. 10780–10788, 2023.
- Zhang et al. (2022)
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al.
Opt: Open pre-trained transformer language models.
*arXiv preprint arXiv:2205.01068*, 2022.

## Appendix A Appendix

## Appendix B Appendix

### B.1 Supplementary Information of SSI Scores

Below is the algorithm we use to implement the SSI Scores described in (Wu et al., 2024). Notice that it is slightly different from the original algorithm in the way of generating background points.

Figure: Algorithm 2 Local Unmarked/Marked SSI Algorithm

### B.2 Supplementary Figures

Figure: Figure 4: Geographical distributions of error rate and local geo-bias scores of NeRF on fMoW, iNaturalist2017 and iNaturalist2018 on different datasets.
Refer to caption: figs/gbmap_nerf_rebuttal_resize.jpg

We visualize the spatial distributions of reported geo-bias scores on selected datasets and tasks. In all visualizations, the values are normalized to the range of 0 to 1. A darker red color indicates higher bias/error. The visual illustration conforms with the conclusions made in Section [5](https://arxiv.org/html/2509.23482v1#S5).

Figure: Figure 5: Geographical distributions of error rate and local geo-bias scores of different remote sensing foundation models on EuroSAT. The spatial distributions of U-SSI across models are the same because local U-SSI scores are unmarked and only dependent on data instead of models.
Refer to caption: figs/gbmap_euro_rebuttal2.jpg

### B.3 Supplementary Tables

#### B.3.1 Experiment setup

**Table 5: Detailed information of datasets and models.**
| Dataset | Description |
| --- | --- |
| iNat2017 | A global species recognition dataset designed for the iNaturalist 2017 challenges (iNaturalist 2018 competition dataset,), containing 675,170 images and 5,089 unique categories. |
| iNat2018 | A global species recognition dataset designed for the iNaturalist 2018 challenges (iNaturalist 2018 competition dataset,), containing 461,939 images and 8,142 unique categories. |
| fMoW | A global RS image classification dataset (Christie et al., 2018) that includes RS images representing a wide range of land use types. |
| fMoW-sentinel | A global Sentinel-2 dataset cross-referenced with fMoW, as a benchmark for training models on multi-spectral satellite imagery. (Cong et al., 2022) |
| WorldStrat-LCCS | A global collection of high-resolution satellite imagery using the Land Cover Classification System (LCCS) to categorize land cover types. (Cornebise et al., 2022) |
| WorldStrat-IPCC | A global collection of high-resolution satellite imagery using the Intergovernmental Panel on Climate Change (IPCC) classification system. (Cornebise et al., 2022) |
| EuroSAT | A European dataset designed for land use and land cover classification using satellite imagery. Helber et al. (2019) |
| Population Density | A uniformly-at-random distributed global RS image dataset Wu et al. (2024) that contains 425,637 samples and corresponding estimations of population density. |
| Forest Cover | A uniformly-at-random distributed global RS image dataset Wu et al. (2024) that contains 498,106 samples and corresponding estimations of forest cover rate. |
| Nightlights Luminosity | A uniformly-at-random distributed global RS image dataset Wu et al. (2024) that contains 492,226 samples and corresponding nightlights luminosity. |
| Elevation | A uniformly-at-random distributed global RS image dataset Wu et al. (2024) that contains 498,115 samples and corresponding elevation. |
| Model | Description |
| No Prior | Model using image classifier only |
| rbf | Mai et al. (2020) is a kernel-based location encoder |
| Space2Vec (theory) | A multi-scale location encoder for Euclidean space Mai et al. (2020) |
| NeRF | A location encoder using Neural Radiance Fields (NeRF). Mildenhall et al. (2021) |
| Sphere2Vec (sphereC) | A multi-scale location encoder for spherical surface. |
| GPT-4o | A LLM developed by OpenAI |
| CROMA | A RS foundation model with contrastive radar-optical masked autoencoders. (Fuller et al., 2024) |
| SATMAE | A RS foundation model using pre-training transformers for temporal and multi-spectral satellite imagery. (Cong et al., 2022) |

#### B.3.2 Hyperparameter sensitivity of SSI Scores

#### B.3.3 Hyperparameter sensitivity of SRE Scores

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 2
presentation: 2
questions: In addition to the weaknesses discussed earlier, I would appreciate the authors' responses on the following:

* In Figure 3, are the bars on the left for a single patch?

* Why was the parameter sensitivity analysis conducted only for a limited subset of models and datasets?

*Why was GPT-4o chosen as one of the fms?
soundness: 2
strengths: The proposed framework is model-agnostic, addressing a key limitation of prior work, which tends to be restricted to specific tasks. They include ways to quantify bias by including intricate details about space, distance, and scale. This ensures the quantifying of bias in a granular way.
summary: The paper introduces an information-theoretic and model-agnostic framework for quantifying geographical bias. This framework can integrate existing metrics and proposes three new metrics that account for multi-scalability, distance decay, and anisotropy. The authors perform experiments across multiple models to reveal that there exist geobias in both task-specific geoAI models and foundation models.
weaknesses: In my opinion, the abstract of the paper is misleading because of the part: “ but will also establish a foundation for integrating spatial fairness into the design, deployment, and evaluation of AI systems”. The framework is purely diagnostic, and no mitigation scheme is mentioned.

Figure 1 is a very important figure that helps motivate the importance of quantifying bias in different ways. But it's hard to understand what the different colored points represent since they are not all explicitly mentioned. This makes the overall message of the figure unclear. Additionally, using the same spatial area to illustrate each metric would make the differences between them easier to compare and understand. The acronym SSI is not defined and explained at this stage (it is done way later in the paper). It would have been to see if the authors described what they meant by geo bias clearly with some easy-to-understand intuitive examples.

The writing of the paper is occasionally difficult to follow. For instance, “Spatial point patterns always involve multiple locations (a single point will not form 'patterns'), so we define the unit to evaluate geo-bias as.”. This is followed by a series of undefined notations (e.g., what i,m,s, and t index). The first paragraph of Section 4 is missing logical flow. It starts off by mentioning the objectives, then explains how it serves the purpose. Where the purpose itself is not mentioned. Additionally, they state “three key factors we need to consider when designing new geo-bias metrics”, what are those key factors? It's very important to clarify the differences between unmarked SSI and marked SSI. Figure 2 can be a little misleading since two different locations are used to visualize the geo-bias scenarios. The main idea is that they would both have the same number of points, except one would just have the points against the unobserved background, and the other would have the same points with different colors representing high and low performance. The placement of Table 1 is rather odd, I would want to have it near the intro, not near the end, when I'm almost done reading. And these acronyms are used throughout the texts, not just in the following section.

For definition 4.3, having #P_k/#N outside the sum function makes no sense as to what the definition intends. They are supposed to be using the weight for each patch, not just one.

Converting regression values to binary might lose valuable information, and the choice of threshold should be justified.

Since SPAD is used as a baseline, details about what exactly it quantifies should be mentioned.

The claim that “The geo-bias scores are significantly lower than the task-specific counterparts” is not supported by the presented tables. While Table 4 shows lower bias values relative to Table 3, the comparison to Table 2 contradicts this assertion: many entries in Table 4 are comparable to, or higher than, those in Table 2. The authors should either (a) clarify exactly which comparisons support the “significantly lower” claim, (b) report formal significance tests (with effect sizes and confidence intervals) for the claimed differences, or (c) revise the statement to accurately reflect the mixed results.

The parameter sensitivity analysis is conducted only for a limited subset of models and datasets, without any justification for their selection. Without explaining why these particular models and datasets were chosen, the analysis cannot be considered a comprehensive or reliable assessment of parameter tuning.

The paper does motivate the importance of a model-agnostic approach and the importance of spatial info like (distance, direction, and scale). The discussion remains at a general level, reporting the presence of bias without interpreting its variation by scale, distance decay, or directional (anisotropic) patterns. Providing such an interpretation would strengthen the link between the theoretical motivation and the empirical findings.

In Figure 3, are the bars on the left for a single patch? The caption itself is not informative enough, and there's no mention of this in the text.
Details about the hyperparameter tuning of the models are also missing.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. How should appropriate "unbiased" reference patterns be selected for different application scenarios? Can you provide a  guiding discussions?
2. Can you provide some discussions showing how these bias scores could guide practitioners in improving their models?
3.  SRE scores themselves show high hyperparameter  sensitivity. How will you address this issue?
soundness: 3
strengths: 1. I like this paper — combining spatial point pattern analysis with information theory for geo-bias evaluation is an innovative perspective that provides a theoretical lens. Although geographic bias might seem like a narrow topic within the broader fairness discussion, this kind of specialized, systematic, and in-depth evaluation has strong practical value.

2. This work successfully demonstrates how existing geo-bias metrics (Unmarked SSI and Marked SSI) can be interpreted within this framework, demonstrating good extensibility. The proposed metrics are model-agnostic and applicable to different types of AI models. 

3. The spatially explicit nature makes evaluation results more interpretable. Experiments cover multiple tasks, datasets, and model types, enhancing the credibility of conclusions
summary: This paper focus on the geo-bias evaluation in AI models, the authors propose GeoBS, an information-theoretic, model-agnostic, and spatially explicit framework for fair geo-bias evaluation. They further introduce three SRE scores considering multi-scalability, distance decay, and anisotropy. Experiments on 3 tasks, 8 datasets, and 8 models show that both GeoAI and several general-purpose models exhibit geo-bias, highlighting the need for spatial fairness in AI design.
weaknesses: 1. The definition of "spatial homogeneity" is oversimplified; it's difficult to define what constitutes an "unbiased" reference pattern in practice. 
2. Lacks sensitivity analysis for different reference pattern choices; choice of KL divergence lacks sufficient justification.
3. Only considers first-order statistics, potentially missing important spatial interaction information.
4. Hyperparameter choices (ROI radius, grid size, ...) significantly impact results, but lack systematic selection guidance.
5. Lacks statistical significance testing, making it difficult to judge whether observed differences are statistically meaningful.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. Can the authors clarify the practical importance of the theoretical frame-work? Does it provide any guarantees, etc?
2. Can the authors provide any way to validate the metrics, for example, on synthetic data?
3. What are confidence intervals for the metrics?
4. What is the advantage of the proposed metric compared to performing independence test (chi-square on contingency tables) between model-correctness and geographical partitions?
soundness: 2
strengths: The new metrics can be valuable for encouraging and enabling design of less geographically biased models. The insights about the existence of geo-bias in foundation models is also important for the users of these models in practice.
summary: This paper proposes a mathematical frame-work for measuring the geographic bias (geo-bias) of predictive models. The frame-work asserts that there are three main aspects in the design space of geo-bias metric: the map, the reference pattern, and the difference measure, whose variations can construct different metrics. The paper also proposes new metrics of geo-bias that can capture bias related to multi-scalability, distance decay, and anisotropy, and provides experiments that utilize these metrics to measure the geo-bias of several foundation models.
weaknesses: 1. I find the mathematical frame-work uninformative. In particular, it is not clear where and how the frame-work can be useful. For example, the new metrics constructed by the paper do not stem from the theoretical framework, nor rely on it for any mathematical guarantees about their validity.

2. The figures in the paper lack clarity in my opinion, and particularly the captions are very uninformative.

3. The justifications for the use of KL divergence is not convincing. The first point is that it is less computationally expensive, but there are other linear-complexity options such as total variation. The second point about KL having a “physical” interpretation seems incorrect to me, providing reference for this claim is important (note that KL is not bounded and also is not a proper distance). Lastly, it is not clear why KL is chosen compared to reverse KL.

4. In Lines 374-377, the use of a binary metric for measuring continuous regression tasks seems unjustified, and the explanation of the binary metric lacks mathematical rigor.

5. The exclusion of some ROIs from evaluation, as pointed out in Lines 400-401, is not well-justified.

6. The choice of hyperparameters seems very ad-hoc, and it is unclear how changing these hyperparameters will affect the results and ranking of the models. This makes these metrics less useful in practice.

7. The proposed metrics, and the reported results, lack any measure of statistical significance (and a way to compute confidence intervals). Therefore, it is unclear how meaningful the differences between these metrics are.

8. The paper provides no validation for its proposed metrics for use in practical applications. How can a user trust conclusions drawn from the proposed metrics?

9. The paper draws some hypotheses from its analyses (Lines 440-444 and 466-472), but does not explore them any further. Providing some evidence to reject/affirm these hypotheses would be valuable for the development of future models.

Typos:

In Definition 4.3, “#” is undefined.


Decision: Reject