No Location Left Behind: Measuring and Improving the Fairness of Implicit Representations for Earth Data

Authors: Daniel Cai, Randall Balestriero
Primary area: datasets and benchmarks
Keywords: implicit neural representations, dataset, fairness in AI, representation learning, geospatial modeling, Earth representation, wavelet, location encoding

Abstract:
Implicit neural representations (INRs) exhibit growing promise in addressing Earth representation challenges, ranging from emissions monitoring to climate modeling. However, existing methods disproportionately prioritize global average performance, whereas practitioners require fine-grained insights to understand biases and variations in these models. To bridge this gap, we introduce FAIR-Earth: a first-of-its-kind dataset explicitly crafted to challenge and examine inequities in Earth representations. FAIR-Earth comprises various high-resolution Earth signals, and uniquely aggregates extensive metadata along stratifications like landmass size and population density to assess the fairness of models. Evaluating state-of-the-art INRs across the various modalities of FAIR-Earth, we uncover striking performance disparities. Certain subgroups, especially those associated with high-frequency signals (e.g., islands, coastlines), are consistently poorly modeled by existing methods. In response, we propose spherical wavelet encodings, building on previous spatial encoding research for INRs. Leveraging the multi-resolution analysis capabilities of wavelets, our encodings yield more consistent performance over various scales and locations, offering more accurate and robust representations of the biased subgroups. These open-source contributions represent a crucial step towards facilitating the equitable assessment and deployment of implicit Earth representations.

Decision: accept (poster)
Reviewer scores: 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730031078729

thread_type:
official_review

<thread_object>
creator: reviewer_jrro

<content>
summary:
This paper addresses fairness issues in Implicit Neural Representations for Earth data. First, it introduces FAIR-EARTH, a comprehensive dataset designed to evaluate biases in Earth representations across various spatial resolutions and features. Second, to address the identified fairness issues, particularly in representing local features, the authors propose using wavelets instead of traditional harmonics, demonstrating improved representation of fine-grained geographical features while maintaining global performance.

strengths:
1. Addresses a critical and underexplored issue of fairness in Earth data representation

2. The proposed dataset provides a comprehensive framework for evaluating and comparing different approaches to Earth representation, which will be valuable for future research

3. The use of wavelets for modeling local features is theoretically well-motivated in representing fine-grained geographical features

weaknesses:
1. While the paper successfully identifies the problem and proposes a promising direction with Spherical Wavelets, it primarily remains at a conceptual level, lacking detailed implementation specifications and comprehensive empirical analysis across different wavelet types.

2. The computational complexity and scalability analysis should be presented, especially given that wavelet-based approaches typically incur higher computational costs than Spherical Harmonics due to their multi-scale nature and the need for both rotation and dilation operations.

questions:
Have you evaluated other wavelet types besides the Morlet wavelet? What were their relative performances, particularly in representing different types of geographical features? Could you provide some intuitive guidelines for selecting wavelet parameters based on the type of geographical features being represented?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730601218547

thread_type:
official_review

<thread_object>
creator: reviewer_fbst

<content>
summary:
1. This paper constructs a new dataset to unify different existing geospatial datasets under a consistent framework.

2. This paper proposes that fairness is reflected in the concurrent performance improvement of the model across different subgroups, and it conducts specific measurement and analysis based on correlation.

3. This paper proposes using SPHERICAL WAVELET for encoding to mitigate fairness issues while maintaining performance.

strengths:
1. This paper products constructs a new dataset, which aggregates extensive metadata along stratifications like landmass size and population density and unifies disparate existing geospatial datasets under a consistent framework.

2. This paper proposes that fairness is reflected in the concurrent performance improvement of the model across different subgroups. Experiments and analysis show significant disparities in the performance of the spherical harmonic method across different subgroups uncovering a strong negative correlation between landmass size and representation loss, with areas corresponding to localized signals and high-frequency features exhibiting consistently poor performance.

3. This paper proposes that encoding using SPHERICAL WAVELET solves many obvious deviations in SPHERICAL HARMONIC on many benchmarks of FAIR-EARTH while maintaining competitive performance.

weaknesses:
1. In this paper, only one of the existing methods is tested on the new dataset, which leads to the lack of convincing for the generality and accuracy of the new dataset. 

2. This paper proposes a method for measuring fairness, but in the experiments, it only conducts specific analyses on two different subgroups and one existing method.

3. This paper proposes using SPHERICAL WAVELET for encoding, but experiments are only carried out on the self-constructed dataset. Experiments should also be carried out on other datasets to verify the effectiveness of the method.

questions:
1. Different methods are needed to illustrate the validity of the new dataset.

2. More groups of examples and different methods are needed to illustrate fairness.

3. Different datasets are needed to verify the validity of methods.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730693958191

thread_type:
official_review

<thread_object>
creator: reviewer_vqbs

<content>
summary:
The paper introduces a new set of datasets called FAIR-EARTH, aimed at addressing and assessing fairness in Earth science. FAIR-EARTH encompasses various types of data, including environmental factors and emissions, while also taking geographical features and population density into account. Experimental results demonstrate that existing spherical harmonics methods fail to maintain accurate local performance, particularly regarding land and sea masses. To address this issue, the authors propose using spherical Morlet wavelets to correct localized biases. Results indicate that this approach outperforms existing methods in most situations.

strengths:
S1. The bias issues in earth science are interesting and important.  
S2. The paper is well-written and easy to understand, even for those who are not familiar with earth or geographical concepts.  
S3. The experiments provide detailed insights into model behavior and validate the effectiveness of spherical wavelets in improving representation fairness.

weaknesses:
W1. The performance trade-offs of spherical wavelets are insufficiently addressed.
W2. The computation efficiency of spherical wavelets is not discussed.
W3. More discussion regarding the flexibility and weakness of the proposed datasets.

questions:
1. While spherical wavelets show good performance on most cases to correct localized biases, their robustness remains uncertain. It is known that correctly parameterizing elements like wave number and scaling factors for specific resolutions are necessary to reach the best performance. The authors should conduct experiments to illustrate the trade-offs associated with using spherical wavelets.

2. Since spherical wavelet is well-defined at poles, what should we do for poles? Would there exist bias in poles?

3. Implementing spherical wavelet encodings requires a more complex parameter selection and may need more computation. Efficiency evaluation should be considered for using spherical wavelets

4. While FAIR-EARTH can address some bias for the ill-represented areas like landmasses, authors should detail the weakness/flexibility of this combination approach. What kind of dataset can be integrated if there is temporal bias or learning representation for poles?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732257754031

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response from Authors to All Reviewers

comment:
# Response to Reviewers

We would like to thank all the reviewers for their careful reading and insightful comments. First, we would like to briefly summarize the core contributions of our submission on Implicit Neural Representations (INRs) for Earth data:

## Core Contributions

### 1. **We developed a novel aggregated-dataset and evaluation framework (FAIR-Earth) for implicit Earth representations**, with the primary goal of assessing fairness issues of state-of-the-art AI systems. By open-sourcing this novel framework and its associated Python library, we aim to improve fairness in future research. Reviewers deemed this contribution "critical and underexplored" and "interesting and important" (R1, R3), and to be a "comprehensive framework" (R3). We also appreciate the acknowledgment of our dataset's value in "aggregating extensive metadata along stratifications" (R2) and providing a foundation for "future research" (R3).

### 2. **We leveraged FAIR-Earth to assess the fairness of various SOTA representations.** This analysis showcased strong limitations in current methods, such as models that accurately predict climate data of large continents at the expense of islands and coastal areas. Reviewers praised our work for providing "detailed insights into model behavior" (R1) and a careful analysis of representation biases across different subgroups (R2).

### 3. **We proposed a new filter-bank encoding for INRs leveraging wavelet methods instead of Fourier.** Our analysis on existing models identified a key conflict between Fourier-basis encodings (which assume global stationary signals) and the local variations exhibited by many Earth signals (e.g., high gradient near coastal areas or around islands). To address this, we propose to leverage wavelet-based encodings, which are known to provide better signal decomposition in those cases. Reviewers found this technical contribution to be a theoretically "well-motivated" solution (R3), and appreciated how Spherical Wavelet improved representation fairness while "maintaining competitive performance" (R2).

## Additional Feedback

The reviewers found our presentation to be **"excellent"** (R3) and **"well-written and easy to understand"** (R1).

## Revisions and Additions

Based on reviewer feedback, we have conducted numerous novel experiments and revised the manuscript. We provide a summary of those additions and changes here, with more details in each per-reviewer answer:

1. **Broader empirical evaluation**:
We perform substantially expanded cross-validation and fine-tuning experiments, involving evaluation on three new SOTA location encodings, two new external datasets, and downstream fairness metadata (A.5: Extended Experimental Results)

2. **Implementation specifications**: We update A.2: Training Specifications to reflect changes to the above experiments. Additionally, we detail in Appendix A.6: Wavelet Notes a detailed examination of our Spherical Wavelet construction.

3. **Computational complexity analysis**: Also available in A.6: Wavelet Notes is thorough analytical and empirical evidence that Spherical Wavelet is more scalable and stable than its Spherical Harmonic counterpart.

---

*Changes are marked in blue in the revised paper. We remain available during this period for any further questions.*
</content>
</thread_object>

</thread>
<thread>
creation date:
1732533958760

thread_type:
official_comment

<thread_object>
creator: area_chair_botj

<content>
title:
Acknowledge the author responses

comment:
Dear Reviewers,

Thank you very much for your effort. As the discussion period is coming to an end, please acknowledge the author responses and adjust the rating if necessary.

Sincerely,
AC
</content>
</thread_object>

</thread>
<thread>
creation date:
1732582947279

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Follow-Up From the Authors

comment:
Dear reviewers, 

As we near the end of the discussion period, we would be grateful to hear about any remaining concern(s) and/or question(s) regarding our submission. We strongly believe that your review and our answers have made our submission stronger, but if any concern remains, we would love to carry out additional experiments and/or provide further clarifications. 

The authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1732797732503

thread_type:
official_comment

<thread_object>
creator: area_chair_botj

<content>
title:
Discussion needed

comment:
Dear Reviewers,

As you are aware, the discussion period has been extended until December 2. Therefore, I strongly urge you to participate in the discussion as soon as possible if you have not yet had the opportunity to read the authors' response and engage in a discussion with them. Thank you very much.

Sincerely,
Area Chair
</content>
</thread_object>

</thread>
<thread>
creation date:
1733021948993

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Answer

comment:
Dear Area Chair and Reviewers,

We are grateful for all the thoughtful reviews and constructive feedback provided. We note that two reviewers who initially rated our submission favorably (score of 6) have explicitly acknowledged our responses and confirmed their positive assessment. Specifically, both Reviewer VqBs and Reviewer Jrro stated they were "happy with the authors response" and will "maintain [their] current rating", indicating that **our extensive revisions and additional experiments successfully addressed their minor concerns about performance trade-offs and computational complexity**.

While Reviewer FbST (score of 5) has not yet acknowledged our rebuttal, we believe **we have comprehensively addressed their core concerns through our substantially expanded experimental evaluation**, which now includes: (1) validation against three additional state-of-the-art baselines, (2) extensive subgroup analysis across multiple modalities and datasets, and (3) thorough examination on external benchmark datasets. **The positive feedback from other reviewers regarding these additions reinforces our belief that we have thoroughly addressed these concerns**.

Lastly, we reiterate that our submission represents a significant step forward in the critical task of uncovering and addressing biases of AI applications in Earth science. Many of the hidden biases we've identified - such as systematic errors in modeling islands and coastal regions - may yield catastrophic real-world consequences if left unaddressed. The strong interest from the reviewers affirms that this is a timely contribution to an emerging and crucial field. **We believe that presenting this work at ICLR would spur important follow-up research and catalyze the needed development of more equitable AI systems for Earth science applications**.

The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper develops a novel aggregated-dataset and evaluation framework (FAIR-Earth) for implicit Earth representations.  The reviewers acknowledged the contribution of the new dataset and the novelty of the problem setting.  I agree with the reviewers that the problem is important and underexplored.  Nevertheless, the reviewers raised reasonable concerns mainly on insufficient evaluation (datasets) and lack of technical depth.  The authors successfully addressed part of the reviewers' concerns, and I admit that this work can trigger a new research direction.

This paper is indeed a borderline paper.  Because of no strong negative opinion and some potential for a new research direction, I now recommend an accept for this paper.  However, in my batch, there are quite sufficient papers which are strongly supported by the reviewers.  Thus, even though I recommend an accept now, the senior AC can change my recommendation to a reject.