SlowGuess/NeurIPS_2023_OCR huggingface dataset:
OCEANBENCH: The Sea Surface Height Edition
J. Emmanuel Johnson*
CNRS UMR IGE
johnsonj@univ-grenoble-alpes.fr
Quentin Febvre*
IMT Atlantique
quentin.febvre@imt-atlantique.fr
Sammy Metref
DATLAS
Maxime Ballarotta
Julien Le Sommer
Ronan Fablet
The ocean is a crucial component of the Earth's system. It profoundly influences human activities and plays a critical role in climate regulation. Our understanding has significantly improved over the last decades with the advent of satellite remote sensing data, allowing us to capture essential sea surface quantities over the globe, e.g., sea surface height (SSH). Despite their ever-increasing abundance, ocean satellite data presents challenges for information extraction due to their sparsity and irregular sampling, signal complexity, and noise. Machine learning (ML) techniques have demonstrated their capabilities in dealing with large-scale, complex signals. Therefore we see an opportunity for these ML models to harness the full extent of the information contained in ocean satellite data. However, data representation and relevant evaluation metrics can be the defining factors when determining the success of applied ML. The processing steps from the raw observation data to a ML-ready state and from model outputs to interpretable quantities require domain expertise, which can be a significant barrier to entry for ML researchers. In addition, imposing fixed processing steps, like committing to specific variables, regions, and geometries, will narrow the scope of ML models and their potential impact on real-world applications. OceanBench is a unifying framework that provides standardized processing steps that comply with domain-expert standards. It is designed with a flexible and pedagogical abstraction: it a) provides plug-and-play data and pre-configured pipelines for ML researchers to benchmark their models w.r.t. ML and domain-related baselines and b) provides a transparent and configurable framework for researchers to customize and extend the pipeline for their tasks. In this work, we demonstrate the OceanBench framework through a first edition dedicated to SSH interpolation challenges. We provide datasets and ML-ready benchmarking pipelines for the long-standing problem of interpolating observations from simulated ocean satellite data, multi-modal and multi-sensor fusion issues, and transfer-learning to real ocean satellite observations. The OceanBench framework is available at github.com/jejjohnson/oceanbench and the dataset registry is available at github.com/quentinf00/oceanbench-data-registry.
1 Motivation
The ocean is vital to the Earth's system [28]. It plays a significant role in climate regulation regarding carbon [40] and heat uptake [87]. It is also a primary driver of human activities (e.g., maritime traffic and world trade, marine resources and services) [106, 93]. However, monitoring the ocean is a critical challenge: the ocean state can only partially be determined because most of the ocean consists of subsurface quantities that we cannot directly observe. Thus, to quantify even a fraction of the physical or biochemical ocean state, we must often rely only on surface quantities that we can monitor from space, drifting buoys, or autonomous devices. Satellite remote sensing, in particular, is one of the most effective ways of measuring essential sea surface quantities [2] such as sea surface height (SSH) [95], sea surface temperature (SST) [77], and ocean color (OC) [53]. While these variables characterize only a tiny portion of the ocean ecosystem, they present a gateway to many other derived physical quantities [93].
Although we can access observable sea surface quantities, they are generally irregularly and extremely sparsely sampled. For instance, satellite-derived SSH data has less than  $5\%$  coverage of the globe daily [95]. These sampling gaps make the characterization of ocean processes highly challenging for operational products and downstream tasks that depend on relevant gap-free variables. This has motivated a rich literature in geoscience over the last decades, mainly using geostatistical kriging methods [95, 102] and model-driven data assimilation schemes [55, 60]. Despite significant progress, these schemes often need to improve their ability to leverage available observation datasets' potential fully. This has naturally advocated for exploring data-driven approaches like shallow ML schemes [7, 6, 97, 71]. Very recently, deep learning schemes [116, 74, 9] have become appealing solutions to benefit from existing large-scale observation and simulation datasets and reach significant breakthroughs in the monitoring of upper ocean dynamics from scarcely and irregularly sampled observations. However, the heterogeneity and characteristics of the observation data present major challenges for effectively applying these methods beyond idealized case studies. A data source could have different variables, geometries, and noise levels, resulting in many domain-specific preprocessing procedures that can vastly change the solution outcome. Furthermore, the evaluation procedure of the methods and their effectiveness can be regionally-dependent as the physical phenomena vary in space and time, which adds another layer of complexity in convincing domain scientists of their trustworthiness. So the entire ML pipeline now requires a unified framework for dealing with heterogeneous data sources, different pre- and post-processing methodologies, and regionally-dependent evaluation procedures.
To address these challenges, we introduce OceanBench, a framework for co-designing machine-learning-driven high-level experiments from ocean observations. It consists of an end-to-end framework for piping data from its raw form to an ML-ready state and from model outputs to interpretable quantities. We regard OceanBench as a key facilitator for the uptake of MLOPs tools and research [66, 94] for ocean-related datasets and case studies. This first edition provides datasets and ML-ready benchmarking pipelines for SSH interpolation problems, an essential topic for the space oceanography community, related to ML communities dealing with issues like in-painting [111], denoising [99, 98], and super-resolution [107]. We expect OceanBench to facilitate new challenges to the applied machine learning community and contribute to meaningful ocean-relevant breakthroughs. The remainder of the paper is organized as follows: in §2, we outline some related work that was inspirational for this work; in §3, we formally outline OceanBench by highlighting the target audience, code structure, and problem scope; in §4, we outline the problem formulation of SSH interpolation and provide some insight into different tasks related to SSH interpolation where OceanBench could provide some helpful utility; and in §5 we give some concluding remarks while also informally inviting other researchers to help fill in the gaps.
Machine learning applied to geosciences is becoming increasingly popular, but there are few examples of transparent pipelines involving observation data. After a thorough literature review, we have divided the field into three camps of ML applications that pertain to this work: 1) toy simulation datasets, 2) reanalysis datasets, and 3) observation datasets. We outline the literature for each of the three categories below.
Figure 1: This figure showcases the OceanBench toolset. We have 1) OceanBench-Data-Registry which uses DVC to store and organize ML-ready datasets, 2) Ocean-Tools which features a suite of task-relevant geoprocessing functions with the xarray-backend, and 3) xrpatcher which can produce arbitrary subsets of xarray datastructures which nicely interface with dataloaders such as PyTorch. OceanBench provides an interface for ML researchers to parameterize arbitrary sequences of transformations to preprocess data from a domain-specific state to a ML-ready state.
Toy Simulation Data. One set of benchmarks focuses on learning surrogate models for well-defined but chaotic dynamical systems in the form of ordinary differential equations (ODEs) and partial differential equations (PDEs) and there are freely available code bases which implement different ODEs/PDEs [52, 96, 3, 64, 8, 103, 56, 85]. This is a great testing ground for simple toy problems that better mimic the structures we see in real-world observations. Working with simulated data is excellent because it is logistically simple and allows users to test their ideas on toy problems without increasing the complexity when dealing with real-world data. However, these are ultimately simple physical models that often do not reflect the authentic structures we see in real-world, observed data.
Reanalysis Data. This is assimilated data of real observations and model simulations. There are a few major platforms that host ocean reanalysis data like the Copernicus Marine Data Store [36, 33, 34, 37], the Climate Data Store [25], the BRAN2020 Model [26], and the NOAA platform [15]. However, to our knowledge, there is no standard ML-specific ocean-related tasks to accompany the data. On the atmospheric side, platforms like WeatherBench [86], ClimateBench [108], ENS10 [10] were designed to assess short-term and medium-term forecasting using ML techniques with recent success of ML [69, 84]. The clarity of the challenges set by the benchmark suites has inspired the idea of OceanBench, where we directly focus on problems dealing with ocean observation data.
Observation Data. These observation datasets (typically sparse) stem from satellite observations that measure surface variables or in-situ measurements that measure quantities within the water column. Some major platforms to host data include the Marine Data Store [32, 31], the Climate Data Store [23, 24, 22], ARGO [110], and the SOCAT platform [11]. However, it is more difficult to assess the efficacy of operational ML methods that have been trained only on observation data and, to our knowledge, there is no coherent ML benchmarking system for ocean state estimation. There has been significant effort by the Ocean-Data-Challenge Group which provides an extensive suite of datasets and metrics for SSH interpolation. Their efforts heavily inspired our work, and we hope that OceanBench can build upon their work by adding cohesion and facilitating the ease of use for ML research and providing a high-level framework for providing ML-related data products.
3 OceanBench
3.1 Why OceanBench?
There is a high barrier to entry in working with ocean observations for researchers in applied machine learning as there are many processing steps for both the observation data and the domain-specific
<sup>1</sup>Ocean Data Challenge group: Freely associated scientist for oceanographic algorithm and product improvements (ocean-data-challenges.github.io)
evaluation procedures. OceanBench aims to lower the barrier to entry cost for ML researchers to make meaningful progress in the field of state prediction. We distribute a standardized, transparent, and flexible procedure for defining data and evaluation pipelines for data-intensive geoscience applications. Proposed examples and case studies provide a plug-and-play framework to benchmark novel ML schemes w.r.t. state-of-the-art, domain-specific ML baselines (see figure 1). In addition, we adopt a pedagogical abstraction that allows users to customize and extend the pipelines for their specific tasks. To our knowledge, no framework embeds processing steps for earth observation data in a manner compatible with MLOps abstractions and standards regarding reproducibility and evaluation procedures. Ultimately, we aim to facilitate the uptake of ML schemes to address ocean observation challenges and to bring new challenges to the ML community to extend additional ML tools and methods for irregularly-sampled and partially-observed high-dimensional space-time dynamics. The abstractions proposed here apply beyond ocean sciences and SSH interpolation to other geosciences with similar tasks that intersect with machine learning.
3.2 Code Structure
OceanBench is lightweight in terms of the core functionality. We keep the code base simple and focus more on how the user can combine each piece. We adopt a strict functional style because it is easier to maintain and combine sequential transformations. There are five features we would like to highlight about OceanBench: 1) Data availability and version control, 2) an agnostic suite of geoprocessing tools for xarray datasets that were aggregated from different sources, 3) Hydra integration to pipe sequential transformations, 4) a flexible multi-dimensional array generator from xarray datasets that are compatible with common deep learning (DL) frameworks, and 5) a JupyterBook [38] that offers library tutorials and demonstrates use-cases. In the following section, we highlight these components in more detail.
OceanBench-Data-Registry. The most important aspect is the public availability of the datasets. We aggregate all pre-curated datasets from other sources, e.g. the Ocean-Data-Challenge [13, 12], and organize them to be publicly available from a single source. We also offer a few derived datasets which can be used for demonstrations and evaluation. Data is never static in a pipeline setting, as one can have many derived datasets which stem from numerous preprocessing choices. In fact, in research, we often work with derived datasets that have already been through some preliminary preprocessing methods. To facilitate the ever-changing nature of data, we use the Data Version Control (DVC) tool [67], which offers a git-like version control of the datasets.
Ocean-Tools The Ocean-Tools library uses a core suite of functions specific to processing geo-centric data. While a few particular functionalities vary from domain to domain, many operations are standard, e.g., data variable selections, filtering/smoothing, regridding, coordinate transformations, and standardization. We almost work exclusively with the xarray [58] framework because it is a coordinate-aware, flexible data structure. In addition, the geoscience community has an extensive suite of specialized packages that operate in the xarray framework to accomplish many different tasks. Almost all Ocean-Tools toolsets are exclusively within the xarray framework to maintain compatibility with a large suite of tools already available from the community.
Hydra Integration. As discussed above, many specific packages accomplish many different tasks. However, what needs to be added is the flexibility to mix and match these operations as the users see fit. Hydra [112] and Hydra-Zen [92] provide a configurable way to aggregate and pipe many sequential operations together. It also maintains readability, robustness, and flexibility through the use of .yaml files which explicitly highlights the function used, the function parameters chosen, and the sequence of operations performed. In the ML software stack, Hydra is often used to manage the model, optimizer, and loss configurations which helps the user experiment with different options. We apply this same concept in preprocessing, geoprocessing, and evaluation steps, often more important than the model configuration in geoscience-related tasks.
XRPatcher Every machine learning pipeline will inevitably require moving data from the geospecific data structure to a multi-dimensional array easily digestible for ML models. A rather underrated, yet critical, feature of ML frameworks such as PyTorch [83] (Lightning [45]) and
Available at:quentinf00/oceanbench-data-registry
Available at: jejjohnson/ocn-tools
4 Available at: jejjohnson/xrpatcher
TensorFlow [1] (Keras [30]) is the abstraction of the dataset, dataloader, datamodules, and data pipelines. In applied ML in geosciences, the data pipelines are often more important than the actual model [89]. The user can control the patch-size and the stride-step, which can generate arbitrary coordinate-aware items directly from the xarray data structure. In addition, XRPatcher provides a way to reconstruct the fields from an arbitrary patch configuration. This robust reconstruction step is convenient to extend the ML inference step where one can reconstruct entire fields of arbitrary dimensions beyond the training configuration, e.g., to account for the border effects within the field (see appendix E) or to reconstruct quantities in specific regions or globally.
JupyterBook. Building a set of tools is relatively straightforward; however, ensuring that it sees a broader adoption across a multi-disciplinary community is much more challenging. We invested heavily in showing use cases that appeal to different users with the JupyterBook platform [38]. Code with context is imperative for domain and ML experts as we need to explain and justify each component and give many examples of how they can be used in other situations. Thus, we have paid special attention to providing an extensive suite of tutorials, and we also highlight use cases for how one can effectively use the tools.
3.3 Problem Scope
There are many problems that are of great interest the ocean community [29] but we limit the scope to state estimation problems [21]. Under this scope, there are research questions that are relevant to operational centers which are responsible for generating the vast majority of global ocean state maps [36, 34, 33, 37] that are subsequently used for many downstream tasks [93]. For example: how can we effectively use heterogeneous observations to predict the ocean state on the sea surface [55, 62, 102, 44, 14, 77]; how can we incorporate prior physics knowledge into our predictions of ocean state trajectories [55, 29, 93]; and how can we use the current ocean state at time  $T$  to predict the future ocean state at time  $T + \tau$  [42, 86, 16]. In the same vain, there are more research questions that are of interest to the academic modeling community. For example: is simulated or reanalysis data more effective for learning ML emulators that replace expensive ocean models [49, 114]; what metrics are more effective for assessing our ability to mimic ocean dynamics [75, 48]; and how much model error can we characterize when learning from observations [18, 68].
We have cited many potential applications of how ML can be applied to tackle the state estimation problem. However, to our knowledge there is no publicly available, standardized benchmark system that is caters to ML-research standards. We believe that, irrespective of the questions posed above and the data we access, there are many logistical similarities for each of the problem formulations where we can start to set standards for a subset of tasks like interpolation or forecasting. On the front-end, we need a way to select regions, periods, variables, and a valid train-test split (see sec. D.1). On the back-end, we need a way to transform the predictions into more meaningful variables with appropriate metrics for validation (see sec. D.2 and D.3). OceanBench was designed to be an agnostic tool that is extensible to the types of datasets, processing techniques and metrics needed for working with a specific class of Ocean-related datasets. We strongly feel that a suite like this is the first step in designing task-specific benchmarks within the ocean community that is compatible with ML standards. In the remainder of the paper, we will demonstrate how OceanBench can be configured to facilitate a ML-ready data challenge involving our first edition to demonstrate OceanBench's applicability: sea surface height interpolation.
4 Sea Surface Height Edition
Sea surface height (SSH) is one of the most critical, observable quantities when determining the ocean state. It is widely used to study ocean dynamics and the adverse impact on global climate and human activities [78]. SSH enables us to track phenomena such as currents and eddies [78, 27, 82], which leads to a better quantification of the transport of energy, heat, and salt. In addition, SSH helps us quantify sea level rise at regional and global scales [4, 39], which is used for operational monitoring of the marine environment [106]. Furthermore, SSH characterization provides a plethora of data products that downstream tasks can use for many other applications [79, 20]. Due to the irregular sampling delivered by satellite altimeter, state-of-the-art operational methods using optimal interpolation schemes [95, 102] or model-driven data assimilation [7, 6, 71, 97] fail to fully retrieve SSH dynamics at fine scales below  $100 - 200\mathrm{km}$  on a global or regional scale, so improving the
space-time resolution of SSH fields has been a critical challenge in ocean science. Beyond some technological developments [51], recent studies support the critical role of ML-based schemes in overcoming the current limitations of the operational systems [14, 55, 116]. The rest of this section gives an overview of the general problem definition for SSH interpolation, followed by a brief ontology for ML approaches to address the problem. We also give an overview of some experimental designs and datasets with a demonstration of metrics and plots generated by the OceanBench platform.
4.1 Problem Definition
We are dealing with satellite observations, so we are interested in the domain across the Earth's surface. Let us define the Earth's domain by some spatial coordinates,  $\mathbf{x} = [\mathrm{Longitude},\mathrm{Latitude}]^{\top}\in \mathbb{R}^{D_s}$ , and temporal coordinates,  $t = [\mathrm{Time}]\in \mathbb{R}^{+}$ , where  $D_{s}$  is the dimensionality of the coordinate vector. We can define some spatial (sub-)domain,  $\Omega \subseteq \mathbb{R}^{D_s}$ , and a temporal (sub-)domain,  $\mathcal{T}\subseteq \mathbb{R}^{+}$ . This domain could be the entire globe for 10 years or a small region within the North Atlantic for 1 year.
$$
\text {S p a t i a l C o o r d i n a t e s :} \quad \mathbf {x} \in \Omega \subseteq \mathbb {R} ^ {D _ {s}} \tag {1}
$$
$$
\text {T e m p o r a l C o o r d i n a t e s :} \quad t \in \mathcal {T} \subseteq \mathbb {R} ^ {+}. \tag {2}
$$
In this case  $D_{s} = 2$  because we only have a two coordinates, however we can do some coordinate transformations like spherical to Cartesian. Likewise, we can do some coordinate transformation for the temporal coordinates like cyclic transformations or sinusoidal embeddings [105]. We have two fields of interest from these spatiotemporal coordinates: the state and the observations.
$$
\text {S t a t e}: \quad \boldsymbol {u} (\mathbf {x}, t): \Omega \times \mathcal {T} \rightarrow \mathbb {R} ^ {D _ {u}} \tag {3}
$$
$$
\text {O b s e r v a t i o n s}: \quad \boldsymbol {y} _ {\text {o b s}} (\mathbf {x}, t): \Omega \times \mathcal {T} \rightarrow \mathbb {R} ^ {D _ {\text {o b s}}} \tag {4}
$$
The state domain,  $u \in \mathcal{U}$ , is a scalar or vector-valued field of size  $D_u$  which is typically the quantity of interest and the observation domain,  $y_{obs} \in \mathcal{Y}_{obs}$ , is the observable quantity which is also a scalar or vector-valued field of size  $D_{obs}$ . Now, we make the assumption that we have an operator  $\mathcal{H}$  that transforms the field from the state space,  $\pmb{u}$ , to the observation space,  $\pmb{y}_{obs}$ .
$$
\boldsymbol {y} _ {\text {o b s}} (\mathbf {x}, t) = \mathcal {H} \left(\boldsymbol {u} (\mathbf {x}, t), t, \varepsilon , \boldsymbol {\mu}\right) \tag {5}
$$