SlowGuess/NeurIPS_2023_OCR huggingface dataset:
This equation is the continuous function defined over the entire spatiotemporal domain. The operator,  $\mathcal{H}(\cdot)$ , is flexible and problem dependent. For example, in a some discretized setting there are 0's wherever there are no observations, and 1's wherever there are observations, and in other discretized settings it takes a weighted average of the neighboring pixels. We also include a generic noise function,  $\varepsilon (\mathbf{x},t)$ . This could stem from a distribution, it could stationary noise operator,  $\varepsilon (\mathbf{x})$ , or it could be constant in space but vary with Time,  $\varepsilon (t)$ . We also include a control parameter,  $\mu$ , representing any external factors or latent variables that could connect the state vector to the observation vector, e.g., sea surface temperature. Our quantity of interest is SSH,  $\eta$ , a scalar-valued field defined everywhere on the domain. In our application, we assume that the SSH we observe from satellite altimeters,  $\eta_{obs}$ , is the same as the SSH state, except it could be missing for some coordinates due to incomplete coverage from the satellite. So our transformation is defined as follows:
$$
\eta_ {o b s} (\mathbf {x}, t) = \mathcal {H} \left(\boldsymbol {\eta} (\mathbf {x}, t), t, \varepsilon , \boldsymbol {\mu}\right) \tag {6}
$$
In practice, the satellite providers have a reasonable estimation of the amount of structured noise level we can expect from the satellite altimetry data; however, unresolved noise could still be present. Finally, we are interested in finding some model,  $\mathcal{M}$ , that maps the SSH we observe to the true SSH given by
$$
\mathcal {M}: \boldsymbol {\eta} _ {\text {o b s}} (\mathbf {x}, t, \boldsymbol {\mu}) \rightarrow \boldsymbol {\eta} (\mathbf {x}, t), \tag {7}
$$
which is essentially an inverse problem that maps the observations to the state. One could think of it as trying to find the inverse operator,  $\mathcal{M} = \mathcal{H}^{-1}$ , but this could be some other arbitrary operator.
4.2 Machine Learning Model Ontology
In general, we are interested in finding some parameterized operator,  $\mathcal{M}_{\theta}$ , that maps the incomplete SSH field to the complete SSH field
$$
\mathcal {M} _ {\boldsymbol {\theta}}: \boldsymbol {\eta} _ {o b s} (\mathbf {x}, t, \boldsymbol {\mu}) \rightarrow \boldsymbol {\eta} (\mathbf {x}, t), \tag {8}
$$
whereby we learn the parameters from data. The two main tasks we can define from this problem setup are 1) interpolation and 2) extrapolation. We define interpolation as the case when the boundaries of the inferred state domain lie within a predefined shape for the boundaries of the spatiotemporal observation domain. For example, the shape of the spatial domain could be a line, box, or sphere, and the shape of the temporal domain could be a positive real number line. We define extrapolation as the case where the boundaries of the inferred state domain are outside the boundaries of the spatiotemporal observation domain. In this case, the inferred state domain could be outside of either domain or both. A prevalent specific case of extrapolation is hindcasting or forecasting, where the inferred state domain lies within the spatial observation domain's boundaries but outside of the temporal observation domain's. In the rest of this paper, we will look exclusively at the interpolation problem. However, we refer the reader to appendix  $\boxed{F}$  for a more detailed look at other subtasks that can arise.
From a ML point of view, we can explore various ways to define the operator in equation (7). We may distinguish three main categories: (i) coordinate-based methods that learn a parameterized continuous function to map the domain coordinates to the scalar values, (ii) the explicit mapping of the state from the observation, (iii) implicit methods defined as the solution of an optimization problem. The first category comprises of kriging approaches, which have been used operationally with historical success [113, 95]. Beyond such covariance-based approaches, recent contributions explore more complex trainable functional models [72], basis functions [102], and neural networks [62]. The second category of schemes bypasses the physical modeling aspect and amortizes the prediction directly using state-of-the-art neural architectures such as UNets and ConvLSTMs [116, 74, 9]. This category may straightforwardly benefit from available auxiliary observations [23, 24, 22] to state the interpolation problem as a super-resolution [107] or image-to-image translation problem [81, 59]. The third category relates to inverse problem formulations and associated deep learning schemes, for example deep unfolding methods and plug-and-play priors [115]. Interestingly, recent contributions explore novel neural schemes which combine data assimilation formulations [21] and learned optimizer strategies [14, 44]. We provide a more detailed ontology of methods used for interpolation problems in appendix G. We consider at least one baseline approach from each category for each data challenge described in section 4.4. While all these methods have pros and cons, we expect the OceanBench platform to showcase to new experimental evidence and understanding regarding their applicability to SSH interpolation problems.
4.3 Experimental Design
Table 1: This table gives a brief overview of the datasets provided to complete the data challenges listed in 4.4 and A. Note that the OSSE datasets are all gridded products whereas the OSE NADIR is an alongtrack product. See figure 2 for an example of the OSSE NEMO Simulations for SSH and SST and pseudo-observations for NADIR & SWOT.
OSSE	OSSE NADIR + SWOT	OSSE SST	OSE NADIR
Data Type	Simulations	Pseudo-Observations	Simulations	Observations
Source	NEMO [5]	NEMO [5]	NEMO [5]	Altimetry [32]
Region	GulfStream	GulfStream	GulfStream	GulfStream
Domain Size	10 × 10°	10 × 10°	10 × 10°	10 × 10°
Longitude Extent	[−65°, −55°]	[−65°, −55°]	[−65°, −55°]	[−65°, −55°]
Latitude Extent	[33°, 43°]	[33°, 43°]	[33°, 43°]	[33°, 43°]
Resolution	0.05° × 0.05°	0.05° × 0.05°	0.05° × 0.05°	7 km
Grid Size	200 × 200	200 × 200	200 × 200	N/A
Num Datapoints	~14.6M	~14.6M	~14.6M	~1.6M
Period Start	2012-10-01	2012-10-01	2012-10-01	2016-12-01
Period End	2013-09-30	2013-09-30	2013-09-30	2018-01-31
Frequency	Daily	Daily	Daily	1 Hz
The availability of multi-year simulation and observation datasets naturally advocates for the design of synthetic (or twin) experiments, referred to as observing system simulation experiments (OSSE), and of real-world experiments, referred to as observing system experiments (OSE). We outline these two experimental setups below.
Observing System Simulation Experiments (OSSE). A staple and groundtruthed experimental setup uses a reference simulation dataset to simulate the conditions we can expect from actual satellite observations. This setup allows researchers and operational centers to create a fully-fledged pipeline that mirrors the real-world experimental setting. An ocean model simulation is deployed over a specified spatial domain and period, and a satellite observation simulator is deployed to simulate satellite observations over the same domain and period. This OSSE setup has primarily been considered for performance evaluation, as one can assess a reconstruction performance over the entire space-time domain. It also provides the basis for the implementation of classic supervised learning strategies [9, 74, 116]. The domain expert can vary the experimental conditions depending on the research question. For example, one could specify a region based on the expected dynamical regime [12] or add a certain noise level to the observation tracks based on the satellite specifications. The biggest downside to OSSE experiments is that we train models exclusively with ocean simulations which could produce models that fail to generalize to the actual ocean state. Furthermore, the simulations are often quite expensive, which prevents the community from having high spatial resolution over very long periods, which would be essential to capture as many dynamical regimes as possible.
Observing System Experiments (OSE). As more observations have become available over the past few decades, we can also design experiments using real data. This involves aggregating as many observations from real ocean altimetry satellites as possible with some specific independent subset left out for evaluation purposes. A major downside to OSE experiments is that the sparsity and spatial coverage of the observations narrow the possible scope of performance metrics and make it very challenging to learn directly from observation datasets. The current standard altimetry data are high resolution but cover a tiny area. As such, it can only inform fine-scale SSH patterns in the along-track satellite direction and cannot explicitly reveal two-dimensional patterns. Despite these drawbacks, it provides a quantitative evaluation of the generalizability of the ML methods concerning the true ocean state.
4.4 Data Challenges
We rely on existing OSSE and OSE experiments for SSH interpolation designed by domain experts [13] and recast them into OceanBench framework to deliver a ML-ready benchmarking suites. The selected data challenges for this first edition address SSH interpolation for a  $1000\mathrm{km}\times 1000\mathrm{km}$  Gulfstream region. We briefly outline them below.
Experiment I (OSSE NADIR) addresses SSH interpolation using NADIR altimetry tracks which are very fine, thin ocean satellite observations (see Figure 2). It relies on an OSSE using high-resolution  $(1/60^{\circ}$  resolution) ocean simulations generated by the NEMO model over one year with a whole field every day.
Experiment II (OSSE SWOT) addresses SSH interpolation using jointly NADIR and SWOT altimetry data where we complement the OSSE NADIR configuration with simulated SWOT observations. SWOT is a new satellite altimetry mission with a much higher spatial coverage but a much lower temporal resolution as illustrated in Figure 2. The higher spatial resolution allows us to see structures at a smaller resolution but at the cost of a massive influx of observations (over  $\times 100$ ).
Experiment III (OSSE SST) addresses SSH interpolation using altimetry and SST satellite data jointly. We complement the OSSE SWOT challenge with simulated SST observations. Satellite-derived SST observations are more abundantly available in natural operational settings than SSH at a finer resolution, and structures have visible similarities [51, 55]. So this challenge allows for methods to take advantage of multi-modal learning [44, 116].
Experiment IV (OSE NADIR) addresses SSH interpolation for real NADIR altimetry data. In contrast to the three OSSE data challenges, it only looks at actual observations aggregated from the currently available ocean altimetry data from actual satellites. It involves a similar space-time sampling as Experiment (OSSE NADIR) to evaluate the generalization of ML methods trained in Experiment I to real altimetry data. The training problem's complexity increases significantly due to the reference dataset's sparsity compared with the OSSE NADIR dataset. One may also explore transfer learning or fine-tuning strategies from the available OSSE dataset.
NADIR Altimetry Tracks
SWOT Altimetry Tracks
NEMO Simulation
MIOST
BFNOG
4DVarNet
(a)
Figure 2: A snapshot at  $27^{th}$  October, 2012 of the sea level anomaly (SLA) from the NEMO simulation for the OSSE experiment outlined in section 4.3. The top row showcases the aggregated NADIR altimetry tracks and the aggregated SWOT altimetry tracks (12 hours before and 12 hours after) as well as the SST from the NEMO simulation. Each subsequent row showcases the following physical variables found in appendix B: (a) Sea Level Anomaly, (b) Kinetic Energy, (c) Relative Vorticity, and (d) Strain. Each column in the subsequent rows showcase the following reconstructed field from the NEMO simulation found in column n (a): (b) MIOST [102], (c) BFN-QG [55], and (d) SDVarNet [14].
4.5 OceanBench Pipelines
For the four data challenges presented in the previous section, we used OceanBench pipelines to deliver a ML-ready benchmarking framework. We used the hydra and the geoprocessing tools outlined in section 3.2 with specialized routines for regridding the ocean satellite data to a uniformly gridded product and vice versa when necessary. Appendix D showcases an example of the hydra integration for the preprocessing pipeline. A key feature is the creation of a custom patcher for the appropriate geophysical variables using our XRPatcher tool, which is later integrated into custom datasets and dataloaders for the appropriate model architecture, e.g., coordinate-based or grid-based.
(a) Normalized RMSE
(b) Isotropic Power Spectrum
(c) Isotropic Power Spectrum Score
(d) NEMO Simulation
(e) MIOST
(f) BFN-QG
(g) 4DVarNet
Figure 3: This figure showcases some statistics for evaluation of the SSH field reconstructions for the OSSE NADIR experiment outlined in section 4. Subfigure (a) showcases the normalized root mean squared error (nRMSE), (b) showcases the isotropic power spectrum decomposition (PSD), (c) showcases isotropic PSD scores. The bottom row showcases the space-time PSD for the NEMO simulation (subfigure (d)) and the PSD scores for three reconstruction models: (e) the MIOST model [102], (f) the BFN-QG model [55], and (g) the 4DVarNet model [14].
Table 2: This table highlights some of the results for the OSSE NADIR experiment outlined in section 4.4 and appendix A. This table highlights the performance statistically in the real and spectral space; the normalized RMSE score for the real space and the minimum spatial and temporal scales resolved in the spectral domain. For more information about the class of models displayed and class of metrics, see appendix G and appendix C respectively. We only showcase the model performance on the alongtrack NADIR data available. For the extended table for each of the challenges, see Table 3
Experiment	Algorithm	Algorithm Class	nRMSE Score	λx[km]	λt[days]
OSSE NADIR	OI [95]	Coordinate-Based	0.92 ± 0.01	175	10.8
OSSE NADIR	MIOST [102]	Coordinate-Based	0.93 ± 0.01	157	10.1
OSSE NADIR	BFNQG [55]	Hybrid Model	0.93 ± 0.01	139	10.6
OSSE NADIR	4DVarNet [14]	Bi-Level Opt.	0.95 ± 0.01	117	7.7
We provide an example snippet of how this can be done easily in section E. OceanBench also features some tools specific to the analysis of SSH. For example, physically-interpretable variables like geostrophic currents and relative vorticity, which can be derived from first-order and second-order derivatives of the SSH, are essential for assessing the quality of the reconstructions generated by the models. Figure 2 showcases some fields of the most common physical variables used in the oceanography literature for the SSH-based analysis of sea surface dynamics. For more details regarding the nature of the physical variables, see appendix B.
Regarding the evaluation framework, we include domain-relevant performance metrics beyond the standard ML loss and accuracy functions. They account for the sampling patterns of the evaluation data. Spectral analytics are widely used in geoscience [55], and here, we consider spectral scores computed as the minimum spatial and temporal scales resolved by the reconstruction methods proposed in [55]. For example, figure 3 showcases how OceanBench generated the isotropic power spectrum and score and the space-time power spectrum decomposition and score. Table 2 outlines some standard and domain-specific scores for the experiments outlined in section 4.3. We give a more detailed description of the rationale and construction of the power-spectrum-specific metrics in
appendix C In terms of baselines, we report for each data challenge the performance of at least one approach for each of the category outlined in Section 4.2
The ocean community faces technological and algorithmic challenges to make the most of available observation and simulation datasets. In this context, recent studies evidence the critical role of ML schemes in reaching breakthroughs in our ability to monitor ocean dynamics for various space-time scales and processes. Nevertheless, domain-specific preprocessing steps and evaluation procedures slow down the uptake of ML toward real-world applications. Our application of choice was SSH mapping which facilities the production of many crucial derived products that are used in many downstream tasks like subsequent modeling [93], ocean health monitoring [101, 73, 47] and maritime risk assessment [106].
Through OceanBench framework, we embed domain-level requirements into the MLOPs considerations by building a flexible framework that adds this into the hyperparameter considerations for ML models. We proposed four challenges towards a ML-ready benchmarking suite for ocean observation challenges. We outlined the inner workings OceanBench and demonstrated its usefulness by recreating some preprocessing and analysis pipelines from a few data challenges involving SSH interpolation. We firmly believe that the OceanBench platform is a crucial step to lowering the barrier of entry for new ML researchers interested in applying and developing their methods to relevant problems in the ocean sciences.
This work was supported by the French National Research Agency (ANR), through projects number ANR-17-CE01-0009-01, ANR-19-CE46-0011 and ANR-19-CHIA-0016); by the French National Space Agency (CNES) through the SWOT Science Team program (projects MIDAS and DIEGO) and the OSTST program (project DUACS-HR); by the French National Centre for Scientific Research (CNRS) through the LEFE-MANU program (project IA-OAC). This project also received funding from the European Union's Horizon Europe research and innovation programme under the grant No 101093293 (EDITO-Model Lab project). This project benefited from HPC and GPU computing resources from GENCI-IDRIS (Grant 2021-101030).
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] All the contributions listed in the abstract are elaborated in sections 3.2, 4.4 and 5
(b) Did you describe the limitations of your work? [Yes] See the last paragraph of section 5 and the appendix as well.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We do not believe that our work has any potential negative societal impacts directly as we do not deal with any confidential or private data. However, we do outline in the appendix how there may be some adverse effects related to downstream uses which could have some negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We do not include any confidential or private data. We only include numerical values which stem from general physical systems or machine learning models. We do not believe they hold any ethical issues. However, we do acknowledge that there would be environmental damage should users go forward and explore methods which obscenely high computing hours. This discussion outlined in the appendix.
(a) Did you state the full set of assumptions of all theoretical results? [N/A] We do not include any theoretical results.
(b) Did you include complete proofs of all theoretical results? [N/A] We do not include any theoretical results.
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] We include the parameters used to reproduce the dataset preprocessing and evaluation procedure in Appendix A and instructions are given to download the data via https://github.com/quentinf00/oceanbench-data-registry and rerun the evaluation procedure in our code repository which is available at https://github.com/jejjohnson/oceanbench
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We showcase all preprocessing steps necessary to reproduce the experimental configurations in Appendix A and the configuration files are available in our code repository at https://github.com/jejjohnson/oceanbench.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [N/A] This is not applicable for this instantiation because we do not include any randomness within the experiment procedure nor the results.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We do not do any model training and leave it up the user for their local or cloud machine. However, we do provide the cloud provider for the data found the data registry which can be found at https://github.com/quentinf00/oceanbench-data-registry
(a) If your work uses existing assets, did you cite the creators? [Yes] We adopted the implementation of the preprocessing procedures and evaluation steps with some modifications. We give proper citation and credit to the authors as well as all other existing software packages included in this work.
(b) Did you mention the license of the assets? [Yes] The appropriate license notices are included in the source code files.
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes] All the processing and evaluation scripts are included in the GitHub repository.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] We only include data that is already publicly available. We also discussed with the original generators of the datasets and keep the appropriate licenses.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] We do not include any personal information or offensive content in our datasets.
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] We do not use crowdsourcing and we do not conduct research with human subjects.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] See the previous point.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] See the previous point.