Continuous Field Reconstruction from Sparse Observations with Implicit Neural Networks
Authors: Xihaier Luo
Wei Xu
Balu Nadiga
Yihui Ren
Shinjae Yoo
Venue: ICLR 2024 poster (2024)
Keywords: implicit neural representations
field reconstruction
sparse observation
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)

Abstract: Reliably reconstructing physical fields from sparse sensor data is a challenge that frequenty arises in many scientific domains. In practice, the process generating the data is often not known to sufficient accuracy. Therefore, there is a growing interest in the deep neural network route to the problem. In this work, we present a novel approach that learns a continuous representation of the field using implicit neural representations (INR). Specifically, after factorizing spatiotemporal variability into spatial and temporal components using the technique of separation of variables, the method learns relevant basis functions from sparsely sampled irregular data points to thus develop a continuous representation of the data. In experimental evaluations, the proposed model outperforms recent INR methods, offering superior reconstruction quality on simulation data from a state of the art climate model and on a second dataset that comprises of ultra-high resolution satellite-based sea surface temperature field. [Website for the Project: Both data and code are accessible.](https://xihaier.github.io/ICLR-2024-MMGN/)

Published as a conference paper at ICLR 2024
CONTINUOUS FIELD RECONSTRUCTION FROM SPARSE OBSERVATIONS WITH IMPLICIT NEURAL NETWORKS
Xihaier Luo, Wei Xu, Yihui Ren, Shinjae Yoo Brookhaven National Laboratory (xluo, uo, xuw, yren, s jyoo) eyyoo@bbll... abnl.gor
Balasubramanya Nadiga
Los Alamos National Laboratory {ba lule @allllnn.... gov
ABSTRACT
Reliably reconstructing physical fields from sparse sensor data challenge that frequently arises in many scientific domains. In practice, the process generat- ing the data often not understood to sufficient accuracy. Therefore, there is a growing interest in using the deep neural network route to address the problem. This work presents a novel approach that learns a continuous representation of the physical field using implicit neural representations (INRs). Specifically, after fac- torizing spatiotemporal variability into spatial and temporal components using the separation of variables technique, the method learns relevant basis functions from sparsely sampled irregular data points to develop a continuous representation of the data. In experimental evaluations, the proposed model outperforms recent INR methods, offering superior reconstruction quality on simulation data from a state- of-the-art climate model and a second dataset that comprises ultra- altra-higgh resolution satellite-ban -iiiie----eddddeee sea surface temperature
Website: data & code]
INTRODUCTION
Achieving accurate and comprehensive representation of complex physical fields is pivotal for tasks spanning system monitoring and control, analysis, and design. However, in a multitude of appli- cations, encompassing geophysics (Reichstein et al., 2019), astronomy (Gabbard et al., 2022), bio- chemistry (Zhong et al., 2021), fluid mechanics (Deng al., 2023), and others, using a sparse sensor network proves to be the most practical and effective solution. In meteorology and oceanography, variables such as atmospheric pressure, temperature, salinity/humidity, and wind/current velocity must be reconstructed from sparsely sampled observations.
Currently, two distinct approaches are used reconstruct full fields from sparse observations. Tra- ditional physics model-based approaches are based on partial differential equations (PDEs). These approaches draw upon theoretical techniques to derive PDEs rooted in conservation laws and fun- damental physical principles (Hughes, 2012). Yet, in complex systems such as weather (Brunton et al., 2016) and epidemiology (Massucci et al., 2016), deriving comprehensive models that are both sufficiently accurate and computationally efficient remains elusive. Moreover, integrating field data into these derived PDEs for validation and calibration poses significant challenges (Raissi et al., approaches emerge as an alternative avenue for non-
2019). Concurrently, machine-learning-baal linear field reconstruction (Mescheder et al., 2019; Sitzmann et al., 2020; Mildenhall et al., 2021).
In contrast standard image and video data, scientific data describing complex physical systems present unique challenges. For example, sparse seismic networks (sparse spatial coverage) can lead to smaller earthquakes being unnoticed or their epicenters being misestimated (Myers & Schultz, 2000). Mcanwhile, fluid dynamics in turbulent flows, an example of high nonlinearity, exhibit non- linear behavior due to interactions between vortices and eddies (Stachenfeld al., 2021). Other examples include sensor mobility, e.g., ocean waves and currents that transport floating buoys (Ro- drigues et al., 2021), and on-off dynamics, e.g., cloud cover impacting solar panels that cause power fluctuations and grid instability (Paletta et al., 2022). These factors are driving the advancement of novel machine learning models, aiming to enhance and refine current approaches for field recon- struction. In this work, we introduce the first implicit neural representation (INR)- model for global field reconstruction of scientific data from sparse observations with the following contribu- tions:

---

Published as a conference paper at ICLR 2024
We introduce context-aware indexing mechanism that compared to standard time index (t)-based INR models, incorporates additional semantic information.
The presented network factorizes target signals into set of multiplicative basis functions, subse- quently applying dlementwwise shift and scale transformations to amalgamate latent information.
Empirical validation demonstrates the proposed model achieves average relative error reduction of 39.19% compared to other state-of-the-- INR models.
RELATED WORK
2.1 CLASSICAL METHOSS
Regression Methods. Field reconstruction resembles regression, predicting new outcomes at new locations. The most straightforward approach is construct linear model using available data. To account for nonlinearity, inverse distance weighting (IDW) employs distance weighting with a power parameter for weighted averaging in spatial interpolation (Shepard, 1968). While popular, IDW assumes isotropy and operates within the convex hull of data points. A more potent alternative Gaussian Process (Rasmussen al., 2006). practice. the application of Gaussian Process can be hindered by the computational complexity of inverting the covariance matrix, which scales with time complexity of (n²) and renders infeasible for extremely large datasets (Angell & Sheldon, 2018; Yadav et al., 2021).
Model Reduction Methods. Model reduction techniques address the reconstruction task by con- verting the continuous spatial representation into composite of basis functions, often referred to as modes: u(2r) ú(x)) = "i= (, (æ)). Coefficients 0, and modes (xx))) typically are de- termined through optimization regression models. Extensions of existing model reduction tech- niques for reconstructing full-field from partial-field measurements typically incorporate a mask function m(u, 2) that is defined O where data are missing and 1 where data are present, such as in the case of Gappy proper orthogonal decomposition (Everson & Sirovich, 1995; Bui-Thank et al., 2004). Recently, deep learning techniques have been employed construct a nonlinear manifold, enhancing model performance for slowly decaying Kolmogorov n-width problems (Lee & Carlberg, 2020: Lusch al., 2018; Kim et al., 2019).
2.2 DEEP LEARNING METHODS
Super Resolution (SR). SR typically focuses on upscaling low-resolution images Noow to higher resolution Whigh. Yet in the context of field reconstruction, we lack such paired data. Our primary focus is generating continuous representation u from a discretized dataset. Recent progress in deep-learning based SR enables continuous magnification through diverse techniques, includ- ing innovative training approaches, such scale-consistent positional encodings (Ntavelis et al., 2022) and variable-size training (Chai et al., 2022); local conditioning methods that use deep sur- rounding features (Chen al., 2021) neighborhood-based interpolation (Luo et al., 2023); and global conditioning methods that leverage continuous coordinates and latent variables. e.g., Mcsh- freeFlowNet (Esmaeilzadch al., 2020) and Neural Implicit Flow (NIF) (Pan al., 2023).
Neural Inpainting. Image inpainting techniques can be broadly classified into two categories: traditional and learning-based methods. Traditional methods primarily rely on low-level features, employing approaches like diffusion- (Bertalmio et al., 2000) or batch-based (Barnes et al., 2009) methods extend information from surrounding regions into the missing areas. On the other hand, learning-bused methods, particularly those employing GANs (Yu et al., 2018; Lee et al., 2020) and probabilistic diffusion models (Song et al., 2023b; Chung et al., 2023), have achieved more precise and semantically meaningful inpainting results. However, these learning-based methods may require post-processing and can be computationally demanding.
Implicit Neural Representations. INR models rely coordinate-based neural networks (Xie et al., 2022). INRs can serve two main purposes: they either parameterize the sensor domain or the densiry domain directly. In the former case, INRs map sensor coordinates to predicted sensor activa- tions, which can be used to enhance real measurement data. For the latter, INRs directly predict the density value a two- three-dimensional (2D/3D) spatial coordinate. Typically, raw sensor mea-
2

---

Published as a conference paper at ICLR 2024
surements are derived from spatially varying density using transformations. Therefore, such direct prediction is supervised by mapping the model's output back to the sensor domain through various transformations, such Radon in computed tomography (Reed et al., 2021), Fourier in magnetic resonance imaging (Song et al., 2023a), or convolution in cryo-electron microscopy (Zhong et al., 2021).
METHODOLOGY
Overview. The objective is accurately reconstruct spatiotemporal continuous physical field, denoted as representing quantities such temperature, velocity, or displacement, This field is inherently a function both spatial coordinates (æ) and time (t). Directly modeling such complex spatiotemporal physical fields poses significant challenges. Consequently. methodologies, e.g., functional separation of variables (Donà et al., 2021), have been devised to mitigate complexity, enhance tractability, and improve physical interpretability. Using these approaches, the underlying physical process (z, f) can be decomposed, for example, in the form the product as fi (a:) f2(t)-
Proposed Method. Conventional spatiotemporal disentangled representation utilizes the time index (t) primarily as a reference to indicate specific time instance. Motivated by the desire for a more context-aware indexing mechanism, we pose the question: Can the pointing process be improved? A natural approach to incorporate available context information in field reconstruction involves using measurements of the underlying physical process at time As the number and positions of available measurements change over time, we propose a design wherein an encoder extracts a latent represen- tation from actual measurements. This latent representation is subsequently employed to guide the model to the target time instance. When coupled with an INR-based decoder, this proposed method achieves continuous field reconstruction. Figure provides a comparison between the conventional scalar-index-blr -ased INR and our context-aware INR.
Problem setup
Prior approach
Our approach
INR
Recodd rotted Miggam cherrcatoon))
INR
u
Figure Field reconstruction from sparse observations: The Prior approach uses the time index (t) as reference to indicate specific time instance. Our approach is context-aware, leveraging available context information by incorporating measurements time t.
Overall Architecture. We introduce a neural network reconstruction method, MMGN (Multiplica- tive and Modulated Gabor Network), that features encoder-decoder architecture. The encoder extracts features from available measurements 44 = {u, ", ....) } at time while the decoder, guided by spatial coordinates æ and context-aware latent code 2,, performs inference for the spe- cific point at time and location æ. Overall, the model is defined Equation (1).
z, = Exp ū(æ, () = Da(zi, VI E and Va E Ω
(1)
No where 12 IR and T = (44 J are the spatial and temporal domains, respectively.
3.1 ENCODER
Autoencoders (AE) and their probabilistic version, variational autoencoders (VAE) (Kingma & Welling, 2014), are commonly used for representation learning due to their natural latent variable formations. However, vanilla AE and VAE struggle with the randomness sensor locations and numbers time. While their graph counterparts handle spatial randomness and modified ver- sions manage dynamic graphs, they become cmmputation-intensiii for large graphs and struggle with long-range dependencies (Pfaff et al., 2021). In contrast to AE and VAE, auto-decoder, as
3

---

Auto-decoder. The aim of an auto-decoder to compress essential information into 2t, enabling the reconstrueted value a/ to closely approximate the original value (6) for any point within the domain. This accomplished through an iterative process z 2 +.... .... employing gradient descent optimization. To initialize the trainable latent codes, we can assume the prior distribution Published as a conference paper at ICLR 2024
loration query
fiter
loss
Triinng
decoder
Airain
anto doo-de
gd
8 c R
SCNDSO
encader
ante-droding tod
decoder
location query
filter
Inference
Figure 2: Architecture of the MMGN Model. The MMGN model employs auto-decoding to infer the latent variable z. Consequently, only the decoder is explicitly defined, and encoding takes place through stochastic optimization. More precisely, the latent code arg min, L(z, θ) is obtained by minimizing a loss function L calculated as an expectation over dataset.
demonstrated in Park al. (2019), exhibits reduced underfitting and enhanced flexibility. It accom- modates free-formed observation grids, including irregular ones those a manifold, without necessitating a specialized encoder architecture as long as the decoder possesses the same prop- erty.
over codes p(z) follow a zero-mean multivariate Gaussian with a spherical covariance o² (Xie
(0) et al., 2022). In practice, we empirically notice that initializing 2 to yields slightly better results than Gaussian initialization:
(0) = 0: (i+1)
(1 a v. C(o)))
u (3) for i O..... -1,
(2)
where a is the learning rate, N is the number of iteration steps, and L(-) is the loss function.
3.2 DECODER
The decoder inputs consist of two parts: spatial coordinates æ and latent codes z. Subjecting to fully connected feed-forward layers yields coordinate-based multilayer perceptron (MLP). While such coordinate-based MLP can offer continuous representation, it struggles to learn high-frequency signals, phenomenon known spectral bias. Recent research indicates this issue can mitigated using positional encoding with Fourier features (Tancik et al., 2020) or periodic nonlinearities in the first hidden layer (Sitzmann et al., 2020). In lieu of Fourier bases, filters to transform the coordinates as Fourier transforms emphasize a global frequency representa- tion, making them less suitable for capturing varying frequency and orientation across different parts of the signal and more susceptible to noise and amplitude fluctuations. Specifically, we employ N Gabor filters the following form:
Gabor
(1) 2
9s (x) exp
(1))
sin ( b(:))
Ng.
(3)
xda denote the respective mean and scale term of the ith Gabor filter. The former is associated with the central frequency of the sinusoidal waveform that g, (x) is designed to identify, while the latter parameter corresponds to the standard deviation of the Gaus- sian envelope that modulates the sinusoidal waveform. This filter exhibits a multiplicative prop- erty (Fathony et al., 2020), which implies the product of the outcomes can be written from any pair
where (6) E dd and (1) E Xd,
4

---

Published as a conference paper at ICLR 2024
of Gabor filters 91 (z), (a) into summation of Gabor bases 91 (z) o (a:) ==== Big, (a) with S1.N denoting the coefficients. This decomposability property facilitates the construction of hierarchical features, allowing the model to capture different levels of abstraction in the data and enhancing the interpretability learned features.
After transforming the coordinates g (z), introduce modulation step where the transformed coordinates are modulated through a multiplicative layer, thereby integrating x and z.
Multiplicative Filter Network. Similar to the multiplicative filter network approach (Fathony et al., 2020), the modulation layer involves the iterative application of nonlinear Gabor filters to the network's input. These filters then multiplied by the linear transformations of both and z. Specifically, in a decoder comprising L layers, the decoding process is defined iteratively as follows:
h() = (x))
h(++1) 9: (a)) ④ [ h x +6(%)) , i 1.=...... -i.....,,
Dø(2,0) x) w + b
(L)
where (6) E Rd44xdd, w E Rdu+ and (()) E Rd++1 denote the weights and bias of the ith layer; (0) E Rdd marks the hidden unit layer i; and indicates the element-wise multiplication. An intriguing aspect of the multiplicative filter network is that the final output can be expressed as a linear composition of Gabor filters:
Fø(U,, Σ
(m) Ch
(x:
Tk (m)) + bias,
(5)
where M >> E N. {ck Jm=1 represents coefficients that depend on {W,, Wz, bn.} individu- Jm=1 set of filter parameters, i.e., 14, We jointly train both the encoder and decoder. Additional information about training procedures and empirical observations related to the initialization of the encoder and decoder be found in Appendix B.2.
MM
ally, and {TA (T(")))
3.3 TEMPORAL ADAPTABILITY
Owing to coordinate-based architecture, the proposed model generates continuous represen- tation in the spatial domain. However, replacing the conventional time index (!)-based INR with a context-aware indexing mechanism introduces a considcration: the direct support for continuous representation in the temporal dimension may be affected. For example: Reconstruction. For time instances where 24; is not available. This can be accomplished through the application in- terpolation techniques, such as linear spline interpolation, to the latent codes. Subsequently, the interpolated latent code, in conjunction with the trained decoder, is employed to make inferences. Nowcasting. Typically involves predicting current very-near-future weather conditions within the next few hours, using real-time observational data. In this context, we assume access to ob- servations for the nowcasting model. During the inference stage. the model generates new latent code based on these observations while keeping all decoder parameters fixed. Forecasting, As the decoder remains fixed after training, the extrapolation for forecasting tasks involves predicting the latent code. This can be achieved through two types of methods: autoregressive and neural operator. Autoregressive methods iteratively update the latent code z(t + At) M(At, z (()), where M R50 x dRd - d is the temporal update (e.g.. recurrent neural network (RNN) (Sher- stinsky, 2020)), while neural operator methods map a stack of latent codes to a future state/states: 2, MMtt 21, .2...., 21-1) (e.g., Fourier neural operator (FNO) (Li et al., 2021)). In summary, the proposed model's temporal adaptability is contingent on the study 's objectives. Detailed temporal interpolation and extrapolation can be seamlessly integrated into the proposed model needed.
5

---

Published as a conference paper at ICLR 2024
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets. For these experiments, we evaluate the model's performance on two challenging datasets (additionally detailed Appendix A): Simulation-ban bastdd data. The Community Earth System Model version 2 (CESM2) (Danabasoglu et al., 2020), fully coupled global climate model, is used to simulate Earth's climate states. This work uses monthly averaged global surface temperature data, representing atmospheric field, for model testing. Note that seasonal cycles have been removed to augment complexity, and the dataset dimensions are 1024 (time), 192 (lat), and 288 (lon).
Satellite-based data. Sea surface temperature data derived from both retrospective dataset with a four-day latency and a near-real-time dataset with a one-day latency (Martin et al., 2012). Wavelets are employed as basis functions for optimal interpolation on a global 0.01-degree grid. We analyze one year of daily data, spanning from August 20, 2022 to August 20, 2023, at the provided resolution of mee-hundredth of degree, focusing on the Gulf Stream region. The dataset dimensions are 360 (time), 901 (lat), and 1001 (lon).
Tasks. We assess the performance of models across diverse field reconstruction tasks to gauge their efficacy in various scenarios. including: Randomness. Four tasks of increasing complexity are defined to evaluate the models' capability in handling sensor-related randomness. In the first task, models are trained with consistent number of data points located at fixed positions. The second task introduces variability by randomly varying the number of data points. In the third task, we maintain fixed number of data points but introduce randomness by randomly sampling grid maps, while the fourth task combines both aspects of variability, involving random sampling of the number of data points and the grid map itself. Sparsity. In each task, we define three sparsity levels. Specifically, the training procedure involves using partial observations sampled from the complete state, i.e., (5%, 25%, 50%) for simulation-based data and E (0.1%, 0.3%, 0.5%) for saelllite-bseed data. Testing employs the complete state (s = 100%). In Appendix B. Figure 8 provides a visual representation of the four tasks with additional task definitions available.
Baselines. Our model is assessed against a series of state-of-the-art implicit neural networks for field reconstruction. ResMLP (Huang & Hoefler, 2023) contains a sequence of six fully connected
Table 1: Performance comparison with four INR baselines on both high-fidelity climate simulation data and real-world aitelite-based benchmarks. MSE is recorded. A smaller MSE denotes supe- rior performance. For clarity, the best results are highlighted bold, while the second-best are underlined. The promotion metric, which indicates the reduction in relative error compared to the second-bee best model for each task, also is included.
Simulation- based Data
Satellite-based Data
Model
Task
Task 2
Task 3
Task 4
Task
Task 2
Task 3
Task 4
Sampling ratio 5%
Sumpling ratio
0.1%
ResMLP
1.951e-2
1.672c-2
1.901e-2
1.468e-2
1.717e-3
1.601e-3
1.179e-3
1.282c-3
SIREN
2.483e-2
2.457e-2
2.730e-1
2.455e-2
3.129e-1
4.398e-2
1.304e-2
9.338c-2
FFN+P
2.974e-2
1.121e-2
1.495e-2
8.927e-3
2.917e-3
2.392e-3
7.912e-4
7.565e-4
FEN+G
2.943e-2
1.948c-2
1.980e-2
1,426e-2
4.90Me-3
7.969e-3
1.005e-3
1.044c-3
MMGN
4.244e-3
4.731c-3
3.148
3.927e-3
1.073e-3
1.131e-3
6.309e-4
6.298c-4
Promotion
78.24%
57.79%
78.94%
56.01%
37.51%
29.35%
20.26%
16.74%
Sampling ratio s 25%
Sampling ratio 8 0.3%
ResMLP
1.593e-2
1.252e-2
1.322e-2
1.378e-2
9,601c-4
7.808
8.264e-4
8.144c-4
SIREN
2.643e-2
2.669e-2
2.730e-1
2.679e-2
7,630e-3
6.421e-3
6.297e-3
9,925e-3
FFN+P
8.374e-3
7.905e-3
7.720e-3
6,514e-3
8,429c-4
8.294c-4
4.823e-4
5.580e-4
FEN+G
1.307e-2
1.360e-2
1.331e-2
1.300e-2
9.169e-4
1.157e-3
8.128e-4
6.253e-4
MMGN
2.955e-3
2.991c-3
2.780e-3
2.802c-3
6.116e-4
5.912
4.582
4.896c-4
Promotion
64.71%
62.16%
63.98%
56.98%
27.44%
24.28%
4.99%
12.25%
Sampling ratio 8 50%
Sampling ratio = 0.5%
ResMLP
1.004e-2
8.461e-3
9.716e-3
1.138e-2
6.741e-4
7.752e-4
5.790e-4
8.317e-4
SIREN
2.728e-1
2.728e-1
1.408e-2
1,3826-2
2,214e33
9.131e-4
5.973e-4
8.019e-1
FFN+P
5.383e-3
6.192e-3
5.782e-3
5,5036-3
5,413e-4
5.807e-4
4.226e-4
5.004e-4
FFN+G
1.182c-2
1.148e-2
1.201e-2
1.124c-2
7.067e-4
8,256e
6.783e-4
5,721e-4
MMGN
2.802c-3
2.824c-3
2.730c-3
2.760c-3
5,081e-4
4.760e-4
4.127G-4
4.124c-4
Promotion
47.94%
54.39%
52.78%
49.84%
6.13%
18.02%
2.34%
17.58%

---

Published as a conference paper at ICLR 2024
blocks, and each block consists of two fully connected feedforward layers, incorporating batch nor- malization and concluding with a skip connection. SIREN (Sitzmann et al., 2020) is instantiated an MLP consisting of five fully connected feedforward layers, each employing sine activation functions. FFN+P/G (Tancik et al., 2020) refers to the Fourier feature network with either posi- tional encoding (P) Gaussian encoding (G). The network consists of 4-layer coordinate- based MLP. It applies element-wise Fourier feature mappings to the input and uses the Gaussian error lin- ear unit, or GELU, function for nonlinear activation. Network details are available in Appendix B.3 and B.4.
4.2 MAIN RESULTS
Quantitative Evaluation. Table presents the reconstruction errors across all experiments. No- tably, MMGN consistently outperforms the other baseline models. This performance advantage is particularly pronounced under low sampling ratios. Generally, as the subsampling ratio decreases, all models experience degradation in performance. MMGN still achieves significant enror reduc- tions, ranging from 56.01% to 78.94% for simulation-based data and 16.74% to 37.51% for satellite- based data when compared to the second-best model, particularly the lowest sampling ratio. In- depth analysis of the detailed results, including extreme-case scenarios and convergence studies, can be found in Appendix C.I and C.2.
Qualitative Evaluation. Figure 3 illustrates relative test errors and models' predictions. The visu- alization highlights the superiority of MMGN over other methods. ResMLP exhibits over-smoothed predictions, struggling to capture high-frequency signals. FFN+P displays some structural checker- board effects, particularly noticeable the satellite-based data. Despite quantitatively performing slightly worse than FFN+P, FFN+G demonstrates significant reconstruction improvements, success- fully capturing both the overall landscape and finer details. In contrast, MMGN's prediction results faithfully reconstruct data with complex structures from sparse measurements, leading to substan- tial reduction errors. Appendix C.4 features the full results.
Reference
MMGN
ResMLP
SIREN
FFN+P
FFN+G
Figure 3: Visualizations of true and reconstructed fields. Global surface temperature derived from multiscale high-fidelity climate simulations and sea surface temperature assimilated using satellite imagery observations. For each dataset, the first column displays the ground truth, the first row showcases predictions from different models, and the second row presents corresponding error maps relative to the reference data. In the error maps, darker pixels indicate lower error levels.
7

---

Published as a conference paper at ICLR 2024
MSE
Robustness to Noise. It is important to investigate the model's performance in the presence of noise. Here, we
0.05
ResMLP
FEN-P
quantify the noise by the channelwise standard deviation specific to that dataset and customize noise ratios, includ- ing scenarios with noise levels set 1 1%, 5%, and 10%. The analysis is conducted on the simulation dataset, and the computed results reveal that the proposed MMGN model surpasses current baselines. Notably, ResMLP does exhibit robust performance in maintaining accuracy as the noise level rises. Meanwhile, MMGN maintains its performance effectively when the noise ratio below 5% with noticeable increase in errors observed when the noise reaches 10%. In contrast, the performance degrada- tion of the two FFN types is more evident.
0.04
FFN-G
MMGN
0.02
0.01
Noise free
5%
10%
Figure 4: Model performance with dif- ferent levels of noise.
Ablations of Gabor Filter. assess the Gabor filter's efficacy in MMGN, we conduct detailed ablations, encompassing the removal filter de- signs and their substitution with alternative types. Key observations from Table include: 1) elimi- nating filters leads significant drop model performance, highlighting the indispensability of filter designs and 2) substituting the Gabor fil- ter with Fourier filter not only diminishes ac- curacy (9.0% drop in simulation data and 18.1% decrease in satellite data), but it also increases the model size.
Table 2: Network Design Ablations. Two types of experiments were conducted: substituting the proposed Gabor filter with alternative designs and removing filters altogether. The evaluation metric is mean squared error (MSE).
Designs
Simulation # Param
Satellite
None Fourier
577 K
1.758e-2
6.883e-3
601 K
4.439e-3
7.422c-4
Gabor 581 K
4.073e-3
6.290e-4
MSE
Ablations of Context-aware Indexing Mechanism. To evaluate the effectivvness of the proposed context-aware indexing mechanism, the latent size of MMGN is inten- tionally reduced to 1. In this configuration, akin to cur- rent INR baselines, the model is equipped with three in- puts: x-coordinate, g-coordinate, and the latent code 2, which is scalar in this specific experiment. Given that z, is learned from the entirety of available measurements time is anticipated encapsulate more semantic information, consequently enhancing the decoder's per- formance. As depicted Figure 5, the results indicate that with a latent size reduced to 1, MMGN exhibits slight but consistent performance improvement over the second-best model, ResMLP, across both datasets.
0.001
ReMLP
FFN+P
0.025
FFN+G
0.020
MMGN
0.015
0.005
0.000
simmlation date
satellite data
Figure 5: Ablation context-aware in- dexing mechanism.
4.3 MODEL ANALYSIS
Model Efficiency. Model efficiency is evaluated based on inference speed and model size (depicted in Fig- ure 6). Each model is executed 10 times to evaluate the entire dataset and compute the average inference speed for each instance. Overall, MMGN exhibits a favor- able balance between accuracy and efficiency, making it the top-performing model. FFN+P ranks as the second- best model in most tasks (refer to Table 1), yet pos- sesses slightly more model parameters, resulting in re- duced efficiency. Concretely, in the case of simulation data, MMGN outperforms other models with relative speed improvement of 9.28%, 2.09%. 7.03%, and 3.65% compared to FFN+P, ResMLP, FFN+G, and SIREN, spectively. Similar results are observed in the satellite- based dataset, and additional details provided in the Appendix C.3.
Sinulation--beas data
Infessnn que Instrnto Eame?"
0.007
Traimblle
0.056
500K
1.NIM
0.0314
FTN+P
0.033
SIREN
ReeLLP
0.001
MMGN
Prrformmaec ranking
Figure 6: Efficiency Comparison. Infer- ence time assessed per instance.
8

---

Published as a conference paper at ICLR 2024
(a) dissimilarity of latent codes
(b) NMSE
[c) correspondence between latent
codes and original data
V0
V00
Figure 7: XAI analysis results: (a) dissimilarity between latent codes, (b) ablation of latent variable, and (c) diagnosis of autoddecoder.
Explainable Artificial Intelligence (XAI) Analysis of Latent Codes. To further explore the influence of the latent code 2 on our model's performance, we train 10 models with different latent sizes, ranging from to 512, by doubling the latent size time and collect the cor- responding learned latent codes Zt, for all time steps ty. We then assemble a matrix Z [x(1.1), (2.1), (2,22) 2 2611211) _512222) , (512,5112) with trained latent codes from different experiments. Using simulation based data, which includes total of 1024 time instances, results in a matrix size of 1024 by 1023- = -1-2--.-
2
512. Similarity between latent codes. For a certain latent space, we compute the pairwise Pearson Correlation of the latent codes and use the standard deviation of all the correlations to represent the overall similarity of latent variables in that latent space. In Figure 7(a), when the latent size increases, the standard deviation (dissimilarity) decreases. To further illustrate how the latent variables correlate across latent spaces, we use t-SNE (Van der Maaten & Hinton, 2008) to embed them into the same 2D plane and compare their similarities. As shown in Appendix C.5, scatter plots of the latent variables become more abundant as the latent size increases, where these latent variables are more similar. * Ablation of latent variable. Each variable inside the latent code referred as a latent variable. For example, if the latent code is of dimension 1024, it consists of 1024 latent variables. In our ablation study, we iteratively remove one variable at a time to regenerate the entire dataset. This process is repeated for each latent variable, and the MSE is calculated accordingly. Subsequently, we compute the percentage increase in MSE results, denoted as "NMSE." To facilitate interpretation of the ablation study across latent variables, we employ a boxplot for each latent space. The boxplot illustrates the distribution of NMSE values after ablating a specific latent variable. As depicted the Figure 7(b), both the mean and variance of the error consistently decrease the latent size increases. Diagnosis of auto-decoder. The design of latent codes captures both temporal and spatial information through the proposed context- aware indexing mechanism. To illustrate, we consider the entire dataset and treat the latent vectors as nigh-dimensional temporal vectors. With 1024 temporal vectors, their dimensions vary from la- tent sizes (e.g., 1, 2, ..... 512) to the original spatial data dimension (192 288). For the temporal vectors in any latent space, we calculate the pairwise Pearson Correlation, comparing them with pairwise Pearson Correlation of the original temporal vectors. Figure 7(c) presents this comparison using MSE to measure the differences and showcase the correspondence of the latent vectors to the original data. We observe that as the latent dimension increases, the error consistently decreases.
5 CONCLUSION
This work introduces MMGN, a novel INR model for scientific data reconstruction. Compared to other time index (f)-based INR models, MMGN introduces a more context-aware indexing mecha- nism via a trainable latent code. Comprehensive experiments have been conducted to showcase the improvements facilitated by such a context-aware representation. One limitation of this current work is its focus on reconstructing a selected trajectory. In the future, we plan to investigate MMGN's potential for generalization to multiple trajectories or even across various climate properties with the goal of recovering arbitrary underlying flow maps. Concurrently, exploring the application of MMGN for optimal sensor placement is worthwhile avenue to explore. Given the challenging combinatorial nature of optimal sensor placement, MMGN can serve as a surrogate model to expe- dite the search process.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
The authors would like to thank Klaus Tan and Avish Parmar for their efforts in analyzing the climate datasets. This material is based upon work supported by the U.S. Department of Energy (DOE)'s Office of Science, Office of Advanced Scientific Computing Research, Office of Biological and Environmental Research, and the Scientific Discovery through Advanced Computing (SciDAC) program under Award Number 9233218CNAA0000 by the DOF's Office of Science under Contract No. DE-SC0012704. This research used the Perl- mutter supercomputer of the National Energy Research Scientific Computing Center, which is sup- ported by DOE's Office of Science under Contract No. DE-AC02-05CHI 1231. The authors also thank the anonymous reviewers for their comments and suggestions that have helped to improve the manuscript's ss quality and clarity.
Brookhaven National Laboratory is supported
REFERENCES
Rico Angell and Daniel R Sheldon. Inferring latent velocities from weather radar data using gaussian processes, Advances in Neural Information Processing Systems, 31, 2018.
Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan Goldman. Patchmatch: A ran- domized correspondence algorithm for structural image editing, ACM Trans. Graph., 28(3):24, 2009.
Marcelo Bertalmio, Guillermo Sapiro, Vincent Caselles, and Coloma Ballester. Image inpainting. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pp. 417-424, 2000.
Steven L Brunton, Joshua L Proctor, and Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the national academy of sciences, 113(15):3932-3993 2016.
Tan Bui-Thanh, Murali Damodaran, and Karen Willcox. Aerodynamic data reconstruction and inverse design using proper orthogonal decomposition. AIAA journal, 42(8):1505-1566, 2004.
Lucy Chai, Michael Gharbi, Eli Shechtman, Phillip Isola, and Richard Zhang. Any-resolution train- ing for high-resolution image synthesis. In European Conference on Computer Vision, pp. 170- 188. Springer, 2022.
Yinbo Chen, Sifei Liu, and Xiaolong Wang. Learning continuous image representation with local implicit image function. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pp. 8628-8638, 2021.
Hyungjin Chung, Jeongsol Kim, Michael Thompson Mccann, Marc Louis Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. In The Eleventh Intera- tional Conference on Learning Representations, 2023. URL Corum?id=Onm GAGTO..
Gokhan Danabasoglu, J-F Lamarque, Bacmeister, DA Bailey, AK DuVivier, Jim Edwards, LK Em- mons, John Fasullo. R Garcia, Andrew Gettelman, et al. The community earth system model ver- sion (cesm2). Journal of Advances in Modeling Earth Systems, 2202):2019MS00191 2020.
Yitong Deng, Hong- Xiin Yu, Jiajun Wu, and Bo Zhu. Learning vortex dynamics for fluid inference and prediction. In The Eleventh International Conference on Learning Representations, 2023.
URL httph:tt//// //ts:://ppeeee /ttp/pppeeeeeeeeeeeeeeeeeeeeeeee net. www.teettoouueeeee forum? id=nYMgxUWFc.3x
Jérémie Donå, Jean-Yve Franceschi, sylvain lamprier, and patrick gallinari. (drve-driven spa-
tiotemporal disentanglement. In International Conference on Learning Representations, 2021. URL
.ttei//wwweeeeeeeeeeeeeeeeeeeeee fiet/forrmt htt.:/oorrmmmtttt..eeEEEE id
Soheil Esmaeilzadeh, Kamyar Azizzadenesheli, Karthik Kashinath, Mustafa Mustafa, Hamdi A Tchelepi, Philip Marcus, Mr Prabhat, Anima Anandkumar, et al. Meshfreeflowwett A physics- constrained deep continuous space-time tice-time super-resolution framework. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. IEEE, 2020.
10

---

Published as a conference paper at ICLR 2024
Richard Everson and Lawrence Sirovich. Karhumen-loeve procedure for gappy data. JOSA A, 12 (8):1657-1664, 1995.
Rizal Fathony, Anit Kumar Sahu, Devin Willmott, and J Zico Kolter. Multiplicatiiv filter networks. In International Conference on Learning Representations, 2020.
Hunter Gabbard, Chris Messenger, Ik Siong Heng. Francesco Tonolini, and Roderick Murray-Smin Bayesian parameter estimation using conditional variational autoencoders for gravitational-want astronomy. Nature Physics, 18(1): 12-111122,177, 2022.
Langwen Huang and Torsten Hoefler. Compressing multidimensional weather and climate data into neural networks, In The Eleventh International Conference on Learning Representations, 2023. URL ptttps://openrevieee neterreeiieeeeeeeeeeeeeee wwwp//ee//rttte SaerrmmiiddYSSEeddaae.
Thomas JR Hughes. The finite element method: linear static and dynamic finite element analysis. Courier Corporation, 2012.
Byungsoo Kim, Vinicius C Azevedo, Nils Thuerey, Theodore Kim, Markus Gross, and Barbara Solenthaler. Deep fluids: A generative network for parameterized fluid simulations. In Computer graphics forum, volume 38, pp. 59-70. Wiley Online Library, 2019.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
Cheng-Han Lee, Ziwei Liu, Lingyum Wu, and Ping Luo. Maskgan: Towards diverse and interactive facial image manipulation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5549-5558, 2020.
Kookjin Lee and Kevin T Carlberg. Model reduction of dynamical systems on nonlinear manifolds using deep convolutional autoencoders. Journal of Computational Physics, 404: 404:108973, 2020.
Zongyi Li, Nikola Borislavov Kovachki, Kamyar Azizzadenesheli, Burigede liu, Kaushik Bhat- tacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations, 2021. URL
.et/ www.mooidoooooo.oooooo.o wto.iiid
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Confer- ence Learning Representations, 2019. URL https: /opp/rppeevvieeeeeeeeeeee..eeeee .wt.tt/w.eeeeeeee..ee....e.eee.e forum? d=
Bkg6RicqY7.
Xihaier Luo, Xiaoning Qian, Nathan Urban, and Byung-Jun Yoon. Reinstating continuous climate patterns from small and discretized data. In /st Workshop on the Synergy of Scientific and Machine Learning Modeling @ ICML2023, 2023.
Bethany Lusch, Nathan Kutz, and Steven L Brunton. Deep learning for universal linear embeddings of nonlinear dynamics. Nature communications, 91)):4950, 2018.
Matthew Martin, Prasanjit Dash, Alexander Ignatov, Viva Banzon, Helen Beggs, Bruce Brasnett, Sran-Francois Cayula, James Cummings, Craig Donlon, Chelle Gentemann, et al. Group for high resolution sea surface temperature (ghrsst) analysis fields inter-comparisons. part 1: A ghrsst multi-product ensemble (gmpe). Deep Sea Research Part II: Topical Studies in Oceanography, 77:21-30, 2012.
Francesco Alessandro Massucci, Jonathan Wheeler, Raúl Beltrán-Debón, Jorge Joven, Marta Sales- Pardo, and Roger Guimerà. Inferring propagation paths for sparsely observed perturbations on complex networks. Science advances, 2 2016.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Oc- cupancy networks: Learning 3d reconstruction in function space. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 4460-4470, 2019.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 6511).991106, 2021.
11

---

Published as a conference paper at ICLR 2024
Stephen C Myers and Craig A Schultz. Improving sparse network seismic location with bayesian kriging and teleseismically constrained calibration events. Bulletin of the Seismological Society of America, 90(())199-21) 2000.
Evangelos Ntavelis, Mohamad Shahbazi, lason Kastanis, Radu Timofte, Martin Danelljan, and Luc Van Gool. Arbitrary-see scalee image synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp- 1 11533-11542, 2022.
Quentin Paletta, Anthony Hu, Guillaume Arbod, Philippe Blane, and Joan Lasenby. Spin: Sim- plifying polar invariance for neural networks application to vision-based irradiance forecasting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
5182-5191, 2022.
Shaowu Pan, Steven L. Brunton, and J. Nathan Kutz. Neural implicit flow: mesh-agnostic dimen- sionality reduction paradigm of spatio-temporal data. Journal of Machine Learning Research, 24 (41):1-60, 2023. URL http: / jmlr org/ pwpprsaprss..eeeeaa /22/2222222222222 html.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pp. 165-174, 2019.
Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzaler and Peter Battaglia. Learning mesh-based simulation with graph networks. In International Conference on Learning Representations, 2021.
URL hwwph:ttp///mm httpp:://ppeeeevieeeeeeeet http///pppneeeeeeeeeeeeeeeeeeeet wet.riiwwweeeeeeeeee.e net/foummt idd L.d=roNgYYO_X0
Maziar Raissi, Paris Perdikaris, and George Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-707, 2019.
Carl Edward Rasmussen, Christopher KI Williams, et al. Gaussian processes for machine learning, volume 1. Springer, 2006.
Albert W Reed, Hyojin Kim, Rushil Anirudh, K Aditya Mohan, Kyle Champley, Jingu Kang, and Suren Jayasuriya. Dynamic et reconstruction from limited views with implicit neural representa- tions and parametric motion fields. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 2258-2268, 2021.
Markus Reichstein, Gustau Camps-Vall Bjorn Stevens, Martin Jung, Joachim Denzler, Nuno Car- valhais, and fam Prabhat. Deep learning and process understanding for data-driven earth system science. Nature, 166(7743):195-204 2019.
Cátia Rodrigues, Miguel Ramos, R Esteves. Correia, Daniel Clemente, Francisco Gonçalves, Nuno Mathias, Mariany Gomes, João Silva, Cândido Duarte, et al. Integrated study of triboelectric nanogenerator for ocean wave energy harvesting: Performance assessment in realistic sea condi- tions. Nano Energy, 84:105890, 2021.
Donald Shepard. A two-dimensional interpolation function for iregularly-spaced data. In Proceed- ings of the 1968 23rd ACM national conference, pp. 517-524, 1968.
Alex Sherstinsky. Fundamentals recurrent neural network (rnn) and long short-term memory (Istm) network. Physica D: Nonlinear Phenomena, 404:132306, 2020.
Lawrence Sirovich. Turbulence and the dynamics of coherent structures. coherent structures.
Quarterly of applied mathematics, 45(3):611-571, 1987.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in neural information processing systems, 33:7462-7473, 2020.
Bowen Song, Liyue Shen, and Lei Xing. Piner: Prior-informdd implicit neural representation leamn ing for test-time adaptation in sparse-view et reconstruction. Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 1928-19333 2023a.
12

---

Published as a conference paper at ICLR 2024
Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz. Pseudoinverse- guided diffusion models for inverse problems. In International Conference on Learning Representations, 2023b.
URL https::////// htttpps://pennevieee... net. wwwp:eiwwww.eetooune.eeeee... Eorum?id=9 3 gaMABMRKO.
Kim Stachenfeld, Drummond Buschman Fielding, Dmitrii Kochkov, Miles Cranmer, Tobias Pfaff, Jonathan Godwin, Can Cui, Shirley Ho, Peter Battaglia, and Alvaro Sanchez-GGozalezz. Learned simulators for turbulence. International conference on learning representationss 2021.
Kunihiko Taira, Steven L Brunton, Scott TM Dawson, Clarence W Rowley, Tim Colonius, Bev- erley J McKeon, Oliver Schmidt, Stanislav Gordeyev, Vassilios Theofilis, and Lawrence Ukeiley. Modal analysis of fluid flows: An overview. Aiaa Journal, 55(12):4013-4000 2017.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Kad Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Process- ing Systems, 33:7537-7547, 2020.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. Neural fields in visual comput- ing and beyond. In Computer Graphics Forum, volume 41, pp. 641-676. Wiley Online Library, 2022.
Mohit Yadav, Daniel Sheldon, and Cameron Musco. Faster kernel interpolation for gaussian pro- cesses. In International Conference Artificial Intelligence and Statistics, pp. 2971-2979. PMLR, 2021.
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Generative image inpainting with contextual attention. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5505-5514, 2018.
Ellen Zhong, Tristan Bepler, Bonnie Berger, and Joseph Davis. Cryodrgn: reconstruction of heterogeneous cryo-em structures using neural networks. Nature methods, 18(2):176-18) 2021.
A DATASET DETAILS
For testing and validation of the proposed methodology. we use two datasets with different char- acteristics. While one dataset comes from a simulation leading climate model, the other is an observation-base based dataset. Furthermore, because the ocean and atmosphere are two key dynamical components of the climate system with widely differing characteristies of spatiotemporal variability, the first dataset is an atmospheric field, while the second is an oceanic field. This choice permits us to evaluate the proposed approach's 's performance in reconstructing continuous fields from sparse observations using implicit neural representations (INRs) over range of varied conditions and demonstrate its potential for a range of other applications.
A. SIMULATION-BASED GLOBAL SURFACE TEMPERATURE
The first dataset is obtained from the pre-industrial control run of the Community Earth System Model version (CESM2), which is state-of-the-arr climate model developed by the National Center for Atmospheric Research (NCAR). This dataset comprises monthly averaged global surface temperature data, representative of an atmospheric field. It is worth noting that the atmospheric fields are complex with spatial and temporal scales of variability that range from small-scale turbulence to large-scale weather systems and climatic patterns. The CESM2 pre-industrial control dataset encapsulates this complexity, providing a robust testing ground for our method.
The CESM2 model simulates interactions between the various components of the earth system, including the atmosphere, oceans, land surface, and sea ice. The pre-industrial control run is de- signed to represent stable climate without any anthropogenic influences, providing baseline for
13

---

Published as a conference paper at ICLR 2024
comparison against changes in future climate scenarios. The monthly averaged global surface tem- perature data from this run captures long-term climate variability, which is an essential factor in understanding and predicting future climate change. This dataset's rich spatial and temporal vari- ability, combined with its simulated nature, allows us to rigorously test the performance and robust- ness of our proposed INR method. These data are made available under the World Climate Research Program (WCRP)'s Coupled Model Intercomparison Project (CMIP) that coordinates independent model intercomparison activities and their experiments and uses a common infrastructure for collect- ing, organizing, and distributing output from models performing common sets of experiments. The U.S. Department of Energy's 's Program for Climate Model Diagnosis and Intercomparison (PCMDI) provides coordinating support and development of software infrastructure in partnership with the Global Organization for Earth System Science Portals to provide the data at
A.2 SATELLITE-BASED SEA SURFACE TEMPERATURE
The second dataset considered is the Group for High Resolution Sea Surface Temperature (GHRSST) Level 4 Multiscale Ultrahigh Resolution (MUR) Global Sea Surface Temperature (SST) dataset obtained from NASA's Earthdata servers. The GHRSST Level SST analysis is pro- duced as retrospective dataset (four-day latency) and near-real-time dataset (one-day latency) at the Jet Propulsion Laboratory (JPL) Physical Oceanography Distributed Active Archive Cen- ter (PO.DAAC) using wavelets as basis functions in an optimal interpolation approach on global 0.01 degree grid. The version 4 MUR L4 analysis is based upon nighttime GHRSST Level 2P skin and subskin SST observations from several instruments.'
Selected data are from the western North Atlantic region, spanning bounding box with the south- west corner at (34N, -69F) and the northeast corner at (44N, -59E). This oceanic dataset provides an entirely different perspective the occanic fields have distinct spatial and temporal scales of variability compared to atmospheric fields. Oceanic fields are characterized by a diverse range of spatial scales, from small-scale eddies and fronts basin-wide gyres and currents. Temporally, vari- ability can occur over short periods, such as tidal cycles, or over much longer periods, e.g., seasonal and interannual cycles. The GHRSST Level 4 MUR Global SST dataset captures this vast range of variability with the added complexity of being derived from ultra-high resolution satellite-based observations. This dataset's high resolution and comprehensivenes: makes an excellent candi- date to test the effectiveness of our proposed method in managing real-world data. NASA provides full and open access this data under its Earth Science Data Systems (ESDS) Program earth- data.nasa Lata.nasa.gov gov. One year of daily data from Aug. 20, 2022 to Aug. 20, 2023 is used at the provided resolution of hundredth of a degree in the region of Gulf Stream (as previously specified).
B EXPERIMENT DETAILS
B. TASKS
We establish a range of tasks with increasing complexity to primarily assess the models' perfor- mance in handling the randomness and sparsity inherent in the observation sensors. In tasks where the quantity of available sensors varies randomly across different time instances, this variation is achieved by sampling a subset of data points from a predefined grid. For example, with a 5% sam- pling rate, initially create a grid map with 5% of the data points then randomly sample from uniform distribution within the range of ([1,5, 5). resulting in different grid maps. Figure 8 provides a visual representation of the four tasks for evaluating model performance with increasing complexity.
'They include the NASA Advanced Microwave Scanning Radiometer-EOS (AMSR-E), JAXA Ad- vanced Microwave Scanning Radiometer 2 GCOM-W1, Moderate Resolution Imaging Spectroradiometer (MODIS) on the NASA Aqua and Terra platforms, U.S. Navy microwave WindSat radiometer. Advanced Very High Resolution Radiometer (AVHRR) on several National Oceanic and Atmospheric Administration (NOAA) satellites, and in situ SST observations from the NOAA iQuam project. Ice concentration data are from the archives at the EUMETSAT Ocean and Sea Ice Satellite Application Facility (OSI SAF) High Latitude Pro- cessing Center and are also used for improved SST parameterization for the high latitudes.
14

---

Published as a conference paper at ICLR 2024
Task fixed number of points and fixed observation locations
tz
4
Task random number of points and fixed observation locations
2000
f3
t.4
Task 3: fixed number of points and random observation locations
h1
l/ f3
14
Task 4: random number of points and random observation locations
t
t2
f3
000
Figure 8: Four field reconstruction tasks with increasing complexity for model evaluation.
B.2
MODEL TRAINING
Given the previously defined auto-decoder and Gabor filter-based network, we assume that the model's 's outputs adhere to a Gaussian distribution:
ultx, x)) ~ N Fo) 0)
(6)
where of noise variance and identity matrix. We assume the likelihood in the posterior calculation (Eq. ??) follows the form of pa (u()) zx; exp (-C (Fo æ), ()) The network prediction, represented ( is associated with its latent code # The loss function L(( (6) (8) penalizes deviations between the network prediction and the actual observation. During
training, we maximize the joint log posterior over all training instances T {lx IR ]
M
N
arg min 0
Σ
C (Fo (e) .d(1)
1 2
Each training instance is initialized with its respective latent code To optimize a specific latent code zg, we first map it to network parameters, calculate the reconstruction loss, and backpropagate this loss to 2ạ and other model parameters for gradient descent update. This gradient descent update is illustrated by the red and yellow lines in Figure 2. During training, this optimization is performed jointly. During inference, the network parameters are fixed, and our attention is solely
15

---

Published as a conference paper at ICLR 2024
optimizing the latent code x" to minimize the reconstruction error. Specifically, auto-decoders employ first-order optimization to solve arg max p(=" [u(*,1) u(+2)) ,... ). effectively using the op- p(="|u!*1))
timization process as an encoder that maps the new observation set Ut latent variables z". This
is equivalent to evaluating x* using Maximmum-a-Posterion (MAP) estimation:
arg min
Σ
C { (2+(22 xllhi) .u(titt)
(z*
o 11
(a(t) (0ti))))
Decoder initialization. Our experiments reveal that crucial factor to consider is the initialization of the y and μ parameters. Given that y effectively functions as an inverse covariance term for Gaussian distribution, a reasonable choice is to use a Gamma random variable I(a, 3), which the conjugate prior the Gaussian inverse covariance. On the other hand, we uniformly distribute over the allowable input space range ~ (mrmn Bmax).
B.3 MODEL ARCHITECTURE AND HYPERPARAMETERS
Designing network architectures and optimizing hyperparameters pose significant challenges in deep learning, often relying on empirical and goblem-sseccffe approaches. To ensure fair comparison, we conduct thorough hyperparameter tuning and architecture search for each model. Table 3 pro- vides detailed model configurations for ResMLP, SIREN, FFN+P/G, and MMGN (Multiplicative and Modulated Gabor Network).
Table Model configurations, hyperparameters, and associated ranges. Hyperparameters
Model
Values
Width
{128, 160, 192, 224, 256, 288, 320, 352, 384, 416, 448, 480, 512)
ResMLP
Depth
13, 4, 6, 7}
Non-linear activation
ReLU, Sigmoid, Tanh, SELU, GELU, Swish (128, 192, 256, 384,
Width
SIREN
Depth
{3, 4, 6,
Weight scale wpp
(1, 10, 15, 20, 25, 30, 35, 40, 45,50,
Width
{128, 192, 256, 384,512)
Depth
{3. 4, 5, 6,
FFN+P
Frequency constant
{30, 40, 50, 60, 70, 80, 90, 100, 110, 120) (150, 160, 170, 180, 190, 200, 210, 220, 230, 240, 250}
Frequency number
Non-linear activation
ReLU, Sigmoid, Tanh, SELU, GELU, Swish
Width
(128, 192, 256, 512,
Depth
{3.4. 4, 5, 6.
FEN+G
Gaussian
{1, 5, 7, 10, 20, 30, 40, 50)
Encode size
(64, 128, 256, 512]
Non-linear activation
ReLU, Sigmoid, Tanh, SELU, GELU, Swish )
Width
(128, 192, 256, 384, 512)
Depth
{3, 4,5, 6,
MMGN
Input scale
[128, 256, 512)
Latent size
[1,2,4, 8, 16, 32, 64, 128, 256, 512)
Latent initialization
Uniform, Gaussian, Ones, 7eros, Orthogonal
B.4 BASELINE IMPLEMENTATION
All models are trained with an L2 loss function for 200 epochs, employing the AdamW opti- mizer (Loshchilov & Hutter, 2019) with initial learning rate of 0.001. We apply learning rate decay of 0.99 for each parameter group of every epoch. Due to variations in resolution between earth simulation and satellite imagery data, set batch sizes 16 and 2, respectively. We conduct all experiments using PyTorch Lightning, repeating each experiment 10 times on a single NVIDIA A 100 40 GB GPU.
16

---

Published as a conference paper at ICLR 2024
C ADDITIONAL RESULTS
C.I EXTREME SPARSITY
The aim of this set of experiments is assess how different models perform when dealing with tremely limited spatial sensor coverage, typically less than 1%. This situation is common in various engineering and scientific upplications. For instance, oceanographic buoys and research vessels may have sparse spatial distribution in vast oceans, sometimes being hundreds of kilometers apart. Here, the focus is satellite-based data. Quantitatively, the proposed MMGN model outperforms other INR models when the sensor coverage is as low as 0.1%, achieving at least a 16. 74% improvement. This improved performance is evident in the visual representations presented in Figure 10. Notably, models like SIREN and FFN+P struggle to produce meaningful results in such sparse scenarios. While quantitatively less accurate than FFN+P, FFN+G exhibits better qualitative predictions, par- ticularly in Tasks 3 and 4. Interestingly, ResMLP appears provide more realistic predictions than models based on Fourier features. However, falls short in capturing high-frequency signals within the climate data and is sensitive to local irregularities, leading to substantial errors as observed in Task 4. In comparison to these baseline models, our proposed MMGN consistently delivers reliable reconstruction results even when working with extremely sparse data.
Reference
MMGN
ResMLP
SIREN
FFN+P
FFN+G
Task I: fixed number of points and fixed observation locations
Task random number of points and fixed observation locations
Task fixed number points and random observation locations
Figure 9: Comparison of reconstruction performance on ultra-high resolution satellite-based sea surface temperature data with sampling ratio 0.1%. The first, third, and fifth rows depict model predictions, while the second, fourth, and sixth rows illustrate the errors quantified by absolute enror. Darker pixels correspond to lower error values.
17

---

Published as a conference paper at ICLR 2024
Reference
ResMLP Task 4: random number points random observation locations
MMGN
SIREN
FFN+P
FFN+G
To further explore MMGN's ability to manage extremely sparse data, we conduct additional ex- periments with sampling ratios of 0.01%, 0.03%, 0.05%, 0.07%, and 0.09%. In addition to the mean squared error (MSE), we also compute the Peak Signal-to-Noise Ratio (PSNR) and the Struc- tural Similarity Index (SSIM) to assess performance. The results, illustrated in Figure 10, reveal substantial decline in performance when transitioning from 0.03% to 0.01% sampling ratios. How- ever, the sampling ratio increases from 0.03% to 0.09%, there is nearly linear improvement in performance. Therefore, to achieve a reliable reconstruction, especially for satellite-based data, minimum of 0.03% data coverage is needed.
PANR
11020
0.063
0.023
0.005
00000
0.033
0.946
0,0004
0.0022
0,0006
Figure Impact sampling ratios on MMGN performance in extreme sparse data scenarios.
C.2 CONVERGENCE STUDY
To determine the minimum number of sensors required for effective reconstruction, we conduct convergence study. The exhaustive grid search of sampling rates can computationally demanding. To expedite this process and establish connection with classical reconstruction methods, we ini- tially use proper orthogonal decomposition (POD). Specifically, for each dataset, we employ POD and sort the computed eigenvalues in descending order. Once cumulative energy is calculated, ap- proximately 750 POD modes are needed for the simulation-based data to reconstruct 90% of the total
Sumplig zate
0.032
0,004 0.006
0.008
1.0
0.016
0.014
0.3
Cuarlativv
MSE
MSE
POD
0.010
0.1
0.0
0.008
500
750
1000
1250
0.002
0.006
Modes
Sumpling rate
Figure 11: Cumulative energy (left) and reconstruction performance (right).
18

---

Published as a conference paper at ICLR 2024
kinetic energy (Figure 11). Notably, this calculation is based on the snapshot POD method (Taira et al., 2017; Sirovich, 1987). Using this information, we define an interval [0.1%, 1%] and conduct grid search estimate the minimal sampling rate. Both quantitative and qualitative findings in- dicate that 1% is the closest approximation to the minimum sampling rate for the simulation data Figure 12).
reference
8 s- 0.1%
0.3%
== 0.5%
= 0.7%
8 8= 0.9%
Figure 12: Convergence sampling in MMGN.
C.3 MODEL EFFICIENCY
Similar to Figure 6, we assess model efficiency by considering both inference speed and model size. Inference time is evaluated across all available data points for each instance with each model executed 10 times to unbiasedly evaluate the entire dataset. The resulting average inference speed for each instance then is computed. Notably, MMGN strikes an advantageous balance between accuracy and efficiency, emerging as the top-performing model in these evaluations. While FFN+G and SIREN exhibit faster inference speeds, they falter in delivering reliable results. In contrast, ResMLP stands out as the most computationally intensive model to run. While slightly faster than MMGN, FFN+P lags behind terms of accuracy.
Satellite- based data
Inference time per instance (sec)
Trainable params
0.12
500K
1.0UM
0.10
ResMLP
MMGN FFN-P
FF
0.06
SIREN
2
5
6
Performance ranking
Figure 13: Efficiency Comparison. Inference time is assessed across all available data points for each instance.
C.4 FULL QUALITATIVE RESULTS
19

---

Published as a conference paper at ICLR 2024
Reference
MMGN
ResMLP
SIREN
HEN+P
FFN+G
sampling rate
5%
sampling rate 25%
sampling rate
50%
Figure 14: Comparison of reconstruction performance for Task 1 simulation data from state- of-the-art climate model. The first, third, and fifth rows depict model predictions, while the second, fourth, and sixth illustrate the errors quantified by absolute error. Darker pixels correspond to lower error values.
20

---

Published as a conference paper at ICLR 2024
Reference
MMGN
ResMLP
SIREN
HFN+P
FFN+G
sampling rate
5%
sampling rate 25%
sampling rate
50%
Figure 15: Comparison of reconstruction performance for Task 2 simulation data from state- of-the-art climate model. The first, third, and fifth rows depict model predictions, while the second, fourth, and sixth illustrate the errors quantified by absolute error. Darker pixels correspond to lower error values.
21

---

Published as a conference paper at ICLR 2024
Reference
MMGN
ResMLL
SIREN
HFN+P
HFN+G
sampling rate = 5%
sampling rate 25%
sampling rate
50%
Figure 16: Comparison of reconstruction performance for Task 3 simulation data from state- of-the-art climate model. The first, third, and fifth rows depict model predictions, while the second, fourth, and sixth illustrate the errors quantified by absolute error. Darker pixels correspond to lower error values.
22

---

Published as a conference paper at ICLR 2024
Reference
MMGN
ResMLL
SIREN
HFN+P
FFN+G
sampling rate
5%
sampling rate 25%
sampling rate 50%
Figure 17: Comparison of reconstruction performance for Task 4 simulation data from state- of-the-art climate model. The first, third, and fifth rows depict model predictions, while the second, fourth, and sixth illustrate the errors quantified by absolute error. Darker pixels correspond to lower error values.
23

---

Published as a conference paper at ICLR 2024
C.5 MODEL EXPLANATION
Figure 18 shows comprehensive results regarding the influence of latent size by comparing the simi- larity of latent variables across all models. Note the experiments are conducted using the simulation- based global surface temperature data with sampling ratio of s 5% for demonstration. We trained 10 models with varying latent sizes, specifically, d2 E (1, 2, 4, 8, 16, 32, 64, 128. 256, 512), while keeping the number of time steps constant at & 1024. For instance, when utilizing a model with a latent size of 32, it operates on a 32-dimensional latent vector corresponding to the 1024 time steps. Thus, matrix of latents for 10 models is formed, where column (of size M x 1) represents a dimension of latent vectors throughout all time steps (we call it a latent variable), and a row (of size 1 x 1023) concatenates the latents of all models in time step (we call them latent vectors). Concretely, the assembled matrix can be represented as:
Z = (2 (2,11) (2222)
(512,1) (512.2) (512,512)
Next, we use t-SNE to reduce the dimension of all the latent variables Z and project them into two-diemsional (2D) space to understand the similarity of latent variables. Figure 18 illustrates the distribution changes of all latent variables. Compared with the distribution of latent size 128, the cases with lower latent sizes (e.g., 2, 4, and 8) show sparse sampling at nearby variables. Their latent spaces simply are the subspaces of the latent space spanned by 128 variables. When the size increases, the sampling keeps improving until the case with the size of 512 overly samples the space. Hence, we find the necessity of increasing the latent size an appropriate granularity. However, the question arises: What constitutes an ideal latent size? The final row of the figure serves as a useful illustration in this regard. We begin to observe overlap between latent sizes of 64 and 128. This overlap becomes more pronounced at latent size 256. Consequently, while increasing the latent size may enhance overall coverage, it also leads to information redundancy and overlapping areas.
128
128
128
128
128
128
32
128
128
128
64
256
512
Figure 18: The impact of latent size by comparing the similarity of all the models' latent variables.
To additionally demonstrate the influence of latent variables, we systematically ablate one latent variable at time to regenerate the entire data and assess the performance quantitatively using vari-
24

---

Published as a conference paper at ICLR 2024
metrics (i.e., MSE, PSNR, and SSIM). In Figure 19, Figure 20, and Figure 21, the results of all cases of latent sizes are concatenated along the x-axis, where the x-coordinate the index of the ablated latent variable. To facilitate the comparison, lation with the prefix "o" before the error metric names. Observations indicate that larger latent sizes enhance reconstruction quality, individual contributions diminish latent sizes increase, and dif- ferences in contribution exist among latent variables within the same space but are less pronounced compared disparity across latent sizes.
show the original emror metrics without ab-
0.837
00:52
0502
0.833
0.83
0:00
1.00
510
1.00
M45
3155
0.000
0.00
0:30
0.00
Figure 19: The ablation study of latent variables with MSE and OMSE.
1930
PS3B
15301
1550 100003
1557
1550 07003
07903
PSAR OPNO01
Figure 20: The ablation study of latent variables with PSNR and OPSNR.
1..0
HiV
1013
1004
1.23
0.20
0.79
0.02
0:03
0.50
0.00
-0.2
034
15
14
2342
09531
-00
300
10
300
Figure 21: The ablation study of latent variables with SSIM and OSSIM.
25


Reviews:
[{"review_id": "57kduklcyi", "reviewer": "Reviewer_MqB4", "summary": "The authors propose a new implicit representation for spatio--temporal fields. The representation is based on separately processing temporal information via an encoder--decoder architecture (the decoder is an auto-decoder). This strategy, combined with other ingredients such as Fourier (here Gabor) features and multiplicative filter networks yields strong empirical performance on a synthetic climate dataset.", "questions": "- in experiments where you recover fields on the surface of a sphere, are you using Cartesian or spherical polar coordinates?\n\n- why not include (different levels of) random noise in observations? this is an important test of robustness\n\n- I am not sure that (Izacard et al., 2019) is the right reference for sparse seismic networks and small earthquakes (or even for sparse spatial coverage in scientific data; it addresses a super-resolution problem)", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- the authors skillfully combine several existing techniques into a new architecture\n\n- compared with the baselines presented in the paper, the proposed method achieves remarkably strong performance\n\n- the idea to use an auto-decoder is creative and nice\n\n- the paper is well illustrated (for example, Figure 2 is very helpful in understanding the architecture)", "weaknesses": "- I find the paper very hard to read. Notation and expressions in Section 3 and the last paragraph of Section 2 are confusing and it takes a long time to understand what is going on (Figure 2 helps here but the math could be clearer). Another reason is very florid prose which I'd hesitate to use in a machine learning paper. Examples: \"The endeavor to tackle the conundrum of global field reconstruction from sparse observations is currently shaping two distinct paradigms of thought\", \"raging ocean waves\", \"intricate physical fields\" (what are they?), \"we present an _innovative_ neural network approach\", \"... a singular viable option\", ...\n\n- While the narrative insists on strong novelty, all ingredients (and various combination thereof) are well known. The separation of variable idea is based on Dona et al 21, multiplicative filter networks are known, local activations (such as Gabor or wavelet) have been used in many papers, e.g. https://openaccess.thecvf.com/content/CVPR2023/html/Saragadam_WIRE_Wavelet_Implicit_Neural_Representations_CVPR_2023_paper.html. In some sense the present paper is a multiplicative filter network with a separate treatment of time through an encoder--decoder. This is perfectly fine but in my view it limits the novelty, and there is no shortage of all sorts of implicit networks.\n\n- In real fields space and time _are_ intertwined so one needs to go beyond the rank-one decomposition f(x) g(t); Dona et al mentions a low rank version \\sum_k f_k(x) g_k(t). It is not clear to me from the current description how this required expresiveness is achieved. It would be nice to explain that more clearly.\n\n- Generally, there is a lack of insight on why this strategy is supposed to perform well and why other approaches are supposed to perform poorly. This is a more general problem: there are now hundreds of implicit networks with different hyperparameters, training strategies, etc, and very little theoretical or other understanding of their mechanics. \nWhat if you compare your approach with an optimally tuned adaptive kernel estimator? How about spline interpolation? How about a simple low pass filter? It will surely not mess up as badly as ResMLP in Figure 3.\n\n- Along the same lines, in datasets description you mention that wavelets were used for initial interpolation of satellite-based data on a grid; why not compare the described method with a sparse wavelet interpolation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vanlTEvhZc", "reviewer": "Reviewer_A6YW", "summary": "This paper proposes an INR–based method to reconstruct continuous fields from sparse sensor data for the application of climate modeling. The proposed method mainly introduces a latent variable to capture temporal information. The experiments are conducted on both simulated and real data.", "questions": "Please see weakness for the details of questions.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written and organized, which is easy to follow.\n- The motivation and target application of the proposed method is clear and convincing.\n- The proposed method introduces a latent vector z to incorporate with INR decoder, which sounds interesting since it seems to make the INR model can be generalized across different images/samples by changing or optimizing latent vectors.\n- The experiments are conducted on both simulated data and satellite-based data with a clear comparison with baseline methods.", "weaknesses": "- About latent code z. The paper claims to use the trainable latent code to represent temporal information instead of using time index. What is the motivation for this? Because the recorded time index is not reliable in this application? Or the latent code can also represent other semantic information? Previous works have shown INR can represent spatial-temporal information in a good way by adding temporal index as input coordinates. \n\n- As described in the paper, during training, both latent code and INR are trained. While during training, INR model parameters are fixed and only latent code is optimized. Since the given information is quite sparse during testing, how reliable it can be to optimize latent code? How robust this method can be when generalized across different objects? By only changing the latent code in the input of INR decoder and freezing all other model weights, can this change the decoded image in an efficient way?\n\n- For the comparison with baseline methods as in Table 1, it seems all the baseline methods use a different encoding than the proposed model, which uses Gabor filters over Fourier bases to transform the coordinates as Fourier transforms emphasize a global frequency representation. Usually, the INR models are quite sensitive with different encoding methods. Can the paper also report the comparison with baseline methods using the same Gabor filters as encoding to disclose how impactful the encoding method is? Similar with the backbone model architecture, are all the comparisons based on the same backbone model? Only in this way, it may be fair to say how efficient the latent code can be in this case.\n\n- The compared methods are all the object-specific or scene-specific models, while the proposed model seems to be a generalized model across multiple images or objects. The method may also need to be compared with: (1) other generalized models based on INR such as SRN [1], etc. (2) other spatial-temporal models using time index such as D-NeRF [2].\n\n- The visualization of latent code is quite interesting but it is not clear what it means for the Figure 5. Can the author explain more about what information is learned or embedded by the latent code from these plots?\n\n[1] Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations, NeurIPS 2019.\n\n[2] ​​D-NeRF: Neural Radiance Fields for Dynamic Scenes, CVPR 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Dc4dDFU4xx", "reviewer": "Reviewer_8E6g", "summary": "Authors consider reconstruction of continuous fields from sparse data, which has important applications in many real world problem (as authors states). They propose a new approach based on implicit neural presentations (INRs) and apply it to two climate model related datasets.", "questions": "Questions and comments:\n\n- Problem setup: They assume that the system is governed by a PDE (1). I don't see any reason why this assumption is made, or is that assumption used somewhere?\n\n- Section 2.1, \"Objective\" (and through the paper): They use notation $\\mathcal U_k=(u(t_k,x)|\\mathcal X_k)_{x\\in X_k}$ . What this \"$|\\mathcal X_k$\" stands in the set? $|$ usually is used for conditioning but I cannot see such a thing here. I would rather remove this extra part here to avoid confusion (I use '(' and ')' instead of curly brackets as this form does not seem to render those (and actually, strictly speaking, '(' and ')' would be preferred if order of measurements have meaning $\\mathcal U_k$))\n\n- The section 2.1 does not talk about noise or randomness and formulations in this section would indicate that $u(t,x)$ is deterministic. However, later sections seems to talk about noise and assume that $u$ is includes noise or is random (e.g. Eq (5) which does not have meaning if $u$ is deterministic). This is contradicting. I would suggest authors that authors would use separate notations for a noiseless quantity $u$ and the measurements, e.g., say something like $y_k$ is a vector of measurements or $u(t_k,x)$ plus noise. \n\n- Section 2.3, \"Neural pinpointing\". They mention that learning-based methods can be computationally demanding. But there are two sides in this: train time computationally complexity and run time/inference computational complexity. GANs have high training complexity but inference may not be soo much, and perhaps even less as the proposed INR depending on the architectures. \n\n- Section 2.4, \"$\\alpha=g(x)\\circ \\eta(t)$. The standard meaning for $\\circ$ is composition of functions, i.e. $(f\\circ g)(x)=f(g(x))$. Here this probable is not the case as I cannot see how you would compose $g$ and $\\eta$ in this case. Of course, authors can define their own notations, but then they should say that how they use the notation. If $\\circ$ is meant to be the multiplication, I would just remove $\\circ$. \n\n- Section 2.4, $\\eta(t_k):\\mathcal U_k=(u(t_k,x)|\\mathcal X_k)_{x\\in \\mathcal X_k}\\mapsto z_k$. Not sure what this is meant to mean. $\\eta$ is a function which maps $t$ to a function which is a mapping between the values of $u$ to $z_k$? \n\n- Section 3 seems to be a list of different things and it not clear to me that how the things are connected. For example, how you use auto-decoder? You have (auto-)decoder as encoder? How Gabor filters are used? Are these somehow connected to $g$ and $\\eta$ earlier? Multiplicative filter network is decoder?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Numerical results looks hood. Their methods seems to provide slower reconstruction accuracy (measured in terms of MSE) thank other INR approaches.", "weaknesses": "Presentation is quite a week and I have to say that I cannot follow it and understand what they are doing. First, mathematical notation is ambiguous and does not seem to follow standard mathematical principles, which creates confusion (Examples are in Questions section). Furthermore, background and description seems to be more like a list of different things and without clear connection. To make presentation more clear, authors could perhaps try to express their method with less mathematically rigours notations, but explaining their method with words (e.g. by providing example) \n\nAs mentioned above, numerical results looks good when compared to other INR methods. However, I would like to see also comparison to other non-INR based methods such as GP (if applicable) or methods that are currently used in meteorology and climate studies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C8Zt8LFFVP", "reviewer": "Reviewer_xFQb", "summary": "This paper proposes a new method to reconstruct spatiotemporal dynamics from sparse measurements by considering implicit neural networks. The spatiotemporal dynamics are separated into two independent spatial and temporal components. Superior performance has been shown in this paper compared with baseline models.", "questions": "- The 4th line of Page 6, “we uniformly distributing” should be “we uniformly distribute”.\n\n- How do the authors determine the sparse setting (Page 7) for those two datasets? The sparsity ratios for those two datasets are quite different. Also, can the authors provide an estimate or convergence analysis of the smallest number of sensors needed for reconstruction? \n\n- Why not consider the ERA5 dataset for climate modeling since the recent climate modeling papers [1-3] all use this benchmark dataset?\n\n- What are the spatial resolutions for the datasets used in this paper?\n\n---\n\nRefs:\n\n[1] Bi, Kaifeng, et al. \"Pangu-weather: A 3d high-resolution model for fast and accurate global weather forecast.\" arXiv preprint arXiv:2211.02556 (2022).\n\n[2] Pathak, Jaideep, et al. \"Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural operators.\" arXiv preprint arXiv:2202.11214 (2022).\n\n[3] Lam, Remi, et al. \"GraphCast: Learning skillful medium-range global weather forecasting.\" arXiv preprint arXiv:2212.12794 (2022).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper proposes a new pipeline by combining implicit neural networks and functional separation of variables. In addition, Gabor filters are applied for learning high-frequency signals. \n- This paper is well-written and easy to follow.", "weaknesses": "- The dataset is not that challenging. This paper considers monthly-averaged temperature data in **Simulation-based data** CESM2. The dynamics are relatively smoother than daily-averaged data. Please see some of my follow-up concerns regarding the dataset in **Questions**. \n\n- The sparsity setting considers some random sampling schemes. If the authors can discuss a bit on the optimal sensor placement, it would enhance the paper.\n\n- Since the authors use the Gabor filter instead of other common schemes to learn high-frequency signals, it would be good to have an ablation study on the Gabor filter part.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)