Title: Accurate Interpolation for Scattered Data through Hierarchical Residual Refinement
Venue: NeurIPS 2023
Accepted: yes
Keywords: Interpolation algorithm

scattered data

deep learning

residual learning
Abstract: Accurate interpolation algorithms are highly desired in various theoretical and engineering scenarios. Unlike the traditional numerical algorithms that have exact zero-residual constraints on observed points, the neural network-based interpolation methods exhibit non-zero residuals at these points. These residuals, which provide observations of an underlying residual function, can guide predicting interpolation functions, but have not been exploited by the existing approaches. To fill this gap, we propose Hierarchical INTerpolation Network (HINT), which utilizes the residuals on observed points to guide target function estimation in a hierarchical fashion. HINT consists of several sequentially arranged lightweight interpolation blocks. The first interpolation block estimates the main component of the target function, while subsequent blocks predict the residual components using observed points residuals of the preceding blocks. The main component and residual components are accumulated to form the final interpolation results. Furthermore, under the assumption that finer residual prediction requires a more focused attention range on observed points, we utilize hierarchical local constraints in correlation modeling between observed and target points. Extensive experiments demonstrate that HINT outperforms existing interpolation algorithms significantly in terms of interpolation accuracy across a wide variety of datasets, which underscores its potential for practical scenarios.

Accurate Interpolation for Scattered Data through Hierarchical Residual Refinement

Abstract

Accurate interpolation algorithms are highly desired in various theoretical and engineering scenarios. Unlike the traditional numerical algorithms that have exact zero-residual constraints on observed points, the neural network-based interpolation methods exhibit non-zero residuals at these points. These residuals, which provide observations of an underlying residual function, can guide predicting interpolation functions, but have not been exploited by the existing approaches. To fill this gap, we propose H ierarchical INT erpolation Network ( HINT ), which utilizes the residuals on observed points to guide target function estimation in a hierarchical fashion. HINT consists of several sequentially arranged lightweight interpolation blocks. The first interpolation block estimates the main component of the target function, while subsequent blocks predict the residual components using observed points residuals of the preceding blocks. The main component and residual components are accumulated to form the final interpolation results. Furthermore, under the assumption that finer residual prediction requires a more focused attention range on observed points, we utilize hierarchical local constraints in correlation modeling between observed and target points. Extensive experiments demonstrate that HINT outperforms existing interpolation algorithms significantly in terms of interpolation accuracy across a wide variety of datasets, which underscores its potential for practical scenarios.

1 Introduction

Scattered data interpolation aims to estimate the unknown values of a continuous function at some target points based on a finite set of observed data points [11]. This technique plays a significant role in extensive theoretical and engineering domains, e.g., partial differential equations (PDEs) solving, physical analysis and thermal engineering, etc [7, 20, 28, 2]. For instance, in thermal engineering, scattered data interpolation is employed to estimate the environmental temperature distribution of buildings [28] and electronic devices [2] based on temperature sensor readings obtained at dispersed points. In many applications, the function distributions are complex and the scattered points are non-uniformly distributed, which poses significant challenges for accurate interpolation algorithms.

Traditional scattered data interpolation algorithms follow a well-established pipeline: firstly a suitable set of basis functions is prepared ( e.g. polynomial, Spline and radial basis functions), then a possible target function is generated by constructing a linear combination of these basis functions [11]. Basis function are selected depending on application scenarios, which highly relies on domain knowledge and human experience. This makes traditional algorithms hardly able to generalize between problems

K =8

BLOCK

MAIN

BLOCK

K =16

RESIDUAL

1

1

BLOCK

RESIDUAL

Interpolation task

Interpolation result

Main component

Residual component

Residual component

Hierarchical local

L-1

K   =4

0

L-1

constraints

and imposes much burden of searching and tuning. Thus, when targeting slightly more complex functions, these methods usually present limited interpolation accuracy.

Recently, the evolution of deep learning technology [17, 15, 16, 12, 3] has given rise to a plethora of neural network-based interpolation algorithms. Notable examples of such models encompass Neural Processes [8, 14, 19, 18], TFR-Transformer [2], and NIERT [4]. These models leverage neural network modules to capture the correlation between observed points and target points, and estimate the target point values based on the correlations and the observed point values. In contrast to traditional methods that rely on hand-crafted and rigid basis functions, these neural interpolators can adaptively discern the underlying distribution of the target function from a function set or scattered data set, consequently achieving superior generalization performance.

Different from the traditional interpolation algorithms that have exact zero-residual constraints on observed points (target function curve passes through the points) [11], the neural networks-based approaches exhibit non-zero residuals at these points. We assume these residuals represent a type of residual function, which maintains residual information sharing between observed points and target points. Although existing methods utilize the correlations between observed points and target points as conditions, few methods explore the use of residual information sharing between them.

To fill this gap, we introduce H ierarchical INT erpolation network ( HINT ) to utilize observed points residuals to guide target function estimation in a hierarchical fashion (depicted in Figure 1). Our key insight is that the target function estimation can be decomposed into a main function component and several residual function components. With this in mind, we design the framework of HINT as a hierarchy of lightweight interpolation blocks, among which the first block (main block) estimates the main component of the target function and subsequent blocks (residual blocks) predict the residual components in a progressively refining manner. In each residual block, to utilize residual information sharing between observed and target points, we predict the residuals of target points from accessible observed point residuals and via correlation modeling between two point domains. Finally, the main component and residual components of target points are accumulated to form interpolation results.

Furthermore, we consider that finer residual prediction for a target point requires more focused attention on its spatially neighboring observed points. Thus, we incorporate hierarchical local

constraints that progressively narrows down the valid range of observed points correlated with target points, to reduce the noise of correlation modelling during residual prediction.

We summarize our contributions as follows: 1) We propose a novel hierarchical residual refining framework, HINT, for scattered point interpolation. We introduce a series of residual blocks that utilize the residual information of observed points to guide target points prediction via correlation modeling in a coarse-to-fine fashion, to enhance interpolation accuracy. 2) We impose hierarchical local constraints on the range of observed points correlated with target points to conform to hierarchical residual scales, thereby further improving interpolation robustness. 3) We validate our framework on four scattered data interpolation task sets across diverse theoretical and application scenarios, which demonstrates our HINT surpasses existing state-of-the-art interpolation methods in terms of accuracy in various tasks consistently.

2 Method

2.1 Problem Statement and Notations

Given n observed points with known values, denoted as O = { ( x i , y i ) } n i =1 , and m target points with values to be estimated, represented by T = { x i } n + m i = n +1 , the objective of the interpolation task is to accurately estimate the values f ( x ) for each target point x ∈ T , where f is the underlying function determined by the information from the observed points in O .

Here, the position of a point is indicated by x i ∈ R d x , while the value of a point is signified by y i = f ( x i ) ∈ R d y . The function mapping is denoted by f : R d x → R d y , with d x and d y representing the dimensions of a point's position and value respectively. The function f originates from a function distribution, which can be implicitly represented by a set of scattered data.

For convenience, we denote the positions of the observed points as x O = { x i } n i =1 , the values of the observed points as y O = { y i } n i =1 , and the positions of the target points as x T = { x i } n + m i = n +1 .

2.2 Hierarchical Residual Refining Framework

The architecture of our proposed HINT is illustrated in Figure 1. Our key insight is the target function estimation can be decomposed into a series of function components at different scales, including a main function component and several residual components. In light of this view, we design a hierarchical interpolation architecture, consisting of a series of L neural network interpolation blocks. The first block (main block) serves as a coarse interpolator, estimating the main component of the target function, and subsequent L -1 blocks (residual blocks) function as finer interpolators, estimating the residual parts of the target function.

The main block takes the positions and ground truth function values of the observed points, x O and y O , along with the positions of the target points x T , as inputs. It outputs the coarse function value predictions at both the observed and target points ˆ y O (1) , ˆ y T (1) .

Then we can immediately obtain the estimated residuals of the target function at the observed points:

After that, the following L -1 residual blocks predict the residual components of the target function in a progressively refining manner. Specifically, each residual block takes the residual of the previous block's predicted function values at the observed points as input and outputs the residual function values at both the observed and target points. The predicted residual function values at the observed points are used to compute the remaining residuals for the next residual block, while the predicted residual function values at the target points are employed to calculate the final function estimation values at the target points:

Transformer Encoder

Attention mask only retaining observed points'

Embedding

stage

Correlation

modeling

Prediction

stage

impact and integrating KNN graph constraint

Ultimately, we obtain the final interpolation function prediction by accumulating the initial main components and subsequent residual components:

2.3 Transformer-based Interpolation Block

Here we elaborate on the structure of our interpolation block illustrated in Figure 2, which is embodied as the main block and residual blocks of HINT. The l -th interpolation block takes the positions and values of the observed points x O and y ( l ) O , and the positions of the target points x T as inputs, and predicts the values of the target function at both the observed points and target points, i.e., ˆ y O ( l ) and ˆ y T ( l ) . To facilitate the implementation of our hierarchical residual refining framework, we propose a novel encoder-only interpolation block building upon the basic structure of NIERT block [4], a state-of-the-art neural interpolation model. Please refer to Supplementary Material for a detailed description of NIERT. Specifically, we make two key modifications to the original NIERT block: 1) A set of 'masked observed' points are introduced to solve the problem of inconsistent distribution between observed points and target points. 2) Local constraints are imposed on the range of observed points for target points in correlation modeling to accommodate different scales of the output function value. We introduce our interpolation block in detail as follows.

Embedding Stage. In the embedding phase, each point's position and value are separately embedded and concatenated to form the point's overall embedding. Following [4], we introduce a learnable token vector MASK y to embed the absent value for target points. Although this avoids structural discrepancy between observed points and target points, there still exists the problem of inconsistent distribution between them. Moreover, observed point residuals and target point residuals serve as the input and output of the block, respectively. Thus consistent distribution between them helps the model to compute residuals more easily. To this end, we introduce masked observed points, which are the replicates of observed points without values. This simulates the absence of value in target points for observed points, ensuring the distribution of them is consistent with target points. After this replication operation. the embeddings of three sets of points can be represented as:

where i -n denotes index shifting.

Correlation Modeling. Then the embeddings of the points are fed into a stack of P Transformer layers, to model the correlations between observed points and target points, and produce resultant points encodings. Through the p -th transformer layer, the encoding of the i -th point h p i are transformed to h p +1 i . Following NIERT [4], we use attention mask mechanism on vanilla self-attention in each Transformer layer, to avoid the unexpected interference of target points on observed points. Thus, all three set of points are treated as queries { q p i } 2 n + m i =1 , while only observed points are treated as keys { k p i } n i =1 and values { v p i } n i =1 , with these concepts referring to [27].

Then, the unnormalized attention score e p ij can be computed via scaled dot product e p ij = q p i · k p j √ d k . Here, d k is the dimension of the query and key vectors. Based on prior knowledge that the observed points near the target points have a greater impact on the interpolation of target points than distant ones [7, 4], we impose controllable local constraints on the attention mask to accommodate different scales of residuals. We retain only the influence of the K nearest observed points (under the Euclidean distance metric) of the i -th target point (we denote N K ( i ) here), masking out the attention scores of distant observed points (see Figure 2):

where K is a hyperparameter for local constraints. Then the attention score can be obtained by softmax normalization α p ij = exp (ˆ e p ij ) ∑ n j =1 exp (ˆ e p i,j ) . With reasonable K , this design helps lower down noise in correlation modelling, thus improves interpolation accuracy.

Estimating stage. For each point i ∈ [1 , 2 n + m ] , we estimate its value ˆ z i through feeding its representation at the final Transformer layer into an MLP, ˆ z i = MLP out ( h P i ) . These values are divided into three parts, including 1) predicted values of target function at target points ˆ y T ( l ) = { ˆ z i } 2 n + m i =2 n +1 , 2) predicted values of target function at 'masked observed' points ˆ y O ( l ) = { ˆ z i } 2 n i = n +1 , and 3) auxiliary reconstruction of observed points ˜ y O ( l ) = { ˆ z i } n i =1 . We use ˆ y O ( l ) rather than ˜ y O ( l ) to compute residuals y ( l ) O -ˆ y O ( l ) as the input of next residual block. This is because ˆ y O ( l ) have consistent distribution with ˆ y T ( l ) , which helps the model conveniently utilize sharing residual information between two types of points in to predict ˆ y T ( l ) . We use ˜ y O ( l ) for auxiliary loss function optimization.

2.4 Hierarchical Local Constraints

We expand local inductive bias [7] of interpolation in a hierarchical way. We assume the main block focuses on relatively global information for it estimates the primary components of the target function, while residual blocks focus on local information for it estimates the residual components with smaller scales. Considering this, we impose hierarchical local constraints on the interpolation blocks by utilizing K-nearest neighbors constraints with decreasing K . Specifically, we establish a simple rule, setting two hyperparameters, K (0) and K min , where K (0) determines the K value for the main block, and K min represents the minimum K value for the residual blocks. We recursively set the K value for the l -th residual block as K ( l ) = max ( K ( l -1) / 2 , K min ) (see Figure 1). It is worth noting that the specific values for K (0) and K min depend on the characteristics of the dataset.

2.5 Loss Function

The loss function consists of two components: one for optimizing the interpolation error L int , and the other as an auxiliary term L aux , i.e. ,

where L int = Error( ˆ y T , y T ) is the primary objective and L aux = 1 L ∑ L -1 l =0 Error( ˜ y O ( l ) , y ( l ) O ) is the auxiliary reconstruction term. The primary objective is to minimize the discrepancy between the predicted function values and the ground truth at the target points. And the auxiliary term serves to constrain the auxiliary reconstruction at observed points for each interpolation module, facilitating

the accurate reconstruction of both observed and target points. Here, The error function used in this context can be either Mean Squared Error (MSE) or Mean Absolute Error (MAE), depending on the precision metric requirements of the specific scenario. λ is the loss balance hyperparameter.

3 Experimental Setting

We evaluate the interpolation performance of HINT on four representative datasets and compare it with existing state-of-the-art approaches. We describe these datasets, the baselines used for comparison and the experimental setup as follows. Please refer to the Supplementary Material for more details.

3.1 Datasets

The four datasets are Mathit-2D and Perlin for synthetic function interpolation, TFRD for temperature field reconstruction, and PTV for particle tracking velocimetry. We provide a brief introduction to these four datasets, and further details and statistical information can be found in the Supp.

Theoretical Dataset I: Mathit-2D . Mathit-2D [4] contains interpolation tasks derived from twodimensional synthetic symbolic functions. 512 data points are sampled within the square region [ -1 , 1] 2 and are randomly divided into observed and target points to form an interpolation task. The number of observed points is randomly chosen from the range [10, 50].

Theoretical Dataset II: Perlin. To evaluate the interpolation performance on functions with fine details, we additionally constructed the Perlin dataset, which is sampled from random two-dimensional Perlin noise functions [23]. Each interpolation task involves randomly sampling from a Perlin function defined on [ -1 , 1] 2 , resulting in 64 observed points and 192 target points.

Application Dataset I: TFRD. TFRD [2] is a two-dimensional interpolation dataset for temperature field reconstruction. It contains three sub-datasets, ADlet, DSine, and HSink corresponding to different field boundary conditions. Each temperature field has 40,000 data points, with 37 points serving as observed points and the remaining points as target points to form interpolation tasks.

Application Dataset II: PTV. PTV [4] is a scattered dataset for real-world two-dimensional particle tracking velocimetry, where each instance corresponds to a set of particles moving in a velocity field extracted from the images of laminar jet experiment scenes [25]. Each instance contains 2,000 to 6,000 data points, from which we randomly select 512 points as observed points, while the remaining points serve as target points for prediction.

3.2 Baselines

The baselines for evaluation include: 1) Conditional Neural Processes (CNPs) [8] utilize MLPs to predict function distributions given observed points. 2) Attentive Neural Processes (ANPs) [14] leverage attention mechanisms to enhance CNPs' performance. 3) Bootstrapping Attentive Neural Processes (BANPs) [19] which employ the bootstrap technique to further improve ANPs' performance. 4) TFR-Transformer [2] incorporates typical encoder-decoder-style Transformers model correlations between observed points and target points. 5) NIERT [4] leverages a Transformer encoder-only architecture to unify the processing of observed and target points in the same feature space to obtain better interpolation performance.

3.3 Experimental Setup

Hyperparameter Setting. We carefully select and tune several hyperparameters of HINT, including the number of interpolation blocks L , the number of attention layers in each block, the local parameter K 0 , and the minimum local parameter K min . For the Mathit-2D dataset, we set L = 2 , with 6 attention layers in the main block and 2 attention layers in the residue block. As for Perlin, PTV, and TFRD datasets, we set L = 4 with 2 attention layers in each block. The choice of L was based on prior research and our experimental observations. To ensure a fair comparison, we maintained consistency between HINT and the baseline models, NIERT and TFR-Transformer, in terms of their key hyperparameter settings. Specifically, we keep the number of attention layers, the number of attention heads, and the dimensionality of the hidden space, consistent across all these models.

Regarding the local parameter K (0) , we set it to the number of observed points in the interpolation task for Mathit-2D, Perlin, and TFRD datasets, as these datasets have a relatively small number of observed points. For the PTV dataset, we set K (0) to 128 to accommodate its larger number of observed points. These values were selected to balance the local constraints with the available information in each dataset. Additionally, we set the minimum local parameter K min to 8 for all datasets. This value was chosen to ensure a sufficient level of local modeling and capture fine-grained details in the interpolation process.

Evaluation metric. Following the precedent set by previous works [2, 4] and for the purpose of facilitating comparison, we adopted the MSE as the evaluation metric for Mathit-2D, Perlin, and PTV datasets. Additionally, for the TFRD dataset, we utilized the MAE as the evaluation metric.

4 Experimental results

4.1 Interpolation Accuracy

We present comprehensive comparisons of test set average interpolation accuracy of our interpolation approach, HINT, with state-of-the-art approaches on the four representative datasets. On the Mathit2D dataset, we observe a notable enhancement of 8.34% in accuracy compared to the best method NIERT (Table 1). Similarly, on Perlin and PTV datasets, our method demonstrates remarkable accuracy gains of 30.23% and 18.61% respectively (Tables 2 and 3). Furthermore, on three subsets of the TFRD dataset, i.e. , HSink, ADlet and DSine, our method exhibits substantial accuracy gains of 41.50%, 49.29% and 44.08% (Table 4) over NIERT. These compelling results validate the efficacy of our approach on interpolation tasks across diverse datasets, highlighting the superiority and versatility of our algorithm in both theoretical and real-world application scenarios.

4.2 Qualitative Analysis

Qualitative Comparison. Here we provide a more intuitive comparison of the interpolation performance between HINT and existing interpolation methods. We randomly select an interpolation task from the Perlin test datasets, visualizing the interpolation results and error maps of HINT and existing interpolation methods. As shown in Figure 3, for this interpolation task from the Perlin test set, our proposed HINT achieves the highest interpolation accuracy (MSE × 1 . 04 × 10 -4 ). The error map of BANP exhibits strong error hotspots. TFR-Transformer and NIERT still have relatively large accuracy difference regions, while HINT has the largest area with minimal discrepancies (dark blue regions) on its error map.

-1

0

1

-1

0

1

Ground truth

0.2

0.1

0.0

0.1

0.2

-1

0

1

-1

0

1

BANP

-1

0

1

-1

0

1

TFR-transformer

-1

0

1

-1

0

1

NIERT

-1

0

1

-1

0

1

0.2

0.1

0.0

0.1

0.2

-1

0

1

-1

0

1

Observed points

-1

0

1

MSE: 3.28e-4

-1

0

1

MSE: 1.79e-4

-1

0

1

MSE: 1.67e-4

-1

0

1

MSE: 1.04e-4

0.00

0.02

0.04

0.06

0.08

0.10

Error map

Interpolation

result

Interpolation

task

-1

0

1

-1

0

1

-1

0

1

-1

0

1

-1

0

1

-1

0

1

-1

0

1

-1

0

1

-1

0

1

0.00

0.05

0.10

0.15

0.20

-1

0

1

-1

0

1

-1

0

1

-1

0

1

-1

0

1

-1

0

1

-1

0

1

-1

0

1

0.2

0.1

0.0

0.1

0.2

-1

0

1

-1

0

1

-1

0

1

-1

0

1

-1

0

1

-1

0

1

-1

0

1

-1

0

1

0.2

0.1

0.0

0.1

0.2

Cumulation

error map

Cumulation

Main block

Residual block

1

Residual block

2

Residual block

3

Block output

-1

0

1

-1

0

1

-1

0

1

Residual Visualization. To investigate the role of each block in our proposed HINT, we visualized the output components of each block, as shown in Figure 4. We can find that the main block produces a relatively smooth overall function shape for the target function, lacking local details. The first residual block outputs more localized fluctuations in the target function with a smaller scale. The second residual block generates even smaller-scale components that appear to capture more local details and the last block contributes minuscule components. Observing the cumulative sum of the outputs from these blocks, we find that the accumulated function becomes progressively more detailed and closely resembles the ground truth of the target function as the outputs of the subsequent blocks are added. By examining the error maps of each cumulative sum, we observe that the output of the main block has strong error hotspots and subsequent blocks gradually refine the prior predictions until the final error map displays uniformly low errors throughout.

4.3 Ablation Study

Ablation on Main Design Choices of HINT. We validate the efficacy of the auxiliary loss term L aux , masked observed points, and local constraints in HINT, as shown in the upper section of Table 5. When the auxiliary loss term is omitted (equivalent to setting the weight λ of the auxiliary loss term L aux to 0), as indicated in the table as 'HINT w/o L aux ', HINT experiences a reduction in accuracy on all three datasets, with a notably significant impact on Perlin and TFRD-ADlet (decreasing by 30% and 125%, respectively). Omitting the local constraints ('HINT w/o KNN') also leads to decreased accuracy on these datasets, particularly affecting Perlin and PTV (with a decrease of 20% and 43%, respectively). Additionally, we assess the effectiveness of masked observed points ('HINT w/o masked obs.'). In this setup, instead of using ˆ y O ( l ) , we utilize ˜ y O ( l ) to compute residuals on the observed points. Experimental results in this configuration reveal a decline in accuracy across all three datasets, most notably on Perlin (a reduction of 55%), underscoring the significance of maintaining a consistent distribution between observed and target points. These findings highlight the advantages of incorporating these design choices within our framework.

The lower half of Table 5 presents the ablation results on different local constraints. For a straightforward evaluation, we employed two ablation settings where the local constraint parameter K was set to either K (0) or K (1) for all interpolation blocks. Here, K (0) represents the local parameter of the first block in the hierarchical local constraint, while K (1) represents the local parameter of the second block. These two ablation configurations of local parameters represent a non-hierarchical approach with similar hyperparameters to our hierarchical approach. These results demonstrate the hierarchical design of local constraints has significant superiority over the non-hierarchical one in our method.

Ablation on Interpolation Blocks Number. Table 6 presents the results obtained by varying the number of blocks from 1 to 5 on two datasets, Perlin and PTV. The results indicate that when there

is only one block (i.e., no residual block), HINT performs poorly on both datasets. However, with the introduction of a single residual block (i.e., two blocks in total), the interpolation performance of HINT improves significantly. This highlights the effectiveness of incorporating the interpolation module. As the number of blocks increases, the interpolation accuracy of HINT shows incremental improvements with smaller margins. When the number of blocks reaches 5, the interpolation accuracy of HINT shows a slight decline on the Perlin dataset and remains relatively stable on the PTV dataset. These results suggest that after a certain number of blocks, the additional blocks no longer contribute to the learning of HINT, and the interpolation accuracy plateaus.

5 Related Work

Traditional Approaches for Scattered Data Interpolation. Traditional interpolation algorithms for scattered data construct the target function through linear combinations of explicit basis functions. For one-dimensional scattered data, techniques such as linear interpolation, Lagrange interpolation, Newton interpolation [11], and spline interpolation [9] are commonly employed. For more general scattered data residing in higher-dimensional spaces, Shepard's methods [26], and Radial Basis Function (RBF) interpolation [24, 6] are widely used. MIR [29] employs the Taylor series for interpolation and determines the weights by optimizing the interpolation error. In engineering applications, hierarchical interpolation methods have also been developed [5], which are often combined with spatial partitioning techniques or the selection of nearby points to reduce computational complexity. All these traditional methods are inherently non-adaptive, as they don't learn the function distribution from data. This restricts their interpolation precision and hampers their generalization across diverse scenarios.

Learning-based Approaches for Scattered Data Interpolation. Recent neural network-based scattered data interpolation algorithms can be broadly categorized into two groups: probabilistic and deterministic. Probabilistic methods, such as the family of Neural Processes [8, 14, 18, 30, 19, 21], estimate the conditional distribution of the target function based on observed points. Deterministic methods, such as TFR-Transformer [2] and NIERT [4], estimate the target function based on observed points with a focus on interpolation accuracy. Technically, these approaches use neural network modules to model the correlation between observed and target points, subsequently obtaining the representation and estimation of target point values. For instance, CNPs [8] employs MLPs to model the correlation between observed and target points, while ANPs [14], TFR-Transformer, and NIERT utilize attention mechanisms and Transformer layers [27] for the same purpose. Among these methods, NIERT employs the architecture of the Transformer encoder to uniformly handle observed and target points, resulting in enhanced interpolation accuracy. It stands as the current state-of-the-art neural interpolation algorithm, serving as inspiration for our interpolation block design. Notably, these current neural interpolators have thus far neglected the information regarding interpolation residuals at observed points, which constitutes a shared limitation.

Residual Learning and Decomposition Structure. Residual connections in neural networks can be traced back to residual modules in ResNet [10] and DenseNet [13]. In this module, the output of each stack is fed into subsequent stacks, for better optimization of deep neural networks. For a different propose, in time series forecasting, N-BEATS [22] presents a doubly residual framework with two distinct residual branches: one branch for backcast prediction and another branch for the forecast prediction at each layer. The residual between the backcast prediction and history input is

exploited to forecast prediction. N-HiTS [1] integrates hierarchical sparse prediction into the similar doubly residual framework, to enhance both forecasting accuracy and interpretability. However, these techniques are primarily tailored for fundamental time series signal decomposition and display a notable limitation when tasked with learning sparse and scattered data interpolation. Although the exploitation of residual is successfully applied in time series forecasting, it is hard to apply in scattered data interpolation straight-forwardly for sparsity and irregularity of scattered data.

6 Conclusion

In this paper, we propose HINT, a novel hierarchical residual refining framework for scattered data interpolation. By employing a series of residual blocks in a coarse-to-fine fashion and incorporating hierarchical local constraints in residual correlation modeling, HINT effectively improves interpolation accuracy. Our extensive validation across diverse theoretical and application scenarios demonstrates the superior performance of HINT compared to existing state-of-the-art interpolation methods. We acknowledge the limitations of our approach regarding the hyperparameter K in hierarchical local constraints, and we plan to explore methods for learning adaptive K values in future work.

References

Cristian Challu, Kin G Olivares, Boris N Oreshkin, Federico Garza, Max Mergenthaler-Canseco, and Artur Dubrawski. N-hits: Neural Hierarchical Interpolation for Time Series Forecasting. arXiv preprint arXiv:2201.12886 , 2022.

Xiaoqian Chen, Zhiqiang Gong, Xiaoyu Zhao, Weien Zhou, and Wen Yao. A Machine Learning Modelling Benchmark for Temperature Field Reconstruction of Heat-Source Systems. arXiv preprint arXiv:2108.08298 , 2021.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pretraining of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805 , 2018.

Shizhe Ding, Boyang Xia, Milong Ren, and Dongbo Bu. NIERT: Accurate Numerical Interpolation through Unifying Scattered Data Representations using Transformer Encoder. arXiv preprint arXiv:2209.09078 , 2023.

Michael S Floater and Armin Iske. Multistep Scattered Data Interpolation Using Compactly Supported Radial Basis Functions. Journal of Computational and Applied Mathematics , 73(12):65-78, 1996.

Bengt Fornberg and Julia Zuev. The Runge Phenomenon and Spatially Variable Shape Parameters in RBF Interpolation. Computers & Mathematics with Applications , 54(3):379-398, 2007.

Richard Franke and Gregory M Nielson. Scattered Data Interpolation and Applications: A Tutorial and Survey. Geometric Modeling , pages 131-160, 1991.

Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and S. M. Ali Eslami. Conditional Neural Processes. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 1704-1713. PMLR, 10-15 Jul 2018.

Charles A Hall and W Weston Meyer. Optimal error bounds for cubic spline interpolation. Journal of Approximation Theory , 16(2):105-122, 1976.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 770-778, 2016.

Michael T Heath. Scientific Computing: An Introductory Survey, Revised Second Edition . Society for Industrial and Applied Mathematics, 2018.

Sepp Hochreiter and Jürgen Schmidhuber. Long Short-term Memory. Neural Computation , 9(8):1735-1780, 1997.

Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely Connected Convolutional Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 4700-4708, 2017.

Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive Neural Processes. In International Conference on Learning Representations , 2019.

Diederik P Kingma and Max Welling. Auto-Encoding Variational Bayes. In International Conference on Learning Representations , 2014.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 , 2016.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet Classification with Deep Convolutional Neural Networks. Communications of the ACM , 60(6):84-90, 2017.

Byung-Jun Lee, Seunghoon Hong, and Kee-Eung Kim. Residual Neural Processes. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 4545-4552, 2020.

Juho Lee, Yoonho Lee, Jungtaek Kim, Eunho Yang, Sung Ju Hwang, and Yee Whye Teh. Bootstrapping neural processes. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 6606-6615. Curran Associates, Inc., 2020.

GR3543275 Liu. An Overview on Meshfree Methods: For Computational Solid Mechanics. International Journal of Computational Methods , 13(05):1630001, 2016.

Tung Nguyen and Aditya Grover. Transformer Neural Processes: Uncertainty-aware Meta Learning via Sequence Modeling. arXiv preprint arXiv:2207.04179 , 2022.

Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-BEATS: Neural Basis Expansion Analysis for Interpretable Time Series Forecasting. arXiv preprint arXiv:1905.10437 , 2019.

Ken Perlin. An Image Synthesizer. ACM Siggraph Computer Graphics , 19(3):287-296, 1985.

Michael JD Powell. Radial Basis Functions for Multivariable Interpolation: A Review. Algorithms for Approximation , 1987.

Andrea Sciacchitano, Douglas R Neal, Barton L Smith, Scott O Warner, Pavlos P Vlachos, Bernhard Wieneke, and Fulvio Scarano. Collaborative framework for piv uncertainty quantification: Comparative assessment of methods. Measurement Science and Technology , 26(7):074004, 2015.

Donald Shepard. A Two-dimensional Interpolation Function for Irregularly-spaced Data. In Proceedings of the 23rd ACM National Conference , pages 517-524, 1968.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems , pages 5998-6008, 2017.

Di Wang and Xi Zhang. Modeling of a 3D Temperature Field by Integrating a Physics-Specific Model and Spatiotemporal Stochastic Processes. Applied Sciences , 9(10), 2019.

Qiqi Wang, Parviz Moin, and Gianluca Iaccarino. A High Order Multivariate Approximation Scheme for Scattered Data Sets. Journal of Computational Physics , 229(18):6343-6361, 2010.

Noah Weber, Janez Starc, Arpit Mittal, Roi Blanco, and Lluís Màrquez. Optimizing over a Bayesian Last Layer. In NeurIPS Workshop on Bayesian Deep Learning , 2018.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the Hierarchical INTerpolation Network (HINT), a hierarchical framework that leverages the residuals on observed points to guide the estimation of the target function. HINT comprises multiple lightweight interpolation blocks arranged sequentially. The first block estimates the main component of the target function, while subsequent blocks predict the residual components using the residuals from the preceding blocks. By accumulating the main component and residual components, HINT produces the final interpolation results. Experiments demonstrate the effectiveness of HINT, and the results demonstrate HINT can get better performance compared with existing interpolation algorithms across a wide range of datasets.\n\nSTRENGTHS:\n1) This paper is easy to follow.\n2) The Ablation Study clearly validates the effectiveness of the proposed auxiliary loss term, masked observed points and local constraint in HINT.\n\nWEAKNESSES:\n1) The main concern is the novelty of this paper is not clearly presented. The major contribution of the paper seems to be the proposed hierarchical residual refining framework but its novelty has not been sufficiently addressed. It is worth noting that previous studies, specifically [1] and [22] as mentioned in the last paragraph of the section of Related Work, have already proposed hierarchical residual architectures for time series forecasting. However, the paper does not discuss the challenges in applying the hierarchical residual refining framework to the field of interpolation, regarding that interpolation can also be viewed as prediction from existing observations. It neither emphasizes the distinction between the proposed architecture and existing architectures. As a result, the relation between this paper and existing literature is not clear and the main contribution of the paper cannot be evaluated.\n2) The presentation is not clear and should be carefully revised. For example, the positions of the observed points are defined twice in Line 83 and Line 84. The expression “It outputs and outputs the coarse function value predictions at both the observed and target points” in Line 94 is wrong.\n3) The experimental results provided in the paper are limited. For instance, the authors did not conduct experiments to investigate the performance of the proposed method with varying ratios of observed points. Therefore, it is hard to evaluate the effectiveness of the proposed method under different data conditions.\n\nQUESTIONS:\nThe motivation of the proposed hierarchical residual refining framework is not clear, could you explain the motivation of hierarchical residual refining framework from the perspective of experiments, such as ablation study, or theoretical analysis?\n\nLIMITATIONS:\nThe limitations of this paper regarding the hyperparameter K in hierarchical local constraints.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors proposed an algorithm named Hierarchical Interpolation Network (HINT) to predict unseen point values in scattered data to replace the usage of manually designed interpolation algorithms. The HINT used the residuals on observed points to guide target function estimation and the hierarchical local constraints in correlation modeling between observed and target points. Experiments show that it outperforms existing interpolation algorithms in accuracy on 3 synthetic datasets.\n\n-- Post rebuttal:\nI read the rebuttal from the auhtors. It addresses some of my concerns on the comparsions and I would like to keep my score.\n\nSTRENGTHS:\nThe algorithm leverages a neural network to adaptively discern the underlying distribution of the target function from a function set or scattered dataset, achieving state-of-the-art performance on four datasets.\n\nWEAKNESSES:\n1. Limited novelty in network design. Mask mechanism and partial attention are used in NIERT. The Embedding Stage and Correlation Modeling in Transformer blocks are very similar to NIERT. \n2. Inadequate method comparisons. The paper compared Neural Processes algorithms proposed in 2018, 2019, and 2020 but the later developments in 2022 not included such as [1] and [2].\n3. It did not use PhysioNet dataset which is compared in NIERT.\n\nIn SM:\n1. Did not explain the reason why it has more FLOPs than NIERT in some datasets.\n2. Mistake： In SM line 105, ‘On the PTV dataset,’ should be ‘On the Perlin dataset,’\n\n[1] Wang, Qi, Marco Federici, and Herke van Hoof. \"Bridge the Inference Gaps of Neural Processes via Expectation Maximization.\" The Eleventh International Conference on Learning Representations. 2022.\n[2] Liu, Huafeng, et al. \"Learning Intrinsic and Extrinsic Intentions for Cold-start Recommendation with Neural Stochastic Processes.\" Proceedings of the 30th ACM International Conference on Multimedia. 2022.\n\nQUESTIONS:\nHow many samples should be observed at least to guarantee the accuracy of your algorithm and this interpolation task? Is there any theory supported by its lower limit?\n\nCan it be used for more dimensional or more complex interpolation？ Such as interpolation for pixels (inpainting, super-resolution, video interpolation)?\n\nLIMITATIONS:\nThe algorithm and the task setting did not consider the situation of incorrect observation (noisy) points which is common in theoretical and engineering domains.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
This work presents a new transformer based approach for scattered data interpolation. By extending NIERT [4] in several aspects, authors achieved improved results in the target task. The extensions include 1) hierarchical residual refinement to produce fine-grained interpolation results and 2) hierarchical local constraints to constrain the set of observed points used in the interpolation block. In short, unlike existing approaches, authors attempt to conduct the scattered data interpolation in a progressive fashion by estimating interpolation residuals within gradually narrowing search space.

STRENGTHS:
1) The proposed components were reasonably designed to obtain fine-grained interpolation results.
2) Intensive ablation studies support the effectiveness of these components.

WEAKNESSES:
1) The datasets used in experiments consists of two-dimensional data. Performance should be evaluated with sparse dataset of much higher dimension.
2) Some explanations were given in the embedding stage, but it is very difficult to interpret why the masked tokens for the given observed points are necessary. 
3) The local constraint K may be problematic, as it can be dependent on the data properties of each application.

QUESTIONS:
Please consider the weaknesses 1), 2), 3).

LIMITATIONS:
N.A.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, a novel hierarchical residual refining framework called HINT (Hierarchical Residual Refining for Scattered Point Interpolation) was proposed to improve interpolation accuracy. The framework utilized residual information from observed points to guide the prediction of target points in a coarse-to-fine manner, employing correlation modeling. HINT consisted of a series of neural network interpolation blocks, including a main block and multiple residual blocks, which estimated different components of the target function at various scales. Local constraints were applied to the interpolation blocks using K-nearest neighbor graph constraints, conforming to hierarchical residual scales. The hierarchical structure of the main and residual components was leveraged, enabling progressively stronger local constraints to enhance interpolation robustness. Experimental evaluations on diverse theoretical and application scenarios demonstrated the superiority of HINT over existing state-of-the-art interpolation methods in terms of accuracy across various tasks.\n\nSTRENGTHS:\nThe paper is well-written and it provides concise descriptions of the proposed framework.\n\nThe experiments conducted in this paper are comprehensive.\n\nWEAKNESSES:\nA couple of assumptions are used, for example in Sec. 2.3, the authors failed to demonstrate that they all holds in experiments.\n\nThe proposed framework is simply a combination of existing modules.\n\nQUESTIONS:\nMore literature reviews should be provide to demonstrate the necessity of using Transformer in interpolation.\n\nWhat are the main differences of the proposed method when compared with  the Transformer encoder-only method in [4]?\n\nThe MSE results in Fig.5 in Supplementary Material can be separated into a table for better clarity, and the authors can provide a case by case analysis for those figures in Supplementary Material.\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
The paper introduces the Hierarchical INTerpolation Network (HINT) as an accurate interpolation algorithm for various theoretical and engineering applications. HINT consists of lightweight interpolation blocks arranged sequentially, where the first block estimates the main component of the target function, and subsequent blocks predict residual components using observed point residuals from preceding blocks.  Moreover, the authors introduce hierarchical local constraints. Extensive experiments demonstrate that HINT significantly outperforms existing interpolation algorithms.

STRENGTHS:
1. The method that utilizing the first and subsequent blocks to estimate the main component and residual components is novel. It applies the residual method to  interpolation algorithm, which is also interesting.
2. The whole method is easy to follow.
3. This paper conduct extensive extriments on various datasets, demonstrateing its superiority.

WEAKNESSES:
1. The comparison of the time cost is missing. It's necessary to compare the proposed HINT and other methods including the traditional ones and deep learning ones.
2. The paper lacks the ablation study of hyper-parameter L and loss scale $\lambda$. Besides, does the value of $\lambda$ keep the same for all the datasets?
3. For Mathit-2D and Perlin, the choich of L and K is (2,6) and (4,2), respectively. Could the author provide some guidlines for the choice ? Or it may Hinder the spread and use of this method.

QUESTIONS:
see above

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-22